Pith. sign in

REVIEW 3 major objections 5 minor 67 references

Visual Instruction Tuning with Chain of Region-of-Interest

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CoRoI lets multimodal language models read high-resolution images by walking a question-guided chain of crops, injecting only those crops into the model's hidden layers.

desk verdict CoRoI's central claim of learned region selection is unsupported by a non-differentiable argmax, and the high-res experiments use upsampled images; the benchmark gains are real but the mechanism is not established. read the letter →

arxiv 2505.06840 v1 pith:DCC6O3Y2 submitted 2025-05-11 cs.CV

Yixin Chen , Shuai Zhang , Boran Han , Bernie Wang This is my paper · ORCID
classification cs.CV
keywords visualinstructiontuninghigh-resolutionimagesmultimodallargelanguagemodelsregionofinterestchainRoIcross-attentioninjectionLLaVA-NeXT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CoRoI is a visual-instruction-tuning method that lets multimodal large language models (MLLMs) exploit high-resolution images without feeding the entire high-resolution image into the language model. It combines a low-resolution global view with the text question, then iteratively selects a chain of regions of interest from upsampled high-resolution images, choosing the most informative crops one at a time. Those crops are injected into the LLM's hidden layers through cross-attention residual connections, so the model can zoom into small text and fine details while the token count stays close to the low-resolution baseline. The paper reports consistent gains over LLaVA-NeXT across 11 benchmarks at 7B-34B scales, with the largest variant matching or surpassing proprietary closed models on several general benchmarks.

What carries the argument

The chain-of-region-of-interest module is the load-bearing mechanism. A score network $f_{\theta_v}$ takes the low-resolution visual features and the question-token embeddings, runs them through self-attention and cross-attention, and outputs a one-channel score map. A sliding-window average-pooling scan over that map selects the highest-scoring patch; the corresponding high-resolution crop is encoded with the frozen ViT and concatenated with the question to select the next patch, building a chain of $n$ regions. The default configuration uses $m=3$ upsampled scales ($\lambda = 2,3,4$) and a chain of $n=4$ regions per scale. Each region's features are then injected into selected LLM layers by a cross-attention module in which the low-resolution hidden states serve as queries and the region features as keys and values, with the output added as a residual. The paper states that this entire pipeline is differentiable.

What would settle it

Run CoRoI under the paper's protocol and compare it with a control that replaces the selected regions with random crops of the same size from the same multi-scale upsampled images. If the random-crop control matches CoRoI's benchmark scores, the learned chain-of-relevance mechanism is doing none of the work. Separately, measure the gradient norm of the selector network $f_{\theta_v}$ during instruction fine-tuning: if it is identically zero, the hard argmax blocks the learning signal the paper's differentiable claim requires.

Watch

Extended reading notes

Core claim

This paper claims that for any given question, the useful content of a high-resolution image is concentrated in a short chain of crops, and a model can find those crops by using the low-resolution image plus the question as a guide. It argues that injecting the selected crops' visual features into the LLM's hidden layers via cross-attention lets the model reason about fine-grained detail, most visibly in OCR-heavy tasks such as TextVQA, while avoiding the quadratic token-cost growth of long high-resolution sequences. The evidence is benchmark performance: CoRoI-v1 and CoRoI-v2, built on Mistral-7B and Hermes-2-Yi-34B, improve on LLaVA-NeXT at comparable sizes, and the 34B variant outperforms Gemini Pro 1.0 on five of six comprehensive benchmarks and GPT-4V on MMB, SEED-I, and MME.

Load-bearing premise

The weaker link is the region-selection step: it picks the single highest-scoring window by a hard argmax, which has zero gradient almost everywhere, so the score network may receive no learning signal from the language-model loss; if that is true, the selected regions are not actually learned and the method becomes the injection of fixed crops into the LLM.

Editorial extensions

If this is right

  • If CoRoI is right, a multimodal LLM can answer questions about high-resolution images while feeding the language model only a low-resolution token sequence, because the informative crops enter through cross-attention residuals rather than as extra input tokens.
  • The consistent gains over LLaVA-NeXT across 7B, 13B, and 34B backbones imply that selective region zooming is a transferable recipe, not a quirk of one model size.
  • On text-heavy benchmarks such as TextVQA, the chain mechanism can move to signs and text blocks that carry the answer, giving the largest relative improvements.
  • A 34B CoRoI model can match or surpass closed-source models on several general multimodal benchmarks, suggesting that efficient input selection can close part of the gap that brute-force resolution scaling leaves open.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the chain mechanism is trained end-to-end and query-guided, a natural extension is multi-turn or agentic look-where-uncertain behavior, where the LLM decides the next region from its own current answer state.
  • Since high-resolution images are produced by bilinear upsampling, the method's ceiling is set by information already present at the base resolution; replacing upsampling with true high-resolution captures or learned super-resolution is a direct next step.
  • The design suggests a general recipe: keep the language model's token budget fixed and spend it on query-relevant crops, which could transfer to videos or gigapixel medical and satellite images with the same cross-attention injection.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CoRoI (Chain of Region-of-Interest), a visual instruction tuning method for high-resolution multimodal LLMs. CoRoI first uses low-resolution visual features and question embeddings to iteratively select a chain of high-resolution image regions through a sliding-window average-pooling argmax, then injects the resulting region tokens into intermediate LLM hidden states via cross-attention residual connections. The authors claim the entire pipeline is differentiable and trainable with the standard causal language modeling loss. Experiments are conducted on 11 benchmarks with Mistral-7B and Hermes-2-Yi-34B backbones, comparing with LLaVA-NeXT and proprietary models, and reporting consistent gains across most benchmarks.

Significance. If the claimed learning mechanism worked as described, CoRoI would be a useful efficiency/accuracy contribution to high-resolution MLLM design, and the experimental setup has notable strengths: the backbones match LLaVA-NeXT, the benchmark coverage is broad, and the ablation study is extensive. However, the central claim that the model learns to select informative regions is not supported by the described method, because the hard-argmax selector receives no gradient from the language modeling loss. This is a load-bearing issue, not a presentation issue. The empirical gains may be genuine, but they cannot be attributed to learned RoI selection as stated.

major comments (3)
  1. [Section 3.1 and Section 3.3] The selection of each RoI uses a hard argmax over a sliding-window average-pooling score map produced by f_theta_v, followed by a non-differentiable integer crop from the high-resolution image. Section 3.3 trains theta_v jointly with the LLM using the causal language modeling loss l = -sum log p_theta(y_i | ...). Because the argmax has zero gradient almost everywhere, and no relaxation (Gumbel-softmax, straight-through estimator, REINFORCE baseline, or auxiliary selection objective) is described anywhere in the paper, the gradient of the LM loss with respect to theta_v is zero almost everywhere. This directly contradicts the Section 2 claim that 'the entire process is differentiable.' Consequently, the paper's central conclusion that CoRoI learns to identify and prioritize the most informative regions is unsupported. The authors must either describe a concrete training signal that reaches the selector, or explicitly state that the selector is fixed; in the latter case, the benchmark gains can only be attributed to injecting fixed or random crops, not to learned RoI selection.
  2. [Section 4.1 and Section 5] The 'high-resolution' images used throughout the method and experiments are generated by bilinearly upsampling 336x336 images to 672x672, 1008x1008, and 1344x1344. Bilinear interpolation does not introduce new high-frequency information, so the model never actually observes genuine high-resolution detail. This is inconsistent with the stated motivation that high-resolution input is essential for small objects and fine-grained recognition, and with the Limitations paragraph, which only says that information loss 'may' occur. The experiments therefore evaluate a multi-scale cropped-view model rather than a model that processes true high-resolution images. The authors should either provide experiments with native high-resolution images or substantially revise the claims to reflect that the inputs are interpolated versions of low-resolution images.
  3. [Tables 3 and 4] The hyperparameters n (number of RoIs), m and lambda (multi-scale factors), and the injection layer positions k are selected based on ablations on TextVQA, MMBench, and MME, and these same benchmarks appear in the final comparison tables (Tables 1 and 2). This selection-on-evaluation procedure introduces optimistic bias into the reported gains, because the model is effectively tuned to the evaluation set. The authors should either perform hyperparameter selection on a held-out validation split or disclose and justify the selection protocol; otherwise the claim of consistent gains across benchmarks is weakened.
minor comments (5)
  1. [Abstract and Table 1] The abstract states that the 34B model surpasses Gemini Pro 1.0 on 'six benchmarks,' but Table 1 shows only five of six comprehensive benchmarks where CoRoI-v2-34B outperforms Gemini Pro (SEED-I, MMB, MME, MMMU, MathVista, but not MM-Vet), and Gemini Pro is not listed in Table 2. Please reconcile the count or specify the exact benchmarks.
  2. [Section 4.3] The text refers to a 'Titanic example' in Figure 3, but the second example in Figure 3 is a radar chart about MM-Vet, not a Titanic-related image. Please correct the description or replace the figure.
  3. [Table 3] Table 3 is difficult to read: the separation into an 'upper part' and 'lower part' is not visually clear, and the delta values (e.g., +5.1, -0.7) are not explicitly defined relative to which baseline. Please restructure the table or clearly state the reference model for each delta.
  4. [Section 3.3] The loss notation p_theta(y_i | hat{y}_{1:i-1}, q) uses a hat on the history, which is confusing under teacher forcing; the history should be ground-truth tokens y_{1:i-1} unless a generation-based training objective is intended. Please clarify.
  5. [Appendix] The layer selection rule described in the appendix says the default layer set is obtained by 'doubling the layers within the section close to the middle of LLM,' but the example for 32 layers starts with {7, 15, 23} and ends with {7, 11, 15, 19, 23, 27}; this is not a simple doubling of a contiguous section, so the selection rule is unclear. Please provide a precise algorithm.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation found; benchmark gains are externally evaluated, and the sole self-citation is a non-load-bearing related-work entry.

full rationale

CoRoI's central claim is evaluated on standard external benchmarks (SEED, MMB, MME, MM-Vet, MMMU, MathVista, TextVQA, GQA, VQA v2, VizWiz, SQAI) using models trained on LLaVA/ALLaVA-style instruction data; no benchmark number is used as a fitted parameter in the forward pass, and no equation in Section 3 defines the reported gains in terms of the method's own inputs. The selection map f_theta_v is trained by the causal LM loss in Section 3.3, and 'most informative region' is operationally defined as the argmax of average-pooled scores in Section 3.1; that is a definition, not a circular reduction of the benchmark results. The only self-citation is [14] (CamML), which appears in a related-work list of fixed-resolution MLLMs ('such as 336x336 and 448x448 [37, 35, 65, 60, 66, 14, 59, 6]') and is not load-bearing. The hard-argmax/non-differentiability issue (Section 3.1 versus Section 3.3) is a genuine correctness risk—if no relaxation is used, the gradient of the LM loss with respect to theta_v is zero almost everywhere—but it is an internal inconsistency, not a circularity. Likewise, ablating hyperparameters (n, m, lambda, layer k) on the same benchmarks used for final tables can bias results but does not make the reported numbers equivalent to the method's inputs by construction. Score 2 reflects only the minor non-load-bearing self-citation; no circular derivation was found.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim depends on three unverified premises: that the score map is learnable despite the hard argmax, that upsampled images are acceptable stand-ins for real high-resolution inputs, and that average-pooled window scores indicate informativeness. None of these is supported by independent evidence in the paper.

free parameters (4)
  • RoI count n = 4
    Number of regions in the chain, set by ablation in Table 3.
  • Multi-scale factors m and lambda = m=3, lambda in {2,3,4}
    HR upsampling factors, selected by ablation; single-scale and pair combinations hurt performance according to Table 3.
  • Injection layer positions k = {7,11,15,19,23,27} for Mistral-7B
    The paper uniformly divides LLM layers into 4 sections and doubles layers near the middle; ablation shows a single layer is worse (Table 3).
  • Learning rates = 1e-3 projector, 2e-5 or 1e-5 instruction
    Standard optimization hyperparameters; not central to the method but required to reproduce.
assumptions (3)
  • domain assumption A sliding-window average-pooling score identifies the most informative region for a question.
    Section 3.1 chooses regions by highest pooled value of a predicted score map; no evidence that this correlates with informativeness.
  • ad hoc to paper Bilinearly upsampled low-resolution images serve as a valid proxy for high-resolution images in evaluating the method.
    Section 4.1 creates 'high-resolution' inputs via bilinear interpolation from 336x336 images, so the experiments do not reflect real HR detail.
  • ad hoc to paper The causal language modeling loss provides a sufficient learning signal to train the hard region-selection module.
    Section 3.3 trains everything except the vision encoder with the standard LM loss, but the argmax selection in Section 3.1 has no obvious gradient path; the paper does not describe a relaxation or auxiliary loss.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Visual Instruction Tuning with Chain of Region-of-Interest." pith.science (2026). https://pith.science/paper/DCC6O3Y2

@misc{pith2026250506840,
  author       = {Pith},
  title        = {Pith review of: Visual Instruction Tuning with Chain of Region-of-Interest},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DCC6O3Y2}},
  note         = {Machine review of arXiv:2505.06840}
}
read the original abstract

High-resolution (HR) images are pivotal for enhancing the recognition and understanding capabilities of multimodal large language models (MLLMs). However, directly increasing image resolution can significantly escalate computational demands. In this study, we propose a method called Chain of Region-of-Interest (CoRoI) for Visual Instruction Tuning, aimed at alleviating the computational burden associated with high-resolution images for MLLMs. Drawing inspiration from the selective nature of the human visual system, we recognize that not all regions within high-resolution images carry equal importance. CoRoI seeks to identify and prioritize the most informative regions, thereby enhancing multimodal visual comprehension and recognition while circumventing the need for processing lengthy HR image tokens. Through extensive experiments on 11 benchmarks, we validate the efficacy of CoRoI across varying sizes, ranging from 7B to 34B in parameters. Our models consistently demonstrate superior performance across diverse multimodal benchmarks and tasks. Notably, our method outperforms LLaVA-NeXT on almost all benchmarks and our finetuned 34B model surpasses proprietary methods like Gemini Pro 1.0 on six benchmarks, as well as outperforming GPT-4V on MMB, SEED-I, and MME.

Figures

Figures reproduced from arXiv: 2505.06840 by the authors.

Figure 1
Figure 1. Zero-shot performance comparison with LLaVA-1.5, LLaVA-NeXT, Gemini-Pro 1.0, and GPT-4V. To address these issues, we propose a method named Chain of Region-of-Interest (CoRoI) for visual instruction fine-tuning. CoRoI consists of two main steps: (1) Chain of Region-of-Interest Extraction: CoRoI captures the most informa￾tive regions from high-resolution images by in￾tegrating visual clues (i.e., low-resolution (LR) … view at source ↗
Figure 2
Figure 2. Illustration of CoRoI. Low-resolution images and text clues are used to identify a chain of [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. CoRoI enables the model to extract a chain of region-of-interest during inference. We [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Modules design details. 3.1 Chain of Region-of-Interest Extraction Here, we introduce the chain of region-of-interest extraction methodology (shown in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Reasoning Generative Capability. The generated images are from DALLE3 using prompts [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 46 canonical work pages

  1. [1]

    https://cdn.openai.com/papers/GPTV_System_Card.pdf, 2023

    Gpt-4v(ision) system card. https://cdn.openai.com/papers/GPTV_System_Card.pdf, 2023. 1, 2, 6, 7

    work page 2023

  2. [2]

    https://sharegpt.com, 2023

    Sharegpt. https://sharegpt.com, 2023. 7

    work page 2023

  3. [3]

    https://www-cdn.anthropic.com/ de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf, 2024

    The claude 3 model family: Opus, sonnet, haiku. https://www-cdn.anthropic.com/ de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf, 2024. 2

    work page 2024

  4. [4]

    https://huggingface.co/datasets/laion/gpt4v-dataset, 2024

    Laion-gpt-4v dataset. https://huggingface.co/datasets/laion/gpt4v-dataset, 2024. 7, 8

    work page 2024

  5. [5]

    https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B , 2024

    Nous-hermes-2-yi-34b. https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B , 2024. 6

    work page 2024

  6. [6]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 2022. 1, 2, 3

    work page 2022

  7. [7]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In ICCV, 2015. 2, 7

    work page 2015

  8. [8]

    Openflamingo, Mar

    Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo, Mar. 2023. 2

    work page 2023

Show all 67 references
  1. [9]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 6, 7

    2023 arXiv

  2. [10]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020. 2

    2020

  3. [11]

    Allava: Harnessing gpt4v-synthesized data for a lite vision-language model

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684, 2024. 6, 7, 8

    2024 arXiv

  4. [12]

    Shikra: Unleashing multimodal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 6

    2023 arXiv

  5. [13]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793,

    arXiv

  6. [14]

    Camml: Context-aware multimodal learner for large models

    Yixin Chen, Shuai Zhang, Boran Han, Tong He, and Bo Li. Camml: Context-aware multimodal learner for large models. arXiv preprint arXiv:2401.03149, 2024. 3

    2024 arXiv

  7. [15]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang,...

    2024

  8. [16]

    Mobilevlm v2: Faster and stronger baseline for vision language model

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766, 2024. 6

    2024 arXiv

  9. [17]

    Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. 6

    2023

  10. [18]

    Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd, 2024

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, a...

    2024

  11. [19]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  12. [20]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 2, 7, 8

    2023 arXiv

  13. [21]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. CVPR, 2018. 2, 7

    2018

  14. [22]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312.08914, 2023. 1, 2, 3

    2023 arXiv

  15. [23]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 2, 7

    2019

  16. [24]

    Hénaff, Matthew M

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J. Hénaff, Matthew M. Botvinick, 10 Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver IO: A gene...

    2022

  17. [25]

    Perceiver: General perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and João Carreira. Perceiver: General perception with iterative attention. In ICML, 2021. 3

    2021

  18. [26]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023

  19. [27]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024

  20. [28]

    Dvqa: Understanding data visualizations via question answering

    Kushal Kafle, Scott Cohen, Brian Price, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In CVPR, 2018. 7

    2018

  21. [29]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Michael Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. ArXiv, abs/1603.07396, 2016. 7, 8

    2016 arXiv

  22. [30]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In...

    2017

  23. [31]

    Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh

    Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023. 6

    2023

  24. [32]

    Seed-bench: Benchmarking multimodal llms with generative comprehension, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension, 2023. 2, 7

    2023

  25. [33]

    Otterhd: A high- resolution multi-modality model

    Bo Li, Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang, Fanyi Pu, and Ziwei Liu. Otterhd: A high- resolution multi-modality model. arXiv preprint arXiv:2311.04219, 2023. 1, 3, 6

    2023 arXiv

  26. [34]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. CoRR, 2023. 2

    2023

  27. [35]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 1, 2, 3, 6, 8

    2023 arXiv

  28. [36]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024. 1, 2, 3, 6

    2024

  29. [37]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Thirty-seventh Conference on Neural Information Processing Systems , 2023. 1, 2, 3, 5, 7

    2023

  30. [38]

    Mmbench: Is your multi-modal model an all-around player? arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhnag, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? arXiv:2307.06281, 2023. 2, 7

    2023 arXiv

  31. [39]

    UNIFIED-IO: A unified model for vision, language, and multi-modal tasks

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. UNIFIED-IO: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations, 2023. 3

    2023

  32. [40]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representat...

    2024

  33. [41]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems ...

    2022

  34. [42]

    A survivor in the era of large-scale pretraining: An empirical study of one-stage referring expression comprehension

    Gen Luo, Yiyi Zhou, Jiamu Sun, Xiaoshuai Sun, and Rongrong Ji. A survivor in the era of large-scale pretraining: An empirical study of one-stage referring expression comprehension. IEEE Transactions on Multimedia, 26:3689–3700, 2024. 1, 3

    2024

  35. [43]

    Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models, 2024

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models, 2024. 1, 3, 6

    2024

  36. [44]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, 2019. 7

    2019

  37. [45]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022 , 2022. 7

    2022

  38. [46]

    Minesh Mathew, Dimosthenis Karatzas, and C. V . Jawahar. Docvqa: A dataset for vqa on document images, 2021. 7, 8

    2021

  39. [47]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019. 7

    2019

  40. [48]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 2

    2023

  41. [49]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML. PMLR, 2021. 2, 5 11

    2021

  42. [50]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages ...

    2020

  43. [51]

    A-okvqa: A benchmark for visual question answering using world knowledge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. arXiv, 2022. 7

    2022

  44. [52]

    Unified model for image, video, audio and language tasks, 2023

    Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. Unified model for image, video, audio and language tasks, 2023. 3

    2023

  45. [53]

    Textcaps: a dataset for image captioningwith reading comprehension

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioningwith reading comprehension. 2020. 7, 8

    2020

  46. [54]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019. 2, 8

    2019

  47. [55]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv e-prints , pages arXiv–2405, 2024. 3

    2024

  48. [56]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1, 2, 6, 7

    2023 arXiv

  49. [57]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint arXiv:2401.06209, 2024. 1, 3

    2024 arXiv

  50. [58]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023

  51. [59]

    Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multi- modal few-shot learning with frozen language models. In NeurIPS, 2021. 1, 3

    2021

  52. [60]

    Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to- sequence learning framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to- sequence learning framework. In ICML. PMLR, 2022. 1, 3

    2022

  53. [61]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023. 1, 2, 3, 6

    2023 arXiv

  54. [62]

    Berg, and Tamara L

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, ECCV, 2016. 7

    2016

  55. [63]

    Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023. 2, 7, 8

    2023

  56. [64]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  57. [65]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention

    Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023. 1, 2, 3

    2023 arXiv

  58. [66]

    Minigpt-4: Enhancing vision- language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 3

    2023 arXiv

  59. [67]

    Uni- perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks

    Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni- perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In CVPR, 2022. 3 12 A Appendix Experimental Details We present the experimental de...

    2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.