Pith. sign in

REVIEW 4 major objections 4 minor 39 references

CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By making a multimodal LLM locate the answer region first and then amplify attention to that region during decoding, this paper claims a training-free boost to fine-grained visual understanding and a reduction in hallucination.

desk verdict CoF is a plausible, training-free attention-reweighting trick with a real but thin empirical case; the missing localization-quality analysis is the main gap. read the letter →

arxiv 2412.16869 v1 pith:WEZRPUUC submitted 2024-12-22 cs.CV

Yeyuan Wang , Dehong Gao , Bin Li , Rujiao Long , Lei Yi , Xiaoyan Cai , Libin Yang , Jinxia Zhang
show 2 more authors
Shanqing Yu Qi Xuan
This is my paper · ORCID
classification cs.CV
keywords multi-modallargelanguagemodelfine-grainedunderstandingvisualgroundingattentionreweightingpromptengineeringhallucinationreductiontraining-freeinferencecoarse-to-fine
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CoF claims that a large part of the fine-grained failure of multimodal large language models is an attention-allocation problem, not just an encoder-resolution problem. The paper proposes a training-free, two-stage inference method: first prompt the model itself to output a coarse bounding box around the image region that contains the answer; then, during decoding, multiply the attention scores of visual tokens inside that box by a scaling factor (equivalently, add $\log\lambda$ to those logits). Across LLaVA-v1.5-7B/13B and InstructBLIP-13B, this lifts scores on MME, MMBench, and the hallucination benchmark POPE while keeping the whole image in context. If right, it means fine-grained perception can be improved cheaply, without finetuning or cropping, by steering attention toward the located region.

What carries the argument

The load-bearing mechanism is attention reweighting by a binary spatial mask, driven by the model's own coarse localization. The paper defines the modified attention distribution as $\hat{A} = \mathrm{softmax}(\log(\lambda)\,M + A)$, where $A$ is the original attention score matrix, $M$ is a mask with 1s on visual tokens inside the located bounding box and 0s elsewhere, and $\lambda>1$ is a scaling factor. Adding $\log\lambda$ inside the softmax increases those tokens' probabilities by a multiplicative factor, shifting the decoder's focus to the answer region while leaving the full image in the token sequence. The two-stage breakdown — grounding prompt $P_g$ first, attention reweighting second — is the coarse-to-fine decomposition the paper names CoF.

What would settle it

Run CoF on a fine-grained VQA set but replace the model-generated bounding box with a random box of the same size and distribution; if scores do not drop back to (or below) the no-mask baseline, the gain is not causally tied to localizing the answer. A complementary check: annotate ground-truth answer regions, measure the IoU of the model's boxes, and verify that the performance gain appears only on examples whose IoU is high.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that coarse-to-fine attention reweighting — locating the answer region first and then boosting attention to it — significantly improves the fine-grained understanding of multimodal LLMs. The first stage uses a grounding prompt $P_g$ ("According to the question, detect the answer in the image") to make the MLLM output JSON bounding-box coordinates for the answer region; the coordinates are post-processed by centering, expanding by a hyperparameter $\alpha$, and clipping to the image. The second stage converts the box into a binary mask $M$, and modifies every decoder attention map as $\hat{A} = \mathrm{softmax}(\log(\lambda)\,M + A)$, which multiplies the softmax probability of visual tokens in the mask by $\lambda$. The paper reports consistent gains over the baselines on MME, MMBench, and POPE, and its ablation shows that attention reweighting alone helps, but adding the grounding stage helps more — evidence that localizing the region is what carries the improvement.

Load-bearing premise

The method assumes that the same model can reliably find the answer region — by emitting a bounding box — before it can answer the question; a wrong or imprecise box would be amplified by the attention scaling and could push answers below the baseline.

Editorial extensions

If this is right

  • Existing MLLMs can gain fine-grained perception and reduced hallucination without any finetuning, extra data, or higher-resolution encoders — only an extra grounding prompt and an attention scaling factor.
  • The full-image context is preserved, so the method avoids the semantic-integrity loss that comes from cropping and re-encoding the target region.
  • Because the gains reproduce across MLP-projector models (LLaVA) and a Q-former model (InstructBLIP), the attention-reweighting mechanism transfers across architectural choices in the vision-language connector.
  • The POPE improvements suggest that focusing attention on the located region reduces object hallucination, pointing to attention allocation as a lever for grounding.
  • The method can be switched on or off at inference time per question, making it a drop-in inference recipe rather than a model modification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report localization accuracy; a natural extension would be to measure box IoU against human annotations and show that CoF's gain scales with localization quality, which would confirm the proposed mechanism rather than a general 'extra prompt helps' effect.
  • If the attention-boosting mechanism is robust, it could combine with multi-crop or high-resolution strategies, potentially reducing the compute needed for fine-grained tasks by making single-pass attention sufficient.
  • One untested risk the paper leaves open: on questions where the answer region is diffuse or the model's box is wrong, the same amplification could amplify background noise; a confidence-weighted version that down-weights or skips the mask when localization is uncertain is a direct testable variant.
  • The two-stage design suggests a broader recipe: any task where a cheap first pass can propose a spatial focus (segmentation, salient object detection, OCR regions) could reuse the same attention-reweighting step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper proposes CoF, a training-free two-stage inference procedure for multimodal large language models (MLLMs). In the first stage, the MLLM is prompted (Section II-B) to output bounding-box coordinates for the image region containing the answer; the box is then expanded by hyperparameter α and clipped to the image. In the second stage, a binary mask derived from this box is used to reweight the attention logits of visual tokens through Eq. (3), with scaling factor λ, before the model generates its answer. The method is applied to LLaVA-v1.5-7B/13B and InstructBLIP-13B and evaluated on MME, MMBench, and POPE. The authors report aggregate improvements over the baselines and an ablation on LLaVA-v1.5-13B showing that combining reweighting and grounding yields the best scores.

Significance. If the reported gains are robust, CoF is an attractive contribution: it requires no training, no additional modules, and no image cropping, while preserving global context and focusing attention on an estimated answer region. The connection between visual grounding and hallucination reduction is plausible, and the authors release code, which aids reproducibility. However, the empirical support is currently incomplete: the two hyperparameters α and λ are tuned separately per model on the same benchmarks, the tables report single-run aggregate scores without sensitivity analysis, and the central grounding stage is not validated independently. Because the ablation in Table II shows that the grounding stage is responsible for a substantial share of the MME Perception gain, the overall claim of significant improvement rests on an unquantified localization step.

major comments (4)
  1. [Section III-A and Table I] The per-model values of α and λ (1.3/2.0, 1.0/4.5, 1.0/22.0) are chosen on the evaluation benchmarks themselves, and no selection procedure, grid, or held-out configuration is reported. Because the headline numbers are the same numbers used to select these constants, the reported gains are partly a statement about the optimizer's success rather than a prediction from a fixed method. Please report a sweep or a principled selection rule (e.g., tuning on a development set disjoint from the test splits) and add sensitivity curves for λ around the chosen values; at minimum, state explicitly how the values were obtained.
  2. [Section II-B and Table II] The method's first stage is assumed to localize the answer region, and the ablation in Table II shows that removing the grounding stage lowers MME Perception from 1545.6 to 1527.2, so the benefit is attributable to the predicted boxes. Yet the paper reports no metric for localization accuracy, no examples of correct or incorrect boxes, no parsing success rate for the JSON output, and no fallback for invalid or empty boxes, which are especially relevant for POPE questions whose correct answer is "No" and for which no answer region exists. Because Eq. (3) applies λ times the mask before softmax, an imprecise or wrong box suppresses the true visual evidence; with λ = 22.0 for InstructBLIP this suppression is strong. Please quantify grounding quality (e.g., IoU against human-annotated regions on a sample, or at least successful-parsing and confidence statistics) and describe the behavior when the first stage fails.
  3. [Table I] All results are single aggregate runs with no variance, no number of repetitions, and no significance tests. Some sub-scores decrease or stay flat (InstructBLIP MME Cognition 291.8 to 290.8 and POPE Random 87.7 to 87.5; LLaVA-v1.5-7B POPE Adversarial 84.2 to 84.2), while the largest gains are concentrated in specific subtasks such as InstructBLIP MMBench Test (36.7 to 50.6). Given the small and uneven deltas, the claim that CoF significantly boosts performance needs either repeated evaluation with standard deviations or per-subtask breakdowns with evidence that the improvements are consistent rather than driven by a few items.
  4. [Section II-C, Eq. (3)] The text says the attention scores of image tokens are multiplied by λ, but Eq. (3) adds log(λ)·M to A before softmax; this is equivalent only if A denotes pre-softmax logits. The paper does not specify whether Eq. (3) is applied before or after softmax, nor does it say which attention layers in the LLaVA decoder receive the modification. In addition, for InstructBLIP the mask is applied to the Q-former rather than to the decoder attention, and this difference is not described precisely. Please clarify the exact insertion point and the Q-former variant, since these details determine whether the reported behavior is reproducible.
minor comments (4)
  1. [Section II-A, Eq. (2)] There is a typo, "scaling facter," and the softmax axis in Eq. (2) is not specified; please state whether the softmax is over the key dimension and which attention heads or layers are reweighted in the experiments.
  2. [Figures 1 and Section II-B] The grounding prompt shown in Figure 1 ("According to the question, detect the answer in the image.") differs from the prompt described in Section II-B ("detail the bounding box ... in JSON format"). The exact prompt matters because it determines whether the MLLM's output is parseable, so please make the two consistent and state which prompt was used in the reported experiments.
  3. [Section III] The paper does not discuss computational overhead: running two forward passes per question doubles inference cost, and this trade-off should be quantified in the experiments, especially for the 13B models.
  4. [References] Several references are to arXiv preprints without venue information, and the citation formats are inconsistent; please standardize the bibliography according to the target style.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: per-model attention hyperparameters (α and λ) are tuned on the same evaluation benchmarks whose improved scores are then presented as evidence of generalization, so the reported gains are partly in-sample fits; the two-stage mechanism itself is not definitionally forced.

  1. fitted input called prediction [Section III-A (hyperparameters paragraph); results in Table I; abstract generalization claim]
    "Regarding hyperparameter settings, the scaling hyperparameter α is set to 1.3, and λ to 2.0 for the LLaV A-v1.5-7B model. For the LLaV A-v1.5-13B model, α is set to 1.0 while λ is adjusted to 4.5. In the case of InstructBLIP-13B, α is set to 1.0 and λ is set to 22.0."

    CoF's reported gains depend directly on the attention scaling factor λ and box expansion α, since Eq. (3) multiplies the mask M by log(λ). The paper states that α and λ are 'set' or 'adjusted' per model but does not describe any validation split; the evaluation in Table I uses the same MME, MMBench, and POPE benchmarks. Thus the reported improvements are in-sample outcomes of the parameter choice, not independent predictions of CoF. The abstract's 'notable generalization' claim is therefore not supported by the evidence as presented. This is partial rather than total circularity because the second stage still processes the actual image tokens through the MLLM, and the mechanism could be tested on held-out settings.

full rationale

The CoF pipeline is not definitionally circular. Equation (3) reweights attention with a mask built from first-stage coordinates, but the final answer is generated by the LLM from the reweighted visual tokens, so one cannot algebraically reduce the output to the input box. The first stage uses the same MLLM to localize the answer region; this is a self-referential loop and a robustness risk, but not a circular derivation because no equation makes the final response equal to the first-stage coordinates or to the fitted constants. The main circularity-adjacent flaw is the hyperparameter story: λ and α are tuned per model on the full evaluation benchmarks, and the same benchmarks then appear as the evidence of improvement. This undercuts the 'notable generalization' statement and makes the headline gains partially selected rather than predicted. The paper also reports no localization accuracy, no failure cases, and no fallback for empty or invalid boxes, especially for negative POPE questions; these are important correctness concerns but lie outside circularity. Self-citations ([12]) appear only as general background and are not load-bearing, and no uniqueness theorem or ansatz is imported from the authors' prior work. Verdict: one partial fitted-input issue; the central mechanism retains independent empirical content.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim relies on two per-model hyperparameters (alpha and lambda), on the untested accuracy of the model's self-localization, and on the assumption that attention logits can be safely reweighted in all decoder layers. No new physical or architectural entities are introduced.

free parameters (2)
  • alpha (bounding box expansion factor) = 1.3 (LLaVA-7B), 1.0 (LLaVA-13B), 1.0 (InstructBLIP-13B)
    Introduced in Section II-B post-processing; chosen per model with no validation protocol or sensitivity analysis reported.
  • lambda (attention scaling factor) = 2.0 (LLaVA-7B), 4.5 (LLaVA-13B), 22.0 (InstructBLIP-13B)
    Introduced in Section II-C, Eq. 3; tuned per model on the evaluation benchmarks; no held-out tuning set described.
assumptions (5)
  • domain assumption Visual tokens in the LLM decoder preserve the spatial layout of the input image, so a bounding box over image coordinates maps one-to-one to a token mask.
    Section II-C converts bounding box coordinates into a binary mask over visual tokens without discussing token grid alignment or resolution mismatch.
  • domain assumption Multiplying attention logits by log(lambda) via the mask in every decoder layer produces the intended region focus and does not destabilize decoding.
    Section II-C, Eq. 3; no analysis of attention distribution shift or decoding degradation is provided.
  • domain assumption The MLLM's own first-stage bounding box output is accurate enough to localize the answer region.
    Section II-B; the method provides no fallback for malformed or wrong coordinates and no localization accuracy measurement.
  • domain assumption Benchmark scores (MME, MMBench, POPE) are reliable proxies for fine-grained understanding and hallucination.
    Section III-A uses these benchmarks as ground truth for the claimed capability improvement.
  • standard math Softmax attention formula and scaling behavior follow standard transformer assumptions.
    Section II-A Eq. 2 and Eq. 3 use standard softmax attention; no novel math is introduced.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models." pith.science (2026). https://pith.science/paper/WEZRPUUC

@misc{pith2026241216869,
  author       = {Pith},
  title        = {Pith review of: CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WEZRPUUC}},
  note         = {Machine review of arXiv:2412.16869}
}
read the original abstract

The impressive performance of Large Language Model (LLM) has prompted researchers to develop Multi-modal LLM (MLLM), which has shown great potential for various multi-modal tasks. However, current MLLM often struggles to effectively address fine-grained multi-modal challenges. We argue that this limitation is closely linked to the models' visual grounding capabilities. The restricted spatial awareness and perceptual acuity of visual encoders frequently lead to interference from irrelevant background information in images, causing the models to overlook subtle but crucial details. As a result, achieving fine-grained regional visual comprehension becomes difficult. In this paper, we break down multi-modal understanding into two stages, from Coarse to Fine (CoF). In the first stage, we prompt the MLLM to locate the approximate area of the answer. In the second stage, we further enhance the model's focus on relevant areas within the image through visual prompt engineering, adjusting attention weights of pertinent regions. This, in turn, improves both visual grounding and overall performance in downstream tasks. Our experiments show that this approach significantly boosts the performance of baseline models, demonstrating notable generalization and effectiveness. Our CoF approach is available online at https://github.com/Gavin001201/CoF.

Figures

Figures reproduced from arXiv: 2412.16869 by the authors.

Figure 1
Figure 1. Overview of (a) baseline approach and (b) our CoF approach. CoF consists of two stages: (1) Location Grounding and (2) Attention Reweighting. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of CoF approach. (a) In the first stage, the model determines the answer area in the input image based on the question. (b) Then the output [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 21 canonical work pages

  1. [1]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  2. [2]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez et al., “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” See https://vicuna. lmsys. org (accessed 14 April 2023), vol. 2, no. 3, p. 6, 2023

    work page 2023

  3. [3]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” 2023

    work page 2023

  4. [4]

    Im- proved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Im- proved baselines with visual instruction tuning,” in Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 296–26 306

    work page 2024

  5. [5]

    Instructblip: Towards general-purpose vision- language models with instruction tuning,

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision- language models with instruction tuning,” 2023. [Online]. Available: https://arxiv.org/abs/2305.06500

    arXiv 2023

  6. [6]

    Making llama see and draw with seed tokenizer,

    Y . Ge, S. Zhao, Z. Zeng, Y . Ge, C. Li, X. Wang, and Y . Shan, “Making llama see and draw with seed tokenizer,” arXiv preprint arXiv:2310.01218, 2023

    arXiv 2023

  7. [7]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, Y . Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  8. [8]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

Show all 39 references
  1. [9]

    Fashionbert: Text and image matching with adaptive loss for cross- modal retrieval,

    D. Gao, L. Jin, B. Chen, M. Qiu, P. Li, Y . Wei, Y . Hu, and H. Wang, “Fashionbert: Text and image matching with adaptive loss for cross- modal retrieval,” in International ACM SIGIR Conference on Research and Development in Information Retrieval, Xi’an, China, 2020, pp. 2251–2260

    2020

  2. [10]

    Vary: Scaling up the vision vocabulary for large vision- language models,

    H. Wei, L. Kong, J. Chen, L. Zhao, Z. Ge, J. Yang, J. Sun, C. Han, and X. Zhang, “Vary: Scaling up the vision vocabulary for large vision- language models,” arXiv preprint arXiv:2312.06109, 2023

    2023 arXiv

  3. [11]

    Osprey: Pixel understanding with visual instruction tuning,

    Y . Yuan, W. Li, J. Liu, D. Tang, X. Luo, C. Qin, L. Zhang, and J. Zhu, “Osprey: Pixel understanding with visual instruction tuning,” in Conference on Computer Vision and Pattern Recognition, 2024, pp. 28 202–28 211

    2024

  4. [12]

    Enhanc- ing fine-grained vision-language pretraining with negative augmented samples,

    Y . Wang, D. Gao, L. Yi, L. Jin, J. Zhang, L. Yang, and X. Cai, “Enhanc- ing fine-grained vision-language pretraining with negative augmented samples,” arXiv preprint arXiv:2412.10029, 2024

    2024 arXiv

  5. [13]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agar- wal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning, 2021, pp. 8748–8763

    2021

  6. [14]

    Scaling up visual and vision- language representation learning with noisy text supervision,

    C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision- language representation learning with noisy text supervision,” in International Conference on Machine Learning, 2021, pp. 4904–4916

    2021

  7. [15]

    Kaleido-bert: Vision-language pre-training on fashion domain,

    M. Zhuge, D. Gao, D.-P. Fan, L. Jin, B. Chen, H. Zhou, M. Qiu, and L. Shao, “Kaleido-bert: Vision-language pre-training on fashion domain,” in Conference on Computer Vision and Pattern Recognition, 2021, pp. 12 647–12 657

    2021

  8. [16]

    Filip: Fine-grained interactive language-image pre-training,

    L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu, “Filip: Fine-grained interactive language-image pre-training,” in International Conference on Learning Representations, 2021

    2021

  9. [17]

    Coarse-to-fine contrastive learning in image-text-graph space for im- proved vision-language compositionality,

    H. Singh, P. Zhang, Q. Wang, M. Wang, W. Xiong, J. Du, and Y . Chen, “Coarse-to-fine contrastive learning in image-text-graph space for im- proved vision-language compositionality,” in Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  10. [18]

    Position- enhanced visual instruction tuning for multimodal large language mod- els,

    C. Chen, R. Qin, F. Luo, X. Mi, P. Li, M. Sun, and Y . Liu, “Position- enhanced visual instruction tuning for multimodal large language mod- els,” arXiv preprint arXiv:2308.13437, 2023

    2023 arXiv

  11. [19]

    Pink: Unveiling the power of referential comprehension for multi-modal llms,

    S. Xuan, Q. Guo, M. Yang, and S. Zhang, “Pink: Unveiling the power of referential comprehension for multi-modal llms,” in Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 838–13 848

    2024

  12. [20]

    Vcoder: Versatile vision encoders for multimodal large language models,

    J. Jain, J. Yang, and H. Shi, “Vcoder: Versatile vision encoders for multimodal large language models,” in Conference on Computer Vision and Pattern Recognition, 2024, pp. 27 992–28 002

    2024

  13. [21]

    What does clip know about a red circle? visual prompt engineering for vlms,

    A. Shtedritski, C. Rupprecht, and A. Vedaldi, “What does clip know about a red circle? visual prompt engineering for vlms,” in International Conference on Computer Vision, 2023, pp. 11 987– 11 997

    2023

  14. [22]

    Fine-grained visual prompting,

    L. Yang, Y . Wang, X. Li, X. Wang, and J. Yang, “Fine-grained visual prompting,” Neural Information Processing Systems, vol. 36, 2024

    2024

  15. [23]

    Vip-llava: Making large mul- timodal models understand arbitrary visual prompts,

    M. Cai, H. Liu, S. K. Mustikovela, G. P. Meyer, Y . Chai, D. Park, and Y . J. Lee, “Vip-llava: Making large mul- timodal models understand arbitrary visual prompts,” in Conference on Computer Vision and Pattern Recognition, 2024, pp. 12 914–12 923

    2024

  16. [24]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v,

    J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao, “Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v,” arXiv preprint arXiv:2310.11441, 2023

    2023 arXiv

  17. [25]

    Regiongpt: Towards region understanding vision language model,

    Q. Guo, S. De Mello, H. Yin, W. Byeon, K. C. Cheung, Y . Yu, P. Luo, and S. Liu, “Regiongpt: Towards region understanding vision language model,” in Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 796–13 806

    2024

  18. [26]

    Prompt highlighter: Interactive control for multi-modal llms,

    Y . Zhang, S. Qian, B. Peng, S. Liu, and J. Jia, “Prompt highlighter: Interactive control for multi-modal llms,” in Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 215–13 224

    2024

  19. [27]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al., “Flamingo: a visual language model for few-shot learning,” Neural Information Processing Systems, vol. 35, pp. 23 716–23 736, 2022

    2022

  20. [28]

    Anymal: An efficient and scalable any-modality augmented language model,

    S. Moon, A. Madotto, Z. Lin, T. Nagarajan, M. Smith, S. Jain, C.- F. Yeh, P. Murugesan, P. Heidari, Y . Liu et al., “Anymal: An efficient and scalable any-modality augmented language model,” arXiv preprint arXiv:2309.16058, 2023

    2023 arXiv

  21. [29]

    Next-gpt: Any-to-any multimodal llm,

    S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua, “Next-gpt: Any-to-any multimodal llm,” arXiv preprint arXiv:2309.05519, 2023

    2023 arXiv

  22. [30]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in International Conference on Machine Learning. PMLR, 2023, pp. 19 730–19 742

    2023

  23. [31]

    Shikra: Unleashing multimodal llm’s referential dialogue magic,

    K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao, “Shikra: Unleashing multimodal llm’s referential dialogue magic,” arXiv preprint arXiv:2306.15195, 2023

    2023 arXiv

  24. [32]

    Qwen-vl: A frontier large vision-language model with versatile abilities,

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A frontier large vision-language model with versatile abilities,” arXiv preprint arXiv:2308.12966, 2023

    2023 arXiv

  25. [33]

    mplug-owl2: Revolutionizing multi- modal large language model with modality collaboration,

    Q. Ye, H. Xu, J. Ye, M. Yan, A. Hu, H. Liu, Q. Qian, J. Zhang, and F. Huang, “mplug-owl2: Revolutionizing multi- modal large language model with modality collaboration,” in Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 040–13 051

    2024

  26. [34]

    Mme: A comprehensive evaluation benchmark for multimodal large language models,

    C. Fu, P. Chen, Y . Shen, Y . Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, Y . Wu, and R. Ji, “Mme: A comprehensive evaluation benchmark for multimodal large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2306.13394

    2024 arXiv

  27. [35]

    Mmbench: Is your multi-modal model an all-around player?

    Y . Liu, H. Duan, Y . Zhang, B. Li, S. Zhang, W. Zhao, Y . Yuan, J. Wang, C. He, Z. Liu et al., “Mmbench: Is your multi-modal model an all-around player?” arXiv preprint arXiv:2307.06281, 2023

    2023 arXiv

  28. [36]

    Evaluating object hallucination in large vision-language models,

    Y . Li, Y . Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen, “Evaluating object hallucination in large vision-language models,” arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  29. [37]

    Multi-modal hallucination control by visual information grounding,

    A. Favero, L. Zancato, M. Trager, S. Choudhary, P. Perera, A. Achille, A. Swaminathan, and S. Soatto, “Multi-modal hallucination control by visual information grounding,” in Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 303–14 312

    2024

  30. [38]

    Visual cot: Unleashing chain-of-thought reasoning in multi- modal language models,

    H. Shao, S. Qian, H. Xiao, G. Song, Z. Zong, L. Wang, Y . Liu, and H. Li, “Visual cot: Unleashing chain-of-thought reasoning in multi- modal language models,” arXiv preprint arXiv:2403.16999, 2024

    2024 arXiv

  31. [39]

    Textcot: Zoom in for enhanced multimodal text-rich image understanding,

    B. Luan, H. Feng, H. Chen, Y . Wang, W. Zhou, and H. Li, “Textcot: Zoom in for enhanced multimodal text-rich image understanding,” arXiv preprint arXiv:2404.09797, 2024

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.