Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistance

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LIRA claims that segmentation and comprehension can be complementary in large multimodal models, reporting state-of-the-art on both and reduced hallucination.

desk verdict A solid LMM-segmentation architecture paper with a convincing SEFE component and a promising but under-tested ILVC idea; the missing ILVC-at-inference ablation is the one thing that should decide revision. read the letter →

arxiv 2507.06272 v3 pith:Z6ZNOE7S submitted 2025-07-08 cs.CV cs.AI

Zhang Li , Biao Yang , Qiang Liu , Shuo Zhang , Zhiyin Ma , Liang Yin , Linger Deng , Yabo Sun
show 2 more authors
Yuliang Liu Xiang Bai
This is my paper
classification cs.CVcs.AI
keywords largemultimodalmodelsreferringexpressionsegmentationhallucinationreductionsemanticfusionlocalregioncouplinggroundedconversationgenerationattributeinferencetokenlogits
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that in large multimodal models, segmentation and comprehension are complementary rather than competing, and that both improve when pixel and semantic features are fused and local regions are tied to text. It proposes LIRA, with a Semantic-Enhanced Feature Extractor (SEFE) that fuses semantic and pixel-level features for precise segmentation, and an Interleaved Local Visual Coupling (ILVC) that crops the segmented region, re-encodes it, and feeds it back into the LLM so descriptions are grounded in visible evidence. The 8B model achieves state-of-the-art on referring segmentation (78.4% on RefCOCOg val) and raised grounded-conversation CIDEr to 38.4, while adding segmentation data to training costs only 0.2% comprehension versus a 14.3% drop for the prior method. The paper also introduces the AttrEval dataset to measure how well a model can infer object attributes, and shows the token's logits encode attributes like position and color.

What carries the argument

Two mechanisms carry the argument. SEFE (Semantic-Enhanced Feature Extractor) concatenates features from a semantic encoder and a pixel encoder, with a multi-head cross-attention fusion and residual connection, aligning segmentation priors to the LLM's input space. ILVC (Interleaved Local Visual Coupling) builds interleaved sequences of global image, instruction, <seg> token, local region feature, and region description, with the local region cropped from the mask and re-encoded at 448x448; this trains the LLM to condition its output on concrete visual patches. The paper also uses the logits of the <seg> token as an interpretable attribute signal and builds the AttrEval dataset to quantify attribute-inference capability.

What would settle it

Measure LIRA on RefCOCOg with ILVC fed with predicted masks versus oracle ground-truth masks. If the predicted-mask condition is no better than removing ILVC entirely, the distribution shift is the load-bearing failure; if oracle masks help substantially, the mechanism works only as well as the upfront segmentation does.

Watch

Extended reading notes

Core claim

The central claim is that segmentation and comprehension are not a zero-sum trade-off in large multimodal models, provided local visual features are explicitly coupled to text. LIRA trains with a mask-region-text triplet: the region is extracted from the ground-truth mask, re-encoded, and re-entered into the LLM before it generates the region's description, so the model learns to ground words in visible patches. With this, LIRA reports state-of-the-art results on RefCOCO, RefCOCO+, and RefCOCOg, and the co-training ablation shows only a 0.2% drop in comprehension accuracy when segmentation data is added, in contrast to the 14.3% drop reported for the prior best model. Additionally, the paper introduces AttrEval and finds that the highest-logit tokens in the <seg> embedding correspond to correct object attributes, tying segmentation precision to the semantic content the model already carries.

Load-bearing premise

ILVC is trained using regions cropped from ground-truth masks, but at inference the regions are cropped from the model's own predicted masks, and those imprecise crops feed the LLM features it never saw during training.

Editorial extensions

If this is right

  • Adding segmentation data to instruction tuning does not have to degrade comprehension; LIRA's drop is 0.2% on five comprehension benchmarks, against the prior model's 14.3%.
  • ILVC lowers object hallucination on ChairS, POPE, and TinyLVLM across both 1.8B and 7B backbones.
  • Grounded conversation generation improves (CIDEr 38.4 on validation), so the mechanism transfers to mask-linked captioning.
  • The <seg> token's top-logit tokens encode object position and color, making part of the segmentation decision interpretable.
  • LIRA improves zero-shot generalized referring segmentation on gRefCOCO without being trained on that benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same <seg>-logit readout used in AttrEval could be turned into an uncertainty signal: when the top attribute logit is low, the model could refrain from committing to a mask or ask for clarification.
  • The train-GT-mask / infer-predicted-mask gap is testable directly: evaluating LIRA on RefCOCOg with oracle masks versus predicted masks would bound how much of ILVC's benefit survives imperfect segmentation.
  • The two-prompt with/without-ILVC control on COCO instance segmentation could become a standard robustness check for any mask-conditioned LMM in multi-object scenes.
  • Re-feeding cropped regions into the LLM is a generic mechanism that could extend to detection, referring grounding, or editing tasks that need region-conditioned text.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes LIRA, a framework that augments InternVL2-based LMMs with two components: SEFE, which fuses a semantic encoder and an OMG-Seg pixel encoder for segmentation, and ILVC, which interleaves local image-region features with textual descriptions using <seg> tokens. The authors report state-of-the-art referring-expression-segmentation results on RefCOCO/RefCOCO+/RefCOCOg, improved grounded conversation generation CIDEr, and claim that co-training with segmentation data causes only a 0.2% drop in comprehension accuracy compared with OMG-LLaVA's 14.3% drop. They also introduce an AttrEval dataset to support the claim that the <seg> token's logits encode object attributes such as color and position.

Significance. If the central claims hold, LIRA would be a useful step toward unified LMMs that segment and comprehend without catastrophic forgetting. The paper's strengths are its broad evaluation (eight comprehension benchmarks, three referring-segmentation splits, gRefCOCO zero-shot, GCG, and three hallucination metrics), its architecture-level ablations of SEFE (Table 4) and ILVC (Table 6), and its candid limitation paragraph acknowledging the low AttrEval accuracy. However, the signature claim that ILVC reduces hallucination while preserving segmentation is currently supported mainly by training-time ablations; the only inference-time controlled experiment (Appendix F) does not support the claim. The introduced AttrEval benchmark is also built from the same RefCOCO data used in training, so the <seg>-token semantic-correlation evidence is vulnerable to circularity. These issues must be resolved before the state-of-the-art claim can be accepted.

major comments (3)
  1. [Sec. 3.3, Appendix F] Section 3.3 trains ILVC with local crops extracted from ground-truth masks but, at inference, crops are taken from the model's own predicted masks. Appendix F, the only controlled comparison under this mismatch, reports that using ILVC at inference lowers COCO instance-segmentation mIoU from 60.6 to 58.9, while the no-ILVC baseline is 60.0; this does not support the claim that ILVC preserves segmentation while reducing hallucination. The paper does not state whether the headline numbers in Tables 1, 2, and 6 use the ILVC prompt or the no-ILVC prompt, nor does it provide an inference-time ILVC on/off ablation on RefCOCO or on the hallucination benchmarks. Please report those numbers and specify the inference prompt; without them, the central complementarity claim is unverified.
  2. [Sec. 5, Appendix D] The evidence for the paper's claimed positive correlation between segmentation accuracy and <seg>-token semantics rests on AttrEval (Section 5 and Appendix D). AttrEval is constructed from RefCOCO, which is part of the instruction-tuning data (Table 8), so the measured correlation may reflect training-data memorization rather than a general property. Please evaluate the same logits-to-attribute mapping on a held-out or external benchmark, or at least verify that no AttrEval instance overlaps with the training set.
  3. [Tables 1-2, Sec. 4.3] Tables 1 and 2 report differences of 0.5-2 points as state-of-the-art without error bars or significance tests; on two comprehension benchmarks, MMB-en and MMB-cn, LIRA-8B is actually below the comprehension-only InternVL2-8B baseline (81.1 vs 81.7 and 80.5 vs 81.2). The abstract and Section 4.3 claim state-of-the-art performance in both segmentation and comprehension, which is stronger than the data support. Provide variance estimates and qualify the comparative claims, and note that some baselines (e.g., PSALM with 100 queries, GLaMM with GranD pretraining) use different inference or training budgets.
minor comments (4)
  1. [Table 8, Appendix D] In Table 8, 'CharQA' should be 'ChartQA' and 'AttriEval' should be 'AttrEval' for consistency; Section 4.3 and Appendix D use both spellings.
  2. [Appendix F, Sec. 4.1] The description of the two prompts that control ILVC in Appendix F is omitted; please include them in the implementation details for reproducibility.
  3. [Fig. 5, Appendix C] Figure 5 and Appendix C show qualitative hallucination comparisons but no quantitative captioning metric; consider adding CHAIR or a similar metric to support the hallucination-reduction claim.
  4. [Sec. 5] The 'Limitation' paragraph in Section 5 reports Acc1=25.7% on AttrEval; this is low and should be discussed in relation to the claim that the <seg> token logits encode object attributes.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: the <seg>-semantics finding is validated on AttrEval, which is constructed from the same RefCOCO data used to train LIRA; other central claims rest on external benchmarks.

  1. fitted input called prediction [Appendix D (AttrEval construction); Sec. 4.2 and Table 8 (training data)]
    "We choose the RefCOCO dataset as the basis for constructing AttrEval. The process of building the dataset is as follows: We predefined a set of attribute categories, including category, location, and color. From multiple descriptions of the same object in RefCOCO, we extract unique attributes of color, location, and category. Using these extracted attributes, we construct the VQA and RefSeg tasks based on different descriptions of the same object."

    LIRA is trained on RefCOCO: Sec. 4.2 says it uses '374k from segmentation datasets [39, 43, 68]' and Table 8 lists 51k RefCOCO samples. AttrEval is built from RefCOCO by recombining descriptions of the same objects. Therefore the reported Acc1/Acc3/VQA results and the claimed 'positive correlation' between segmentation precision and <seg>-token semantics are measured on a re-packaged version of the model's training distribution, not on an independent benchmark. The attribute-inference 'prediction' is thus statistically tied to the fitted training data, making the AttrEval-based finding self-referential rather than external confirmation.

full rationale

LIRA's main performance claims (SEFE/ILVC gains on RefCOCO/RefCOCO+/RefCOCOg, comprehension benchmarks, GCG, hallucination ablations) are tested on standard external benchmarks and are not derived from the method's own assumptions by construction. The train/inference mismatch in ILVC (GT-mask crops at training vs predicted-mask crops at inference; Appendix F reports 58.9 vs 60.6 mIoU with ILVC at inference on COCO) is a robustness limitation, not a circularity. Self-citations (Monkey data, PSALM comparison) are not load-bearing. The one self-referential element is the AttrEval-based finding that <seg> semantics correlate with segmentation precision: AttrEval is constructed from RefCOCO, which is also LIRA's training source, so the quantitative 'semantic inferring ability' numbers are a repackaged view of the training distribution. This supports a moderate circularity score but does not invalidate the externally benchmarked central claims.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim is an empirical architecture claim, so the ledger is dominated by domain assumptions about feature fusion, training/inference consistency, and the validity of the new AttrEval benchmark. No free parameters are fit to data in the sense of an equation fit; alpha, LoRA rank, crop resolution, and ILVC ratio are manually chosen hyperparameters that affect the reported results. No new physical or conceptual entities beyond the AttrEval dataset are introduced.

free parameters (4)
  • mask loss weight alpha = not reported
    Balances text loss and segmentation loss in Eq. 7; the co-training result that comprehension degrades only 0.2% depends on this balance, but the value is not given.
  • LoRA rank = 128 for 2B, 256 for 8B
    Capacity of LLM fine-tuning; chosen per model size in Section 4.1.
  • local crop resolution = 448x448
    Local regions are resized to 448x448 before re-encoding (Section 3.3); segmentation and captioning results could depend on this choice.
  • ILVC data ratio in main training = not specified; 50% only in COCO instance segmentation experiment (Appendix F)
    The fraction of segmentation data trained with interleaved ILVC supervision is not reported for the main RefCOCO/GranDf runs; this ratio directly controls how strongly the model learns the coupling.
assumptions (5)
  • domain assumption Semantic and pixel encoders provide complementary features that can be aligned by MLP projection and cross-attention without retraining the encoders.
    Invoked in Section 3.2, Eqs. 1-3; SEFE freezes both encoders and trains only MLPs and MHCA. If the two feature spaces cannot be fused this way, the segmentation improvement would not materialize.
  • domain assumption Ground-truth mask cropping during ILVC training transfers to predicted masks at inference.
    Section 3.3 uses GT masks for local region extraction during training but predicted masks during inference; Appendix F acknowledges error accumulation risk and only partially mitigates it with prompt-controlled ILVC usage.
  • ad hoc to paper The top logits of the <seg> token encode object attributes such as color and position.
    Section 5 and Figures 2 and 7 present this as the motivation for AttrEval; the claim is the explanatory basis for the paper's analysis but is only demonstrated on selected examples and on AttrEval, which is derived from the training distribution.
  • domain assumption RefCOCO referring expressions provide reliable ground-truth attribute labels when multiple descriptions of the same object are compared.
    Appendix D constructs AttrEval by extracting unique attributes from multiple descriptions of the same object in RefCOCO; if the parser or descriptions are noisy, the VQA ground truth is noisy.
  • ad hoc to paper The interleaved sequence format with special tokens <p> and </p> preserves local-region-text correspondence in a way the LLM can learn from next-token prediction.
    Section 3.3, Eq. 4 defines the interleaved sequence S; no theoretical guarantee or controlled comparison against alternative ordering is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistance." pith.science (2026). https://pith.science/paper/Z6ZNOE7S

@misc{pith2026250706272,
  author       = {Pith},
  title        = {Pith review of: LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z6ZNOE7S}},
  note         = {Machine review of arXiv:2507.06272}
}
read the original abstract

While large multi-modal models (LMMs) demonstrate promising capabilities in segmentation and comprehension, they still struggle with two limitations: inaccurate segmentation and hallucinated comprehension. These challenges stem primarily from constraints in weak visual comprehension and a lack of fine-grained perception. To alleviate these limitations, we propose LIRA, a framework that capitalizes on the complementary relationship between visual comprehension and segmentation via two key components: (1) Semantic-Enhanced Feature Extractor (SEFE) improves object attribute inference by fusing semantic and pixel-level features, leading to more accurate segmentation; (2) Interleaved Local Visual Coupling (ILVC) autoregressively generates local descriptions after extracting local features based on segmentation masks, offering fine-grained supervision to mitigate hallucinations. Furthermore, we find that the precision of object segmentation is positively correlated with the latent related semantics of the <seg> token. To quantify this relationship and the model's potential semantic inferring ability, we introduce the Attributes Evaluation (AttrEval) dataset. Our experiments show that LIRA achieves state-of-the-art performance in both segmentation and comprehension tasks. Code will be available at https://github.com/echo840/LIRA.

Figures

Figures reproduced from arXiv: 2507.06272 by the authors.

Figure 1
Figure 1. The performance of LIRA on comprehension and seg [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. “Please segment the red bus closest to the white car.” is [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The overall architecture of LIRA. Global and local features are extracted using SEFE, with ILVC introduced to align [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Different training paradigms: (a) refines only by refeed [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Comparing the hallucination of wo/w ILVC, with hallu￾cination content highlighted in red. improve segmentation accuracy. Effectiveness of ILVC. Building on the implementation of SEFE, we conducted ablation studies to explore the ef￾fectiveness of further integrating IL…
Figure 6
Figure 6. Figure 6: Comparison with OMG-LLaVA in image caption, referring expression segmentation, and grounded conversation generation. The [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Visualization of the logits values for the [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The visualization results of the VQA and Image Caption tasks. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: The visualization results of the referring expression segmentation (RefSeg) and grounded conversation generation (GCG) tasks. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Comparison of Hallucinations in Image Caption [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Word Cloud of the AttriEval Dataset. User: Is front guy in black? Please answer yes or no. LIRA: yes User: Is guy wearing black shirt on front? Please answer yes or no. LIRA: yes User: Is car next to cab on right? Please answer yes or no. LIRA: no User: Is car next to…
Figure 12
Figure 12. Figure 12: Visualization Results of LIRA on the AttriEval Dataset. The bar chart presents the top five attribute names with the highest [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    The survey formalizes MLLM perception as a unified vision-language capability and traces its evolution via a new five-stage taxonomy while outlining future challenges.

Reference graph

Works this paper leans on

75 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  2. [2]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3, 2023. 2, 6

    arXiv 2023

  3. [3]

    Internlm2 technical report, 2024

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiax- ing Li, Jingwen Li, Linyang Li,...

    work page 2024

  4. [4]

    Shikra: Unleashing multi- modal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 3

    arXiv 2023

  5. [5]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 5

    arXiv 2023

  6. [6]

    Pali-3 vision language models: Smaller, faster, stronger

    Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul V oigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199, 2023. 2

    arXiv

  7. [7]

    Sam4mllm: Enhance multi- modal large language model for referring expression seg- mentation

    Yi-Chia Chen, Wei-Hua Li, Cheng Sun, Yu-Chiang Frank Wang, and Chu-Song Chen. Sam4mllm: Enhance multi- modal large language model for referring expression seg- mentation. In European Conference on Computer Vision , pages 323–340. Springer, 2025. 3

    work page 2025

  8. [8]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 3, 6

    arXiv 2024

Show all 75 references
  1. [9]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 1

    2023 arXiv

  2. [10]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904–691...

    2017

  3. [11]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  4. [12]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281– 1...

    2024

  5. [13]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Represen- tations, ICLR 2022, Virtual Event, April 25-29, 2022. ...

    2022

  6. [14]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024. 6

    2024 arXiv

  7. [15]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 5, 6, 13

    2019

  8. [16]

    Dvqa: Understanding data visualizations via ques- tion answering

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via ques- tion answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656,

  9. [17]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pages 235–

    2016

  10. [18]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 3

    2023

  11. [19]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9579–9589, 2024. 1, 3, 4, 6

    2024

  12. [20]

    Text4seg: Reimagining image segmentation as text genera- tion

    Mengcheng Lan, Chaofeng Chen, Yue Zhou, Jiaxing Xu, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Text4seg: Reimagining image segmentation as text genera- tion. arXiv preprint arXiv:2410.09855, 2024. 3

    2024 arXiv

  13. [21]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2

    2023

  14. [22]

    Omg-seg: Is one model good enough for all segmentation? arXiv preprint arXiv:2401.10229, 2024

    Xiangtai Li, Haobo Yuan, Wei Li, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, and Chen Change Loy. Omg-seg: Is one model good enough for all segmentation? arXiv preprint arXiv:2401.10229, 2024. 5

    2024 arXiv

  15. [23]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empir- ical Methods in Natural Language Processing, 2023. 6

    2023

  16. [24]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814,

    arXiv

  17. [25]

    Mon- key: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2024

  18. [26]

    Gres: Gener- alized referring expression segmentation

    Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Gener- alized referring expression segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23592–23601, 2023. 5, 6

    2023

  19. [27]

    Gres: Gen- eralized referring expression segmentation

    Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Gen- eralized referring expression segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 23592–23601, 2023. 6

    2023

  20. [28]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 2, 6

    2024

  21. [29]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2, 5, 13

    2024

  22. [30]

    Llava-plus: Learning to use tools for creating multi- modal agents

    Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multi- modal agents. arXiv preprint arXiv:2311.05437, 2023. 3

    2023 arXiv

  23. [31]

    Textmonkey: An ocr-free large multimodal model for understanding document

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473, 2024. 2

    2024 arXiv

  24. [32]

    Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer, 2025. 6

    2025

  25. [33]

    Interngpt: Solving vision-centric tasks by interacting with chatgpt beyond language

    Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang, Shoufa Chen, Qinglong Zhang, Zeqiang Lai, Yang Yang, Qingyun Li, et al. Interngpt: Solving vision-centric tasks by interacting with chatgpt beyond language. arXiv preprint arXiv:2305.05662, 2023. 3

    2023 arXiv

  26. [34]

    Chain-of-spot: Interactive reasoning improves large vision-language models

    Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Ji- wen Lu. Chain-of-spot: Interactive reasoning improves large vision-language models. arXiv preprint arXiv:2403.12966,

    arXiv

  27. [35]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems ...

    2022

  28. [36]

    Feast your eyes: Mixture-of- resolution adaptation for multimodal large language models

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xi- aoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of- resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003, 2024. 2

    2024 arXiv

  29. [37]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 5, 6, 13

    2019

  30. [38]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022. 5, 13

    2022 arXiv

  31. [39]

    Mod- eling context between objects for referring expression under- standing

    Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Mod- eling context between objects for referring expression under- standing. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pages 792–807. Springer,

    2016

  32. [40]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. 3, 5, 6

    2023 arXiv

  33. [41]

    Cogcom: Train large vision-language models diving into details through chain of manipulations

    Ji Qi, Ming Ding, Weihan Wang, Yushi Bai, Qingsong Lv, Wenyi Hong, Bin Xu, Lei Hou, Juanzi Li, Yuxiao Dong, et al. Cogcom: Train large vision-language models diving into details through chain of manipulations. arXiv preprint arXiv:2402.04236, 2024. 2

    2024 arXiv

  34. [42]

    Reasoning to attend: Try to understand how¡ seg¿ token works

    Rui Qian, Xin Yin, and Dejing Dou. Reasoning to attend: Try to understand how¡ seg¿ token works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. 3

    2025

  35. [43]

    10 Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. 10 Glamm: Pixel grounding large multimodal model. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pat...

    2024

  36. [44]

    Pixellm: Pixel reasoning with large multimodal model

    Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26374–26383, 2024. 3, 6

    2024

  37. [45]

    Object hallucination in image cap- tioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035–4045,

    2018

  38. [46]

    A-okvqa: A benchmark for visual question answering using world knowl- edge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowl- edge. In European conference on computer vision , pages 146–162. Springer, 2022. 5, 13

    2022

  39. [47]

    Tinylvlm-ehub: Towards com- prehensive and efficient evaluation for large vision-language models, 2024

    Wenqi Shao, Meng Lei, Yutao Hu, Peng Gao, Kaipeng Zhang, Fanqing Meng, Peng Xu, Siyuan Huang, Hongsheng Li, Yu Qiao, and Ping Luo. Tinylvlm-ehub: Towards com- prehensive and efficient evaluation for large vision-language models, 2024. 7

    2024

  40. [48]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 13

    2019

  41. [49]

    Cambrian-1: A fully open, vision-centric explo- ration of multimodal llms

    Shengbang Tong, Ellis L Brown II, Penghao Wu, Sanghyun Woo, ADITHY A JAIRAM IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric explo- ration of multimodal llms. In The Thirty-eighth Annual Con- fer...

    2024

  42. [50]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 3, 18

    2024 arXiv

  43. [51]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023. 3

    2023 arXiv

  44. [52]

    Visionllm: Large language model is also an open- ended decoder for vision-centric tasks

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open- ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024. 3

    2024

  45. [53]

    Hierar- chical open-vocabulary universal image segmentation

    Xudong Wang, Shufan Li, Konstantinos Kallidromitis, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. Hierar- chical open-vocabulary universal image segmentation. Ad- vances in Neural Information Processing Systems, 36, 2024. 3

    2024

  46. [54]

    Segllm: Multi-round reasoning segmenta- tion

    XuDong Wang, Shaolun Zhang, Shufan Li, Konstantinos Kallidromitis, Kehan Li, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. Segllm: Multi-round reasoning segmenta- tion. arXiv preprint arXiv:2410.18923, 2024. 3

    2024 arXiv

  47. [55]

    Lasagna: Language-based segmentation assistant for complex queries

    Cong Wei, Haoxian Tan, Yujie Zhong, Yujiu Yang, and Lin Ma. Lasagna: Language-based segmentation assistant for complex queries. arXiv preprint arXiv:2404.08506, 2024. 6

    2024 arXiv

  48. [56]

    General object foundation model for images and videos at scale

    Junfeng Wu, Yi Jiang, Qihao Liu, Zehuan Yuan, Xiang Bai, and Song Bai. General object foundation model for images and videos at scale. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 3783–3795, 2024. 6

    2024

  49. [57]

    Visionllm v2: An end-to-end general- ist multimodal large language model for hundreds of vision- language tasks

    Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Wenhai Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end general- ist multimodal large language model for hundreds of vision- language tasks. arXiv preprint arXiv:2406.08394, 2024. 3

    2024 arXiv

  50. [58]

    V*: Guided visual search as a core mechanism in multimodal llms

    Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084–13094, 2024. 2

    2024

  51. [59]

    Gsva: Generalized segmentation via multimodal large language models

    Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3858–3869, 2024. 6

    2024

  52. [60]

    Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images

    Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403.11703, 2024. 2

    2024 arXiv

  53. [61]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jian- wei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jin...

    2024

  54. [62]

    An improved baseline for reasoning segmentation with large language model

    Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.17240, 2023. 3

    2023 arXiv

  55. [63]

    Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. In The 2023 Conference on Empirical Methods in Natu...

    2023

  56. [64]

    mplug-owl3: Towards long image-sequence understanding 11 in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding 11 in multi-modal large language models. arXiv preprint arXiv:2408.04840, 2024. 3, 6

    2024 arXiv

  57. [65]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. 2

    2023 arXiv

  58. [66]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , ...

    2024

  59. [67]

    Ferret: Refer and ground anything anywhere at any granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In The Twelfth International Conference on Learning Representations, 2024. 3

    2024

  60. [68]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer, 20...

    2016

  61. [69]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output. arXiv preprint arXiv:2407.03320, 2024. 3

    2024 arXiv

  62. [70]

    Gpt4roi: Instruction tuning large language model on region- of-interest

    Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region- of-interest. arXiv preprint arXiv:2307.03601, 2023. 3

    2023 arXiv

  63. [71]

    Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding

    Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Y AN Shuicheng. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. In The Thirty-eighth An- nual Conference on Neural Information Processing Systems,

  64. [72]

    Psalm: Pixelwise segmentation with large multi-modal model

    Zheng Zhang, Yeyao Ma, Enming Zhang, and Xiang Bai. Psalm: Pixelwise segmentation with large multi-modal model. In European Conference on Computer Vision, pages 74–91. Springer, 2025. 3, 6

    2025

  65. [73]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2

    2023 arXiv

  66. [74]

    When the provided information is insufficient, respond with ‘Unanswerable,’

    Yichen Zhu, Minjie Zhu, Ning Liu, Zhiyuan Xu, and Yaxin Peng. Llava-phi: Efficient multi-modal assistant with small language model. In Proceedings of the 1st International Workshop on Efficient Multimedia Computing under Limited, pages 18–22, 2024. 6 12 A. Summary of the Instr...

    2024

  67. [251]

    Springer, 2016. 5, 13 9

    2016

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.