Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MMGrounded-PostAlign claims that forcing an MLLM to ground answers in a visual localization and a selective rationale reduces hallucination, lifting POPE accuracy from 83.3 to 86.6 at 7B scale.

desk verdict Useful integrative method for grounding-based hallucination control, but the undisclosed training data makes the headline gains unverifiable as-is. read the letter →

arxiv 2506.17901 v1 pith:R24PPKZ5 submitted 2025-06-22 cs.CV

Yixuan Wu , Yang Zhang , Jian Wu , Philip Torr , Jindong Gu This is my paper
classification cs.CV
keywords multimodallargelanguagemodelshallucinationmitigationvisualgroundingtextualnegativerejectionselectivereasoningpost-trainingalignmentfine-grainedunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that a large part of multimodal hallucination is driven by linguistic priors, and that a post-training alignment layer can correct it. The proposed framework, MMGrounded-PostAlign, makes the model produce a visual grounding token that localizes the referred object, a special rejection token when no such object exists, and a textual rationale only when the query needs one. On hallucination benchmarks this raises POPE accuracy from 83.3 to 86.6 at 7B scale and from 85.4 to 88.9 at 13B scale, while VQAv2 and other reasoning scores stay flat or rise. If the claim holds, hallucination suppression does not require a new architecture or losing general reasoning ability; it can be added on top of an existing model with inexpensive LoRA fine-tuning.

What carries the argument

The load-bearing machinery is the grounding-token interface: the MLLM is trained to emit <LOC> for the referred object, and its last-layer embedding is fed through an MLP into a multi-task decoder built on a frozen SAM encoder to produce a mask and bounding box. When no referent exists, <REJ> replaces <LOC> and bypasses decoding with an empty mask, under a dedicated negative-rejection loss; the query-complexity tokens <SIMPLE>/<COMPLEX> similarly route the textual branch under a selective-reasoning loss. These four structured tokens turn grounding into a differentiable training signal that is then added, via LoRA, to the standard language modeling loss. The framework does most of its claimed work through these token-level routing decisions, not through a new pretraining objective.

What would settle it

Retrain the identical pipeline with all POPE, HaloQuest, and MME question-image pairs removed from the training mixture; if the POPE and HaloQuest gains shrink to near baseline, the hallucination suppression comes from benchmark contamination rather than the grounding mechanism. A second check is to remove the <REJ> branch alone while keeping every other component and measuring HaloQuest's False Premise category, since the paper attributes most of that category's improvement to rejection.

Watch

Extended reading notes

Core claim

The central claim is that anchoring the answer-generation process in explicit multimodal grounding corrects an MLLM's over-reliance on linguistic priors. During decoding, the model first outputs a grounding token; its embedding is passed to a segmentation and bounding box decoder to locate the referent, and when the referent is absent the model must instead emit <REJ>, which is assigned an empty mask and box. A parallel textual-grounding branch decides, via a learned <SIMPLE>/<COMPLEX> classification, whether to generate a rationale before the final answer. The paper reports that these two mechanisms together suppress object hallucinations on POPE and HaloQuest, sharpen attention on image regions, and preserve or slightly improve performance on MME, MMBench, and VQAv2.

Load-bearing premise

The training data recipe — how absent referents are turned into <REJ> negatives and how queries are labeled <SIMPLE> or <COMPLEX> — is not disclosed, so the reported gains assume these labels and negatives are created correctly and do not overlap the test benchmarks.

Editorial extensions

If this is right

  • Hallucination suppression can be layered onto existing MLLMs via LoRA, without replacing the backbone or sacrificing general VQA capability.
  • When a query names an object that is not in the image, models trained with <REJ> learn to say that the object does not exist rather than predicting a co-occurring object.
  • Generating a rationale is not always beneficial; routing simple queries straight to the answer avoids overthinking and keeps the whole process in a single inference pass.
  • The same grounding interface yields competitive referring-expression comprehension and segmentation, suggesting that grounding can be a byproduct of hallucination correction rather than a separate training goal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the <REJ> mechanism is trained on absent referents; a natural test is whether it transfers to attribute or spatial hallucinations, such as a present object with the wrong color, which the paper's loss does not explicitly target.
  • Beyond the paper: the measured 89.2% overlap of hallucinated tokens between image-present and image-absent decoding could be used as a cheap synthetic-hallucination probe before running full benchmarks.
  • Beyond the paper: selective reasoning suggests a capacity and compute tradeoff curve; measuring answer quality against rationale length across difficulty buckets would tell whether the learned <SIMPLE>/<COMPLEX> router is well calibrated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes MMGrounded-PostAlign, a post-alignment framework for multimodal large language models that couples a visual grounding module (with <LOC> and <REJ> tokens) to a textual grounding mechanism with selective reasoning (<SIMPLE>/<COMPLEX>). The method is trained with LoRA on top of LLaVA-1.5-7B/13B and a frozen ViT-H SAM encoder, with losses for negative rejection, selective reasoning, grounding, and language modeling. The reported experiments cover hallucination benchmarks (HaloQuest, POPE), general VQA/reasoning benchmarks (VQAv2, MMBench, MME), and grounding benchmarks (RefCOCO, ReasonSeg), claiming improved hallucination suppression while preserving general capabilities. The main evidence is Table 2 (e.g., POPE adversarial 78.2→82.3 at 7B and 79.2→85.6 at 13B) and Table 1's large HaloQuest gains when the <REJ> mechanism is enabled.

Significance. If the reported improvements are genuine and reproducible, the framework is a useful contribution: it makes explicit visual grounding and negative rejection into lightweight post-alignment training for MLLMs, and the selective-reasoning ablation in Table 3 is a sensible design comparison. The paper also provides a transparent list of loss terms in Section 3.4, which is a strength. However, the central evaluation currently depends on an undisclosed training dataset and an unspecified inference/parsing protocol, so the headline claims cannot be separated from train/evaluation overlap or formatting artifacts. The contribution is therefore potentially significant but not yet verifiable from the manuscript as written.

major comments (5)
  1. [Section 4.1] The training data is described only as 'a diverse multimodal training dataset' with <SIMPLE>/<COMPLEX> labels and <REJ> negative samples. The paper does not state the image corpus, the number of training instances, the procedure for constructing negative referents (how an absent object is chosen, whether captions/objects are sampled, what ratio of negatives is used), the rule for assigning SIMPLE/COMPLEX labels, or whether the evaluation benchmarks' test images were excluded from training. Because POPE and RefCOCO both use MSCOCO images, and HaloQuest contains false-premise questions, the large gains in Tables 1 and 2 (e.g., HaloQuest False Premise 9.9→33.2 with <REJ>; POPE adversarial 78.2→82.3 at 7B) could be inflated by image overlap or template overlap. Please disclose the training data source, the negative-sample construction procedure, and an explicit train/evaluation disjointness statement.
  2. [Section 4.2, Tables 1 and 2] The baseline definition is inconsistent across tables. In Table 2, 'Baseline' is defined as the proposed framework with the visual grounding module removed while retaining selective reasoning, but Table 1's first row is described as 'the baseline where the visual grounding module is entirely removed' without stating whether the textual grounding/selective reasoning components are present. This makes it impossible to attribute the improvements to the visual grounding module, the negative rejection mechanism, or the selective reasoning mechanism. Please use one coherent baseline and clearly state which components each row includes.
  3. [Sections 3.1 and 4.2] The inference protocol is not specified: the paper does not explain how the structured output tokens (<LOC>, <REJ>, <SIMPLE>, <COMPLEX>, rationale, final answer) are parsed into benchmark answers for POPE's yes/no questions, VQAv2's short answers, HaloQuest's free-form answers, or MMBench/MME's multiple-choice prompts. Without this detail, the reported improvements could reflect a better match between the generated format and the benchmark's answer parser rather than improved grounding. Please describe the exact answer extraction and scoring procedure used for each benchmark.
  4. [Tables 1–3] No error bars or significance tests are reported for any of the main results. Several comparisons in Table 2 are within about one point (e.g., VQAv2 79.1→79.9 at 13B, MME 1520.3→1517.4), so without multiple seeds or a statistical test it is unclear which differences are meaningful. Please add variance information or at least state whether results are averaged over multiple runs.
  5. [Section 3.3 and Eq. (4)] The selective reasoning mechanism depends on SIMPLE/COMPLEX training labels, but the labeling rule is never defined beyond two examples. The paper also does not state how y_reason in Eq. (4) is obtained (manual annotation, automatic heuristic, or model-generated) or how the proportion of simple versus complex queries is chosen. This is load-bearing for Table 3's conclusion that selective reasoning is optimal, because the method's benefit could simply reflect the specific labeling distribution used. Please specify the labeling criteria and distribution.
minor comments (5)
  1. [Section 4.1] Implementation details omit several hyperparameters needed for reproducibility: the LoRA rank and alpha, the values of λ1 and λ2 in Eq. (5), the total training steps, and the number of negative samples per batch.
  2. [Reference list, entry [57]] The MME benchmark is cited through an unrelated survey reference ('A survey of multimodel large language models'); the correct MME paper should be cited.
  3. [Section 4.3 and Figure 4] The attention analysis is described only qualitatively. The paper reports an 89.2% hallucinated-token overlap after removing the image, but does not define how hallucinated tokens are identified in this analysis or how the 'without image' condition is constructed (e.g., blank image, masked image, or text-only prompt). Please clarify the protocol.
  4. [Section 4.2, Finding 5] The phrase 'zero-shot grounding capabilities' is misleading because the model is fine-tuned on a multimodal grounding training set; 'unseen-category grounding' or 'grounding without task-specific fine-tuning' would be more precise.
  5. [Abstract and Section 3] There are typos such as 'visiual understanding' and the duplicated sentence 'In this section, we introduce MMGrounded-PostAlign...' at the start of Section 3. Please proofread.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the reported gains come from an external training/evaluation protocol, not from equations that presuppose the outcome.

full rationale

The paper is an empirical post-training framework. Its derivation chain is: define a structured output A = {V, T, F}; introduce a negative rejection loss L_rej and a selective reasoning loss L_reason; fine-tune LLaVA with LoRA plus a grounding decoder; and evaluate on external benchmarks (POPE, HaloQuest, VQAv2, MME, MMBench, RefCOCO, ReasonSeg). None of these steps defines the target metric in terms of the training signal. L_rej and L_reason are standard cross-entropy losses whose targets are data annotations, not the benchmark scores; the benchmark scores in Tables 1-4 are measured after training, not read off from the loss definitions. The self-citations ([14] and [22]) appear only in related-work context and are not load-bearing for any claim. There is no imported uniqueness theorem, no ansatz smuggled in via citation, and no renaming of a known result as a new prediction. The main verifiability concern is that Section 4.1 ('Training Data Formulation') does not disclose the image corpus, negative-sample construction, or whether POPE/HaloQuest images or templates were excluded from training; that is a contamination/reproducibility risk, but it is not a circularity of the paper's own equations, and the paper provides no quote-level evidence of train/eval overlap. Under the hard rule that circularity must be exhibited by a specific reduction, no such step exists, so the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 3 invented entities

The framework introduces three types of special tokens as internal model vocabulary. They are not external physical entities, but they are invented additions whose functionality is demonstrated only through the paper's own experiments. The main hidden assumptions are about training data fairness and the sufficiency of token embeddings as prompts.

free parameters (4)
  • lambda_1 (L_rej weight) = not specified
    In Eq. 5 the total loss is a weighted sum; lambda_1 must be chosen by hand and affects the balance between hallucination rejection and other objectives, but no value is given.
  • lambda_2 (L_reason weight) = not specified
    Same as above for the selective reasoning loss in Eq. 4.
  • LoRA rank = not specified
    The rank controls the capacity of the adaptation and is a standard hyperparameter; not reported.
  • SIMPLE/COMPLEX labeling rule = not quantified
    The criterion for labeling a query as complex is described by example only, leaving a hand-tuned judgment call that affects training data.
assumptions (4)
  • domain assumption Autoregressive generation of the grounding token, rationale, and final answer makes the final answer causally conditioned on visual evidence.
    Sec 3.1 states the outputs act as 'implicit constraints', but the paper does not prove that the grounding tokens actually force the model to rely on image evidence.
  • domain assumption The last-layer embedding of <LOC> is a sufficient prompt for the SAM mask decoder.
    Sec 3.1 feeds this embedding through an MLP into the multi-task decoder; no analysis of information loss is given.
  • ad hoc to paper The undisclosed training set is representative of the evaluation benchmarks and does not contain their test data.
    Sec 4.1 describes dataset construction only vaguely, so the fairness of the evaluation cannot be verified.
  • domain assumption POPE, HaloQuest, VQAv2, MME, and MMBench are valid measures of hallucination and general visual understanding.
    The paper relies on these benchmarks without discussing their limitations or the reliability of their metrics.
invented entities (3)
  • <LOC> visual grounding token independent evidence
    purpose: Instructs the multi-task decoder to segment and localize the object referred to by the question
    Table 4 reports competitive RefCOCO and ReasonSeg results, suggesting the token carries localization information.
  • <REJ> rejection token independent evidence
    purpose: Marks that the referred object does not exist, triggering empty mask/box and suppressing hallucinations
    Table 1 shows large HaloQuest gains when this token is included.
  • <SIMPLE>/<COMPLEX> tokens independent evidence
    purpose: Trigger selective generation of textual rationales depending on query complexity
    Table 3 shows selective reasoning outperforms always-reason and interleaved-reason baselines.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs." pith.science (2026). https://pith.science/paper/R24PPKZ5

@misc{pith2026250617901,
  author       = {Pith},
  title        = {Pith review of: PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R24PPKZ5}},
  note         = {Machine review of arXiv:2506.17901}
}
read the original abstract

Multimodal Large Language Models (MLLMs) excel in vision-language tasks, such as image captioning and visual question answering. However, they often suffer from over-reliance on spurious correlations, primarily due to linguistic priors that distract the model from leveraging actual visual information. To address these issues, we introduce MMGrounded-PostAlign, a post-multimodal alignment framework designed to enhance the visual understanding capabilities and mitigate the hallucinations of MLLMs. Our framework incorporates a multimodal grounding module for both visual grounding, which identifies the referred object in the image, and textual grounding, which generates the rationale for the final answer, ensuring that outputs are anchored in both visual and textual evidence. To mitigate the hallucinations, we introduce a negative rejection mechanism in the visual grounding module to distinguish grounded entities from non-existent objects influenced by linguistic biases. On the textual grounding side, we propose a selective reasoning mechanism that adjusts the model's reasoning strategy based on query complexity. Extensive evaluations are conducted on benchmarks such as POPE, HaloQuest, VQAv2, MME, and MMBench showing significant improvements in fine-grained visual understanding and hallucination suppression.

Figures

Figures reproduced from arXiv: 2506.17901 by the authors.

Figure 1
Figure 1. (a) Current MLLMs often struggle with spurious correlations, leading to co-occurrence [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The pipeline of our proposed MMGrounded-PostAlign. Given an image and text query, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (a)(b) Token probability distributions across transformer layers, showing distinct trends [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Average attention to image features without (left) and with grounding (right). To investigate how grounding influences the model’s attention to image features, we visualize the average attention weights assigned to image features over 500 MSCOCO samples during response…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Frame Selection: Generative Latent Evidence Aggregation for Long-Video Understanding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A query-conditioned latent evidence aggregator after frozen frame selection improves long-video QA by up to +5.2 average / +10.1 LVBench with 0.11–0.40% token overhead.

Reference graph

Works this paper leans on

62 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  2. [2]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024

    arXiv 2024

  3. [3]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    arXiv 2024

  4. [4]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  5. [5]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  6. [6]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  7. [7]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  8. [8]

    A survey on hallucination in large vision-language models

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253, 2024

    arXiv 2024

Show all 62 references
  1. [9]

    Hallucination of multimodal large language models: A survey

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930, 2024

    2024 arXiv

  2. [10]

    Analyzing and mitigating object hallucination in large vision-language models

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754, 2023

    arXiv

  3. [11]

    The deluge of spurious correlations in big data

    Cristian S Calude and Giuseppe Longo. The deluge of spurious correlations in big data. Foundations of science, 22:595–612, 2017

    2017

  4. [12]

    Spurious correlations in machine learning: A survey

    Wenqian Ye, Guangtao Zheng, Xu Cao, Yunsheng Ma, and Aidong Zhang. Spurious correlations in machine learning: A survey. arXiv preprint arXiv:2402.12715, 2024

    2024

  5. [13]

    Mitigating object hallucinations in large vision-language models through visual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2024

  6. [14]

    Described object detection: Liberating object detection with flexible expressions

    Chi Xie, Zhao Zhang, Yixuan Wu, Feng Zhu, Rui Zhao, and Shuang Liang. Described object detection: Liberating object detection with flexible expressions. Advances in Neural Information Processing Systems, 36:79095–79107, 2023

    2023

  7. [15]

    Mattnet: Modular attention network for referring expression comprehension

    Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1307–1315, 2018

    2018

  8. [16]

    A fast and accurate one-stage approach to visual grounding

    Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo. A fast and accurate one-stage approach to visual grounding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4683–4693, 2019. 10

    2019

  9. [17]

    Mdetr: Modulated detection for end-to-end multi-modal under- standing

    Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, Nicolas Carion, and Armand Joulin. Mdetr: Modulated detection for end-to-end multi-modal under- standing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780–1790, 2021

    2021

  10. [18]

    Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection

    Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection. Advances in Neural Information Processing Systems, 35:9125–9138, 2022

    2022

  11. [19]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages ...

    2022

  12. [20]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pages 38–55. Springer, 2024

    2024

  13. [21]

    Phrasecut: Language- based image segmentation in the wild

    Chenyun Wu, Zhe Lin, Scott Cohen, Trung Bui, and Subhransu Maji. Phrasecut: Language- based image segmentation in the wild. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10216–10225, 2020

    2020

  14. [22]

    Advancing referring expression segmentation beyond single image

    Yixuan Wu, Zhao Zhang, Chi Xie, Feng Zhu, and Rui Zhao. Advancing referring expression segmentation beyond single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2628–2638, 2023

    2023

  15. [23]

    Language as queries for referring video object segmentation

    Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4974–4984, 2022

    2022

  16. [24]

    Kosmos-2: Grounding multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023

    2023 arXiv

  17. [25]

    Shikra: Unleashing multimodal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023

    2023 arXiv

  18. [26]

    Llava-grounding: Grounded visual chat with large multimodal models

    Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Leizhang, Chunyuan Li, et al. Llava-grounding: Grounded visual chat with large multimodal models. In European Conference on Computer Vision, pages 19–35. Springer, 2024

    2024

  19. [27]

    Bubogpt: Enabling visual grounding in multi-modal llms

    Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding in multi-modal llms. arXiv preprint arXiv:2307.08581, 2023

    2023 arXiv

  20. [28]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024

    2024

  21. [29]

    Gsva: Generalized segmentation via multimodal large language models

    Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3858–3869, 2024

    2024

  22. [30]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2024

  23. [31]

    Pixellm: Pixel reasoning with large multimodal model

    Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26374–26383, 2024. 11

    2024

  24. [32]

    Ground- hog: Grounding large language models to holistic segmentation

    Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. Ground- hog: Grounding large language models to holistic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14227–14238, 2024

    2024

  25. [33]

    Woodpecker: Hallucination correction for multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(12):220105, 2024

    2024

  26. [34]

    V olcano: mitigating multimodal hallucination through self-feedback guided revision

    Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Minjoon Seo. V olcano: mitigating multimodal hallucination through self-feedback guided revision. arXiv preprint arXiv:2311.07362, 2023

    2023 arXiv

  27. [35]

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conferenc...

    2024

  28. [36]

    Halc: Object hallucination reduction via adaptive focal-contrast decoding

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425, 2024

    2024 arXiv

  29. [37]

    Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding

    Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding. arXiv preprint arXiv:2402.18476, 2024

    2024 arXiv

  30. [38]

    Mitigating object hallucination in large vision-language models via classifier-free guidance

    Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu. Mitigating object hallucination in large vision-language models via classifier-free guidance. arXiv preprint arXiv:2402.08680, 2024

    2024 arXiv

  31. [39]

    Seeing is believing: Mitigating hallucination in large vision-language models via clip-guided decoding

    Ailin Deng, Zhirui Chen, and Bryan Hooi. Seeing is believing: Mitigating hallucination in large vision-language models via clip-guided decoding. arXiv preprint arXiv:2402.15300, 2024

    2024 arXiv

  32. [40]

    Mllm can see? dynamic correction decoding for hallucination mitigation

    Chenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian, Haoming Xu, Shumin Deng, and Huajun Chen. Mllm can see? dynamic correction decoding for hallucination mitigation. arXiv preprint arXiv:2410.11779, 2024

    2024 arXiv

  33. [41]

    Mitigat- ing hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigat- ing hallucination in large multi-modal models via robust instruction tuning. arXiv preprint arXiv:2306.14565, 2023

    2023 arXiv

  34. [42]

    Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data

    Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,...

    2024

  35. [43]

    Mitigating fine-grained hallucination by fine-tuning large vision-language models with caption rewrites

    Lei Wang, Jiabang He, Shenshen Li, Ning Liu, and Ee-Peng Lim. Mitigating fine-grained hallucination by fine-tuning large vision-language models with caption rewrites. InInternational Conference on Multimedia Modeling, pages 32–45. Springer, 2024

    2024

  36. [44]

    Less is more: Mitigating multimodal hallucination from an eos decision perspective

    Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mitigating multimodal hallucination from an eos decision perspective. arXiv preprint arXiv:2402.14545, 2024

    2024 arXiv

  37. [45]

    Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839, 2023

    2023 arXiv

  38. [46]

    Silkie: Preference distillation for large visual language models

    Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312.10665, 2023

    2023 arXiv

  39. [47]

    Detecting and preventing hallucinations in large vision language models

    Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, pages 18135–18143, 2024. 12

    2024

  40. [48]

    Aligning large multimodal models with factually augmented rlhf

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023

    2023 arXiv

  41. [49]

    Mul- timodal chain-of-thought reasoning in language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Mul- timodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023

    2023 arXiv

  42. [50]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022

    2022

  43. [51]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023

    2023

  44. [52]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  45. [53]

    Haloquest: A visual hallucination dataset for advancing multimodal reasoning

    Zhecan Wang, Garrett Bingham, Adams Wei Yu, Quoc V Le, Thang Luong, and Golnaz Ghiasi. Haloquest: A visual hallucination dataset for advancing multimodal reasoning. In European Conference on Computer Vision, pages 288–304. Springer, 2024

    2024

  46. [54]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  47. [55]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904–6913, 2017

    2017

  48. [56]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024

    2024

  49. [57]

    A survey of multimodel large language models

    Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pages 405–409, 2024

    2024

  50. [58]

    Referitgame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014

    2014

  51. [59]

    Empowering segmentation ability to multi-modal large language models

    Yuqi Yang, Peng-Tao Jiang, Jing Wang, Hao Zhang, Kai Zhao, Jinwei Chen, and Bo Li. Empowering segmentation ability to multi-modal large language models. arXiv preprint arXiv:2403.14141, 2024

    2024 arXiv

  52. [60]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310...

    arXiv

  53. [61]

    Ferret: Refer and ground anything anywhere at any granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023

    2023 arXiv

  54. [62]

    Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks

    Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. Advances in Neural Information Processing Sy...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.