Pith. sign in

REVIEW 3 major objections 5 minor 50 references

See Different, Think Better: Visual Variations Mitigating Hallucinations in LVLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training vision-language models on paired images that differ in a single object or attribute—visual variations—reduces hallucination and sharpens fine-grained understanding across three open LVLMs.

desk verdict A useful dataset and a plausible pipeline, but the central causal claim—visual variations specifically drive the gains—is not actually tested. read the letter →

arxiv 2507.22003 v2 pith:3VUJ7W3U submitted 2025-07-29 cs.CV

Ziyun Dai , Xiaoqiang Li , Shaohua Zhang , Yuanchen Wu , Jide Li This is my paper
classification cs.CV
keywords Largevision-languagemodelHallucinationmitigationVisual-semanticalignmentVisualvariationimagesinstructiontuningControllableimagegenerationCounterfactualtrainingdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that vision-language models hallucinate—confidently describing objects or attributes that are not in the image—because they are trained on text-heavy instruction data that does not force fine-grained visual-semantic alignment. ViHallu is a data-centric fix: it generates visual variation images that are near-identical to an original except for one controlled change (an object, attribute, or scene element), then builds paired question-answer instructions that contrast the two versions. Fine-tuning LLaVA-1.5, MiniGPT-4 v2, and Qwen2-VL on these pairs reportedly reduces object, attribute, and relation hallucinations across POPE, LLaVA-Bench, and MMHal-Bench while preserving or improving general visual question answering. If right, this gives a simple recipe: instead of refining text only, show models images that differ in exactly the way you want them to learn to see.

What carries the argument

The load-bearing mechanism is the visual variation image: a text-to-image generated image that shares the segmentation-mask layout of the source image but follows an edited caption that changes one object category or attribute. The pipeline extracts the original caption with Tag2Text and segmentation masks with MobileSAM, has DeepSeek-chat V2 rewrite the caption through concept substitution, feeds mask plus edited caption into a controllable text-to-image model (cited as reference [27]) to render the variation, and filters generations with VQAScore at a threshold of 0.6. On the instruction side, a baseline LVLM writes a description (which may contain hallucinations), Grounded-SAM provides object tags for full coverage, DeepSeek-chat V2 generates questions—including ones targeted at hallucinated elements—and InternVL2.5 answers based on the actual image, with a panel of three LVLM experts keeping QA pairs endorsed by at least two. The core idea is that asking the same question about an original and its variation yields different correct answers, forcing the model to attend to the changed visual detail rather than relying on text prior.

What would settle it

Retrain the same models on the ViHallu instruction set with the variation images replaced by the original images (no visual change) under identical hyperparameters; if POPE and MMHal improvements do not shrink significantly, the visual variation is not the active ingredient. A complementary check: have annotators compare a random sample of pairs and flag any with unintended changes, and look for a correlation between annotation failure rate and benchmark gains.

Watch

Extended reading notes

Core claim

ViHallu's central claim is that training on visual variation image pairs—images generated to keep the global structure of an original while altering only a target object or attribute—strengthens the visual-semantic alignment of large vision-language models and thereby curbs hallucinations. The paper reports that fine-tuning three open LVLMs (LLaVA-1.5-7B, MiniGPT-4 v2, Qwen2-VL-7B) on the accompanying ViHallu-Instruction dataset (6,770 images, roughly 50k instructions) improves accuracy and F1 on the POPE object-hallucination benchmark across random, popular, and adversarial settings for most models, raises scores on all three LLaVA-Bench task categories, and increases overall score while lowering hallucination rate on MMHal-Bench. The paper frames this as the first tailored instruction dataset built from visual variation images, and claims the mechanism is counterfactual: putting objects in rare or surprising contexts during training reduces the model's reliance on statistical co-occurrence, forcing it to judge from visual evidence.

Load-bearing premise

The entire training signal assumes that each generated variation image differs from its original only in the targeted object or attribute, with no unintended changes or artifacts; the paper filters with a global VQAScore threshold of 0.6 but does not verify that the modifications are localized and semantically exact.

Editorial extensions

If this is right

  • Fine-tuning on visual variation pairs reduces object hallucinations on POPE for LLaVA-1.5 and MiniGPT-4 v2 across random, popular, and adversarial settings, and slightly improves Qwen2-VL on the first two.
  • General visual reasoning on LLaVA-Bench improves for all three models after ViHallu tuning, with the largest gains in conversation and complex reasoning.
  • MMHal-Bench overall score rises and hallucination rate falls for LLaVA-1.5, MiniGPT-4 v2, and Qwen2-VL after tuning, including gains in adversarial and environment categories.
  • The released ViHallu-Instruction dataset (6,770 images, roughly 50k instructions) offers a reusable resource for hallucination mitigation and fine-grained visual-semantic alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A no-variation control experiment would disentangle whether gains come from the visual change itself or from the richer single-image instruction data; the paper does not report one.
  • Because VQAScore at 0.6 filters out many generations, the pipeline may over-select simple, easily editable scenes; extending evaluation to harder images with verified localized edits would test whether the recipe scales.
  • The variation axis could be extended to counting, spatial relations, time, or style, generating counterfactual pairs for other hallucination categories beyond object, attribute, and relation.
  • Combining ViHallu's visual variations with text-centric negative samples (hard negative captions, preference pairs) may yield compounding reductions, since the two target different failure modes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ViHallu, a data-centric framework for reducing hallucination in Large Vision-Language Models (LVLMs). The method generates visual variation images from original images by editing captions and using segmentation-mask-controlled text-to-image generation, then constructs visual instruction data (ViHallu-Instruction) with QA pairs that highlight differences between original and variation images. The authors fine-tune LLaVA-1.5, MiniGPT-4 v2, and Qwen2-VL on this dataset and report improvements on POPE, LLaVA-Bench, and MMHal-Bench, together with scaling and consistency analyses. The core claim is that training on these visual variations improves fine-grained visual understanding and reduces object, attribute, and relation hallucinations.

Significance. If the central claim is established, the paper offers a practical and novel recipe: instead of relying on text-only negative examples, it uses controllable image generation to create counterfactual visual samples that explicitly target visual-semantic alignment. The released dataset and code are useful resources, and the evaluation across three architectures and multiple benchmarks gives the work broad relevance. The paper also correctly identifies that text-centric methods cannot address fine-grained visual distinctions. The main weakness is that the empirical design does not isolate the contribution of visual variations, so the causal attribution in the title and abstract is not yet supported.

major comments (3)
  1. [Section 4.2 and Section 5] No ablation isolates the visual-variation component. Every fine-tuning experiment uses the full ViHallu-Instruction mixture, which contains both original and variation images with tailored QA pairs. The comparisons in Table 2 are against text-centric methods using different data and training paradigms, so they do not control for the effect of generic visual instruction tuning on fine-grained questions. The scaling analysis in Section 5.2 also samples only from the full mixture, so it cannot separate 'more data helps' from 'visual variations specifically help'. A control condition fine-tuned on an instruction set built solely from original images, with matched QA pairs, is load-bearing for the paper's central claim that visual variations cause hallucination reduction. Without this control, the reported gains could be due to additional visual instruction tuning rather than the proposed variation mechanism.
  2. [Section 3.1] The claim that generated visual variation images differ from the original only in the targeted object or attribute is not verified. The paper states that segmentation masks ensure structural layout and object positioning, and that the edited caption guides generation, but it provides no human or automated check that modifications are localized and semantically exact. VQAScore, with a threshold of 0.6, only checks global image-text alignment; it does not confirm that the visual change is confined to the intended region. If the generated image differs in other, unintended ways, the paired QA training may teach the model spurious correlations rather than the intended fine-grained visual-semantic alignment. The authors should report either a human evaluation of localization accuracy or a quantitative metric (e.g., mask-region edit localization or paired-image difference analysis).
  3. [Table 1 and Table 2] All results are single-run numbers without multiple seeds or error bars. Several reported improvements are very small: Qwen2-VL gains 0.07% accuracy on the popular set and loses 0.34% accuracy on the adversarial set. Given the absence of variance information, the claim of consistent improvement across models and settings is not statistically supported. The post hoc explanation for the Qwen2-VL adversarial decrease ('minor bias effects') is speculative and does not address this. Additionally, Table 2 lists the vision-centric baseline VH with F1 only and an empty accuracy cell ('/'), so the paper's statement that ViHallu outperforms other methods is not fully supported for that baseline. The authors should run at least three seeds per configuration, report means and standard deviations, and complete or justify the missing VH accuracy value.
minor comments (5)
  1. [Abstract] The phrase 'the challenges of visual-semantic alignment significantly limit their effectiveness' is grammatically awkward; consider revising to 'these methods are limited by weak visual-semantic alignment, especially in fine-grained scenarios.'
  2. [Section 1, contributions] The first contribution bullet reads 'The ViHallu propose a novel visual variation image generation approach' — this should be 'ViHallu proposes' and the sentence should end with a period instead of a comma before 'while maintaining the overall image structure.'
  3. [Table 2] The entry 'LLaVA-1.5 w/VH [9] / 84.80' shows a slash where accuracy should be; please either provide the accuracy value or state clearly why it is unavailable.
  4. [Section 4.1] The dataset size is given as 'approximately 50k (±10k) tailored instructions', which is imprecise; please report the exact number of instruction samples and the per-model breakdown.
  5. [Section 5.2] The claim about a 'general upward trend' in Figure 13(a) would be stronger if error bars or per-subset variance were shown; as written, individual points could lie within noise.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central empirical claim is tested against external benchmarks; the lack of an original-image-only control is an experimental design limitation, not a circular derivation.

full rationale

ViHallu's central claim is an empirical fine-tuning result rather than a formal derivation. Visual variation images are produced by an external controllable T2I model (ControlNet++) using segmentation masks and edited captions; the instruction QA pairs are generated by DeepSeek-V2 and InternVL-2.5 and filtered by a panel of LVLM experts. All reported evaluations use external benchmarks (POPE, LLaVA-Bench, MMHal-Bench), with MMHal-Bench scored by GPT-4o, so the improvements are not measured on the training distribution. The paper contains no fitted parameter that is later renamed as a prediction, no equation in which the target result is defined into existence, and no self-citation chain that supplies a load-bearing premise. The deliberate construction of questions targeting the base model's known hallucination patterns is a data-design choice, and because the evaluation is external, it does not force the observed gains by construction. A separate and non-circular concern is that no ablation isolates the visual-variation component from generic visual instruction tuning on the same instruction data; this threatens causal attribution but does not make the derivation circular.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

All free parameters are hand-set thresholds or hyperparameters that shape the dataset and fine-tuning; none are derived from theory. The central empirical claim rests on unverified assumptions about generation fidelity, QA filtering reliability, benchmark cleanliness, and the counterfactual-training mechanism.

free parameters (5)
  • VQAScore quality threshold = 0.6
    Hand-set in Section 3.1 and Appendix B.1; images scoring below 0.6 are discarded, and dataset size and content depend directly on this cut.
  • Expert panel agreement threshold = 2 of 3 experts
    QA pairs are retained only when at least two of three LVLM experts answer 'Yes' (Section 3.2, Appendix B.3); changing this threshold changes the instruction set.
  • Questions per image = 7
    Section 4.1: DeepSeek-chat V2 extracts seven questions per image from descriptions; this determines instruction quantity and coverage.
  • VQAScore evaluation model = LLaVA-1.5-13B
    The quality score and therefore the filtered dataset depend on this specific scorer; a different scorer would produce a different dataset.
  • Fine-tuning hyperparameters = LLaVA: lr 2e-5; MiniGPT/Qwen: lr 1e-5; LoRA rank 64 alpha 16
    Section 4.2: learning rates, LoRA settings, and trainable modules are chosen by hand; the magnitude of gains could vary with these choices.
assumptions (5)
  • domain assumption ControlNet++ with segmentation masks and edited captions produces images that preserve global structure while altering only the targeted region.
    Stated in Section 3.1 and relied on throughout; no quantitative verification of localization is provided.
  • domain assumption VQAScore above 0.6 indicates a usable variation image for training.
    Used to filter 7,209 generated images down to 5,051 (Section 3.1, B.1); the score measures caption-image alignment, not whether the edit is exactly the intended one.
  • domain assumption Majority agreement among LLaVA-1.5, MiniCPM-V 2.6, and mPLUG-OWL3 is a valid proxy for QA pair correctness.
    Section 3.2 and B.3; expert judgments are themselves LVLM outputs and may share systematic hallucinations.
  • ad hoc to paper Training on counterfactual object-scene pairings reduces reliance on co-occurrence priors.
    This is the paper's causal story (Section 3.1, counterfactual interventions); it is plausible but not directly tested.
  • domain assumption The evaluation benchmarks are not contaminated by images used to build ViHallu-Instruction.
    The source of the 2,000 original images is not stated (Section 4.1), so overlap with POPE, LLaVA-Bench, or MMHal-Bench cannot be excluded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of See Different, Think Better: Visual Variations Mitigating Hallucinations in LVLMs." pith.science (2026). https://pith.science/paper/3VUJ7W3U

@misc{pith2026250722003,
  author       = {Pith},
  title        = {Pith review of: See Different, Think Better: Visual Variations Mitigating Hallucinations in LVLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3VUJ7W3U}},
  note         = {Machine review of arXiv:2507.22003}
}
read the original abstract

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in visual understanding and multimodal reasoning. However, LVLMs frequently exhibit hallucination phenomena, manifesting as the generated textual responses that demonstrate inconsistencies with the provided visual content. Existing hallucination mitigation methods are predominantly text-centric, the challenges of visual-semantic alignment significantly limit their effectiveness, especially when confronted with fine-grained visual understanding scenarios. To this end, this paper presents ViHallu, a Vision-Centric Hallucination mitigation framework that enhances visual-semantic alignment through Visual Variation Image Generation and Visual Instruction Construction. ViHallu introduces visual variation images with controllable visual alterations while maintaining the overall image structure. These images, combined with carefully constructed visual instructions, enable LVLMs to better understand fine-grained visual content through fine-tuning, allowing models to more precisely capture the correspondence between visual content and text, thereby enhancing visual-semantic alignment. Extensive experiments on multiple benchmarks show that ViHallu effectively enhances models' fine-grained visual understanding while significantly reducing hallucination tendencies. Furthermore, we release ViHallu-Instruction, a visual instruction dataset specifically designed for hallucination mitigation and visual-semantic alignment. Code is available at https://github.com/oliviadzy/ViHallu.

Figures

Figures reproduced from arXiv: 2507.22003 by the authors.

Figure 1
Figure 1. By inputting segmentation masks along with guid [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our framework ViHallu. The left shows the visual variation image generation process: (1) original image [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of caption editing prompt design. The [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Samples from the ViHallu-Instruction dataset. The top row displays two pairs of images: original images (left) and their [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Performance comparison of different models be [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Response comparison between baseline LLaVA-1.5 and LLaVA-1.5 [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: (a) F1 score on POPE benchmark of LVLMs fine [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 9
Figure 9. Figure 9: Pie chart represents the proportion of QA pairs that [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 8
Figure 8. Figure 8: Distribution of VQAScore evaluated by LLaVA-1.5- [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 10
Figure 10. Figure 10: Prompt template for generating diverse question types using detailed descriptions and object tags. [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Prompt template for LVLM expert evaluation of QA pairs. The template enables binary assessment of alignment [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: Sample pairs from ViHallu-Instruction, showing original and visual variation images with their corresponding QA [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Sample outputs from base models and their ViHallu fine-tuned counterparts [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Sample outputs from base models and their ViHallu fine-tuned counterparts [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 20 canonical work pages

  1. [9]

    Wen Huang, Hongbin Liu, Minxin Guo, and Neil Zhenqiang Gong. 2024. Vi- sual hallucinations of multi-modal large language models. arXiv preprint arXiv:2402.14683 (2024)

    arXiv 2024

  2. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  3. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

    2020

  4. [3]

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. 2024. Allava:Harnessinggpt4v-synthesizeddataforalitevision-languagemodel. arXiv preprint arXiv:2402.11684(2024)

    arXiv 2024

  5. [4]

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechu Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023. MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning.arXiv preprint arXiv:2310.09478(2023)

    arXiv 2023

  6. [5]

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. 2024. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271(2024)

    arXiv 2024

  7. [6]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv:2305.06500 [cs.CV] https://arxiv.org/abs/2305.06500

    arXiv 2023

  8. [7]

    DeepSeek-AI. 2024. DeepSeek-V2: A Strong, Economical, and Efficient Mixture- of-Experts Language Model. arXiv:2405.04434 [cs.CL]

    arXiv 2024

Show all 50 references
  1. [8]

    Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou

  2. [10]

    Tag2Text:GuidingVision-Language Model via Image Tagging.arXiv preprint arXiv:2303.05657(2023)

    Xinyu Huang, Youcai Zhang, Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, YaqianLi,YandongGuo,andLeiZhang.2023. Tag2Text:GuidingVision-Language Model via Image Tagging.arXiv preprint arXiv:2303.05657(2023)

    2023 arXiv

  3. [11]

    Tri Huynh, Simon Kornblith, Matthew R Walter, Michael Maire, and Maryam Khademi. 2022. Boosting contrastive self-supervised learning with false negative cancellation. InProceedings of the IEEE/CVF winter conference on applications of computer vision. 2785–2795

    2022

  4. [12]

    ChaoyaJiang,HaiyangXu,MengfanDong,JiaxingChen,WeiYe,MingYan,Qing- hao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. 2024. Hallucination augmented contrastive learning for multimodal large language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  5. [13]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment Anything.arXiv:2304.02643 (2023)

    2023 arXiv

  6. [14]

    Blip-2:Bootstrapping language-image pre-training with frozen image encoders and large language models

    JunnanLi,DongxuLi,SilvioSavarese,andStevenHoi.2023. Blip-2:Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning. PMLR, 19730–19742

    2023

  7. [15]

    Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, and Chen Chen. 2025. ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback. InEuropean Conference on Computer Vision. Springer, 129–147

    2025

  8. [16]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen

  9. [17]

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. 2025. Evaluating text-to-visual gen- eration with image-to-text generation. InEuropean Conference on Computer Vision. Springer, 366–384

    2025

  10. [18]

    Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355(2023)

    2023 arXiv

  11. [19]

    ImprovedBaselines with Visual Instruction Tuning

    HaotianLiu,ChunyuanLi,YuhengLi,andYongJaeLee.2023. ImprovedBaselines with Visual Instruction Tuning

    2023

  12. [20]

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang

  13. [21]

    InThe Twelfth International Conference on Learning Representations

    Mitigating hallucination in large multi-modal models via robust instruction tuning. InThe Twelfth International Conference on Learning Representations

  14. [22]

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024. A survey on hallucination in large vision-language models.arXiv preprint arXiv:2402.00253(2024)

    2024 arXiv

  15. [23]

    Improvedbaselines with visual instruction tuning

    HaotianLiu,ChunyuanLi,YuhengLi,andYongJaeLee.2024. Improvedbaselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 26296–26306

    2024

  16. [24]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruc- tion tuning.Advances in neural information processing systems36 (2024)

    2024

  17. [25]

    OpenAI. 2022. Introducing ChatGPT. https://openai.com/blog/chatgpt

    2022

  18. [26]

    ShilongLiu,ZhaoyangZeng,TianheRen,FengLi,HaoZhang,JieYang,Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. 2023. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499 (2023)

    2023 arXiv

  19. [27]

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. 2024. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 4296–4304

    2024

  20. [28]

    Xiaoye Qu, Qiyuan Chen, Wei Wei, Jishuo Sun, and Jianfeng Dong. 2024. Al- leviating hallucination in large vision-language models with active retrieval augmentation. arXiv preprint arXiv:2408.00555(2024)

    2024 arXiv

  21. [29]

    Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker. 2020. Deep struc- tural causal models for tractable counterfactual inference.Advances in neural information processing systems33 (2020), 857–869

    2020

  22. [30]

    Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. 2023. Unicontrol: A unified diffusion model for controllable visual generation in the wild.arXiv preprint arXiv:2305.11147(2023)

    2023 arXiv

  23. [31]

    Ugur Sahin, Hang Li, Qadeer Khan, Daniel Cremers, and Volker Tresp. 2024. Enhancing multimodal compositional reasoning of visual language models with generative negative mining. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 5563–5573

    2024

  24. [32]

    Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan

  25. [33]

    6967–6977

    Filtering,distillation,andhardnegativesforvision-languagepre-training.In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 6967–6977

  26. [34]

    GroundedSAM:Assembling Open-World Models for Diverse Visual Tasks

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang,HongyangLi,QingJiang,andLeiZhang.2024. GroundedSAM:Assembling Open-World Models for Diverse Visual Tasks. arXiv:2401...

    2024 arXiv

  27. [35]

    Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. 2023. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574(2023)

    2023 arXiv

  28. [36]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. 2023. Aligning Large Multimodal Models with Factually Augmented RLHF. (2023)

    2023

  29. [37]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971(2023)

    2023 arXiv

  30. [38]

    HaonanWang,MinbinHuang,RunhuiHuang,LanqingHong,HangXu,Tianyang Hu, Xiaodan Liang, Zhenguo Li, Hong Cheng, and Kenji Kawaguchi. 2023. Boosting visual-language models by exploiting hard samples.arXiv preprint arXiv:2305.05208 (2023)

    2023 arXiv

  31. [39]

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2024. mPLUG-Owl3: Towards Long Image-Sequence UnderstandinginMulti-ModalLargeLanguageModels. arXiv:2408.04840[cs.CV] https://arxiv.org/abs/2408.04840

    2024 arXiv

  32. [40]

    Lei Wang, Jiabang He, Shenshen Li, Ning Liu, and Ee-Peng Lim. 2024. Mitigating fine-grained hallucination by fine-tuning large vision-language models with caption rewrites. InInternational Conference on Multimedia Modeling. Springer, 32–45

    2024

  33. [41]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191 (2024)

    2024 arXiv

  34. [42]

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. MiniCPM-V: A GPT-4V Level MLLM on Your Phone.arXiv preprint arXiv:2408.01800(2024)

    2024 arXiv

  35. [43]

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding conditional con- troltotext-to-imagediffusionmodels.In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3836–3847

    2023

  36. [44]

    Kun Zhou Jinpeng Wang Wayne Xin Zhao Yifan Li, Yifan Du and Ji-Rong Wen

  37. [45]

    InThe 2023 Conference on Empirical Methods in Natural Language Processing

    Evaluating Object Hallucination in Large Vision-Language Models. InThe 2023 Conference on Empirical Methods in Natural Language Processing. https: //openreview.net/forum?id=xozJw0kZXF

    2023

  38. [46]

    QifanYu,JunchengLi,LonghuiWei,LiangPang,WentaoYe,BoshengQin,Siliang Tang,Qi Tian, andYuetingZhuang. 2024. Hallucidoctor:Mitigating hallucinatory toxicity in visual instruction data. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12944–12953

    2024

  39. [47]

    Fastersegmentanything:Towardslightweight sam for mobile applications.arXiv preprint arXiv:2306.14289(2023)

    Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee,andChoongSeonHong.2023. Fastersegmentanything:Towardslightweight sam for mobile applications.arXiv preprint arXiv:2306.14289(2023)

    2023 arXiv

  40. [49]

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. 2023. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization.arXiv preprint arXiv:2311.16839(2023)

    2023 arXiv

  41. [50]

    Yes”/“No

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2023. Analyzing and mitigating object hallucination in large vision-language models.arXiv preprint arXiv:2310.00754 (2023). A Overview In this supplementary material, ...

    2023 arXiv

  42. [2023]

    arXiv preprint arXiv:2302.09778(2023)

    Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09778(2023)

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.