Pith. sign in

REVIEW 4 major objections 5 minor 51 references

Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a single synthetic-data post-training framework, Antidote, simultaneously reduces hallucinations from counterfactual presuppositions and object perception in LLaVA models, without external supervision or…

desk verdict A practical, well-evaluated post-training recipe for a real failure mode; the main caveat—unvalidated grounding labels—is worth an extra audit, not a rejection. read the letter →

arxiv 2504.20468 v2 pith:V6U7C7J2 submitted 2025-04-29 cs.CV

Yuanchen Wu , Lu Zhang , Hang Yao , Junlong Du , Ke Yan , Shouhong Ding , Yunsheng Wu , Xiaoqiang Li This is my paper · ORCID
classification cs.CV
keywords largevision-languagemodelshallucinationmitigationcounterfactualpresuppositionquestionspreferenceoptimizationdirectsyntheticdatapipelineobjectperceptionCP-Bench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that current hallucination fixes miss a blind spot: a model that correctly says "no car" will still answer "BMW" when asked what brand the absent car is. To fix this, the authors build Antidote, a fully synthetic pipeline that generates images in which statistically co-occurring objects are stripped out (a speedboat with no bridge, a street with no car), uses a grounding detector to verify what is and is not present, and then lets the model itself correct its answers using that factual information. Those corrected answers become preferred training samples and the original hallucinated answers become rejected samples in a DPO-style preference optimization. Applied to LLaVA-1.5 and LLaVA-Next, the recipe raises CP-Bench F1 by over 50 points, improves POPE by 1.8–3.3%, and cuts CHAIR/SHR hallucinations by roughly 30–50%, without degrading general benchmarks. The paper also contributes CP-Bench, a benchmark of 1,000 curated counterfactual-presupposition and true-presupposition questions.

What carries the argument

The load-bearing object is the synthetic data pipeline plus the preference pair it produces. For each caption, a language model lists objects that are present ($O_{pre}$) and objects that commonly co-occur but will be absent ($O_{hallu}$); Stable Diffusion 3 renders the scene using $O_{hallu}$ as a negative prompt; and Grounding-DINO, an open-set grounding detector, confirms that $O_{pre}$ objects appear and $O_{hallu}$ objects do not, discarding triplets where verification fails. The verified facts are then prepended to the question ("Given the fact that there is no car in the image, what is the brand of the car?"), and the model's self-corrected answer becomes the DPO-preferred response over its original hallucinated answer. This transforms hallucination mitigation into a preference-optimization problem without any external expert model or human preference labels.

What would settle it

Take a random sample of the generated training images, have human annotators label whether each $O_{pre}$ and $O_{hallu}$ object is truly present, and measure the Factual Assessor's precision and recall. If recall on $O_{pre}$ or precision on $O_{hallu}$ is materially below 100%, the preference pairs contain systematically mislabeled examples; a second check would be to train Antidote with those mislabeled pairs excluded and see whether the CP-Bench gains shrink.

Watch

Extended reading notes

Core claim

The central claim is that counterfactual-presupposition hallucination—a model accepting a false premise embedded in a question and answering about an object that is not in the image—is a distinct and widespread failure mode, and that it can be mitigated together with classic object-perception hallucination by a single preference-alignment post-training method. The mechanism: synthesize scenes where common co-occurrences are decoupled, verify the resulting object sets with an open-set grounding detector, inject the verified facts into the question prompt, collect the model's self-corrected answer as the preferred response and the original hallucinated answer as the rejected response, and optimize with DPO. On LLaVA-1.5-7B/13B and LLaVA-Next, the method is reported to raise CPQ correctness dramatically, reduce object-existence and description hallucinations, and preserve general ability.

Load-bearing premise

The whole training signal depends on the grounding detector being correct: if a supposedly absent hallucination-candidate object is actually present in a generated image, or a supposedly present object is missed, then the factual prior and the DPO preference labels are wrong, and the model is trained to deny or assert things that are false.

Editorial extensions

If this is right

  • A model trained with Antidote should refuse to answer questions whose presupposed object is absent, stating the absence rather than inventing an attribute or brand.
  • Object-existence hallucinations on adversarial POPE sets—where objects are chosen by co-occurrence frequency—should shrink, since the training explicitly decouples frequent co-occurrences.
  • The gains should transfer across LLaVA-scale architectures and, the paper argues, should not come at the cost of general benchmarks such as MMBench or ScienceQA.
  • Because the recipe uses only synthetic data and the model's own self-correction, it should be reproducible for other LVLMs without access to proprietary expert models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own observation that post-training makes models cautious on LLaVA-Wild suggests a tunable trade-off between refusal and informativeness; an obvious extension is to weight CPQ samples against description samples or mix in a small amount of general instruction data to control this.
  • Because the CP-Bench dev set is generated by the same pipeline used for training, part of the reported CP-Bench gain may reflect distribution overlap; the strongest test of the paper's claim would be a human-written CPQ set drawn from scenes not seen in any training caption.
  • The same decoupling idea—remove a statistically expected object and ask about it—could be applied to attribute and relation hallucinations by extending the factual prior to attributes rather than existence alone.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Antidote, a synthetic-data-driven post-training framework that uses Stable Diffusion 3 generated images with decoupled object co-occurrences, a Grounding-DINO-based Factual Assessor, and DPO to reduce hallucinations in counterfactual presupposition questions (CPQs) and object perception. The authors also introduce CP-Bench, a benchmark with a dev set generated by the same pipeline and a manually curated test set from CC3M, evaluated by GPT-4o. Experiments on LLaVA-1.5-7B/13B and LLaVA-Next-Mistral-7B report substantial gains on CP-Bench, POPE, CHAIR, and SHR, with minimal loss on general benchmarks.

Significance. If the results are reproducible, the paper makes a useful contribution: it identifies a neglected failure mode (CPQs), proposes a fully automated training-data pipeline that does not require stronger LVLMs for preference-pair construction, and reports large improvements on both the new benchmark and existing hallucination benchmarks. The manually curated test set is a strength, and the release of code will aid reproducibility. However, the central mechanism depends on the correctness of the Factual Assessor's grounding labels, which is not validated, and the evaluation protocol has gaps (single runs, sole GPT-4o judge) that need to be addressed before the claims can be fully accepted.

major comments (4)
  1. [Sec. 3.2, Step 3, Fig. 4] The correctness of the DPO preference pairs in Eq. (1) depends on the Factual Assessor's ability to verify the presence of Opre and the absence of Ohallu in every synthetic image. The paper reports only that approximately 4k of 14k triplets were filtered, with no precision/recall audit of Grounding-DINO on the generated images. A false negative on an Ohallu object leaves a sample labeled 'verified absent' even though the object is visible, and a false positive on an Opre object retains a sample where the object is actually absent; either error teaches the model the wrong preference. Because the CP-Bench dev set is built by the same pipeline, assessor errors can also inflate the dev results. Please report per-object precision/recall on a human-annotated sample of the generated images, and describe precisely how the filtering handles individual objects (e.g., whether a missing Opre object invalidates the triplet).
  2. [Sec. 4, CP-Bench dev set] The dev set of CP-Bench is generated by the same synthetic pipeline used to create Antidote's training data. This shared distribution can make dev-set gains reflect overfitting to the pipeline rather than genuine improvement in presupposition discrimination. While the manually curated test set provides some independence, the paper often reports dev F1 (Table 1) without noting this caveat. Please either report the main results on the test set only, or provide an analysis (e.g., human evaluation of dev-set questions, or a comparison of model behavior on dev vs. test) that supports the dev set as a valid held-out benchmark.
  3. [Sec. 5.1, Tables 1-3] All experimental results appear to be single runs, with no standard deviations, confidence intervals, or significance tests. This is especially problematic for POPE, where the claimed improvements are 1.8-3.3 F1 points, and for the differences among baseline methods within the same table. Please report results over multiple seeds (or equivalent variance estimates) and perform paired significance tests where appropriate.
  4. [Sec. 4, Evaluation] CP-Bench relies exclusively on GPT-4o to convert open-ended responses into binary presupposition-correctness judgments, with no human agreement or alternative judge. Since the headline claim is a >50% F1 improvement on this benchmark, the reliability of the judge is load-bearing. Please provide a human-annotated subset (e.g., 200 samples) with inter-annotator agreement between GPT-4o and humans, or at minimum agreement with a second strong judge, and report the correlation.
minor comments (5)
  1. [Sec. 5.2] The text mentions 'VPF-Bench' in the paragraph on model size; this appears to be a typo for 'CP-Bench'.
  2. [Eq. (2)] The reward margin defined in Eq. (2) includes only the preferred-response log-ratio; the standard DPO reward margin is the difference between the preferred and rejected log-ratios. Please correct the definition or clarify.
  3. [Sec. 3.2, Step 3] The description of the filtering rule is ambiguous: 'If an object in Opre is not detected, it will be removed' could mean the object is removed from Opre or the triplet is discarded. Please clarify precisely how the object-level filtering affects the triplet and the downstream QA generation.
  4. [Tables 2-3] The citations for VDD and VCD appear inconsistent with the reference list; please verify the numbering.
  5. [Abstract] The phrase 'all without relying on external supervision from stronger LVLMs or human feedback and introducing noticeable catastrophic forgetting issues' is ambiguous; add 'without' before 'introducing' for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the CP-Bench test set is manually curated and the object-perception claims are anchored by external benchmarks; the shared dev/train pipeline is a caveat, not a circular reduction.

full rationale

The paper's central derivation chain is not circular. Antidote's training pairs are constructed by synthesizing images with SD3 using O_pre as the prompt and O_hallu as the negative prompt, verifying presence/absence with the Grounding-DINO-based Factual Assessor, and then using the baseline model's original answer as y_neg and its fact-prior-conditioned self-correction as y_pos in the DPO loss of Eq. (1). The evaluation supporting the headline claims is not defined in terms of these same labels: CP-Bench test images come from CC3M, the object candidates are filtered and manually selected, and the queries are manually curated; POPE, CHAIR, and SHR are established external benchmarks with their own annotations. The unvalidated precision/recall of the Factual Assessor on SD3 images is a genuine correctness risk, because mislabeled preference pairs would teach the model wrong behavior, but that is an input-quality problem rather than a circularity: the paper does not define the target metric in terms of the assessor's outputs. The one self-consistency caveat is that CP-Bench-dev images and questions are generated using the same data synthetic pipeline as the Antidote training set, so dev-set gains in Table 1 are partly a fit to the pipeline's own labeling distribution and should not be treated as independent evidence. This caveat does not propagate to the central claim, because the main CP-Bench column is the manually annotated test set and the object-perception results are anchored by external benchmarks. No load-bearing self-citation, imported uniqueness theorem, or ansatz-by-citation is present. Therefore, under the strict definition of circularity, the finding is no significant circularity, score 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method's training signal is assembled from a chain of external models and detectors: DeepSeek-V2 for world knowledge, Stable Diffusion 3 for image synthesis, Grounding-DINO for presence/absence verification, and GPT-4o for benchmark scoring. Each link is an assumption that is not independently validated inside the paper.

free parameters (5)
  • LoRA rank r = 64
    Selected via Table 5 sweep; r=128 causes forgetting and r=256 causes model collapse.
  • LoRA alpha = 128
    Set to 2x rank by default; not independently swept.
  • DPO beta = 0.1
    Preference margin hyperparameter; no sensitivity analysis is shown.
  • Training data composition = 5k CPQ, 5k TPQ, 2k object existence, 8k description
    Manually chosen proportions; appendix discussion is referenced but not present in the main text.
  • Response filtering cosine threshold = not specified
    BGE-m3 cosine similarity cutoff is used to discard about 15% of samples, but the threshold is not reported.
assumptions (5)
  • domain assumption Grounding-DINO detections uniquely determine whether O_pre objects are present and O_hallu objects are absent in generated images.
    Used in Step 3 to build factual labels; detection misses or false positives inject wrong preference pairs. No precision/recall validation is reported.
  • domain assumption DeepSeek-V2's world knowledge correctly identifies typical co-occurring objects and produces valid factual priors.
    Used in Steps 2 and 3 for O_pre/O_hallu selection, verification, and question generation; erroneous world knowledge would create mislabeled CPQ/TPQ samples.
  • domain assumption Stable Diffusion 3, with positive prompt Cimg and negative prompt O_hallu, generates images in which O_hallu are reliably absent.
    The decoupling of statistically co-occurring objects is the core novelty; if the generator reintroduces hallucination-candidate objects, the training data does not test what it claims.
  • domain assumption Self-correction with injected factual priors yields preferred responses that are both fluent and factually correct, and DPO over these pairs transfers to inference without priors.
    The preferred response is generated by prompting the baseline model with ground truth; the paper does not measure how often self-correction fails or how well it transfers.
  • domain assumption GPT-4o binary judgments are a valid and reliable measure of CP-Bench correctness.
    All CP-Bench scores depend on GPT-4o classifications; no human agreement or consistency analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception." pith.science (2026). https://pith.science/paper/V6U7C7J2

@misc{pith2026250420468,
  author       = {Pith},
  title        = {Pith review of: Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V6U7C7J2}},
  note         = {Machine review of arXiv:2504.20468}
}
read the original abstract

Large Vision-Language Models (LVLMs) have achieved impressive results across various cross-modal tasks. However, hallucinations, i.e., the models generating counterfactual responses, remain a challenge. Though recent studies have attempted to alleviate object perception hallucinations, they focus on the models' response generation, and overlooking the task question itself. This paper discusses the vulnerability of LVLMs in solving counterfactual presupposition questions (CPQs), where the models are prone to accept the presuppositions of counterfactual objects and produce severe hallucinatory responses. To this end, we introduce "Antidote", a unified, synthetic data-driven post-training framework for mitigating both types of hallucination above. It leverages synthetic data to incorporate factual priors into questions to achieve self-correction, and decouple the mitigation process into a preference optimization problem. Furthermore, we construct "CP-Bench", a novel benchmark to evaluate LVLMs' ability to correctly handle CPQs and produce factual responses. Applied to the LLaVA series, Antidote can simultaneously enhance performance on CP-Bench by over 50%, POPE by 1.8-3.3%, and CHAIR & SHR by 30-50%, all without relying on external supervision from stronger LVLMs or human feedback and introducing noticeable catastrophic forgetting issues.

Figures

Figures reproduced from arXiv: 2504.20468 by the authors.

Figure 1
Figure 1. The hallucination responses induced by CPQ. Though recent hallucination mitigation methods improves LVLMs in ob￾ject perception, while their models can still be easily deceived by CPQs and induce severe hallucinatory responses. mation, remains a significant challenge. It undermines their reliability and limits applications in sensitive domains like healthcare and autonomous systems. In LVLM, studies of hallucination… view at source ↗
Figure 2
Figure 2. Performance comparison of LVLMs on benchmarks of general capabilities (MMBench [26]), hallucination of object per￾ception (POPE [18]) and CPQ (the proposed CP-Bench). Higher values indicate better performance on corresponding benchmarks. current image but frequently appear in similar visual con￾texts, underscoring a potential limitation of existing halluci￾nation alleviation techniques. We call this type of question… view at source ↗
Figure 3
Figure 3. Examples of hallucination induced by CPQs and the synthetic samples of Antidote. The CPQs are selected from the test set of the proposed CP-Bench, which will be introduced in Section 4. “Hallucination candidates” are the non-existent objects that commonly appear in similar scenes. More examples can be viewed in Appendix. alleviate hallucination [41, 50, 51]. In contrast, the pro￾posed Antidote not only effectively i… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The data synthesis pipeline for Antidote. The pipeline consists of three stages: (a) construction of caption Pool; (b) visual scene understanding; (c) data synthesis. objects”, “generating objects with visible entities”, and “avoiding conflicts in Opre and Ohallu”. Ste…
Figure 5
Figure 5. Figure 5: Overview of the proposed Antidote post-training. The factual information from the synthetic data is seamlessly integrated into the input task prompt. The LVLMs can utilize this information to self-correct the responses as “positive” samples. For the original responses,…
Figure 6
Figure 6. Figure 6: The statistical details of CP-Bench (test) and CPQ examples. CP-Bench includes four types (i.e., scene, knowledge, item, and activity) of CPQs and TPQs from different scenes. It can comprehensively evaluate the LVLMs’ ability to discriminate the correctness of presuppo…
Figure 7
Figure 7. Figure 7: Attention visualization between the text tokens and vision tokens. The intensity of each text token’s background indicates the attention weight magnitude of image tokens, with darker highlights representing higher attention. The attention values above the 0.995th quant…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 12 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 7

    arXiv 2024

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  3. [3]

    Claude 3.5 sonnet model card adden- dum

    Anthropic. Claude 3.5 sonnet model card adden- dum. https : / / www - cdn . anthropic . com / fed9cc193a14b84131812372d8d5857f8f304c52/ Model_Card_Claude_3_Addendum.pdf, 2024. 2, 7

    work page 2024

  4. [4]

    Bge m3-embedding: Multi- lingual, multi-functionality, multi-granularity text embed- dings through self-knowledge distillation

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi- lingual, multi-functionality, multi-granularity text embed- dings through self-knowledge distillation. arXiv preprint arXiv:2402.03216, 2024. 4

    arXiv 2024

  5. [5]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 1, 2, 6, 7

    work page 2024

  6. [6]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  7. [7]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learn- ing, 2024. 3, 4

    2024

  8. [8]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools

    GLM-Team, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, 2024. 7

    arXiv 2024

Show all 51 references
  1. [9]

    Cogvlm2: Visual language mod- els for image and video understanding

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Jun- hui Ji, Zhao Xue, et al. Cogvlm2: Visual language mod- els for image and video understanding. arXiv preprint arXiv:2408.16500, 2024. 7

    2024 arXiv

  2. [10]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 6

    2021 arXiv

  3. [11]

    Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Confer...

    2024

  4. [12]

    Self-introspective de- coding: Alleviating hallucinations for large vision-language models

    Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self-introspective de- coding: Alleviating hallucinations for large vision-language models. arXiv preprint arXiv:2408.02032, 2024. 7

    2024 arXiv

  5. [13]

    A survey on locality sensitive hashing algorithms and their applica- tions

    Omid Jafari, Preeti Maurya, Parth Nagarkar, Khand- ker Mushfiqul Islam, and Chidambaram Crushev. A survey on locality sensitive hashing algorithms and their applica- tions. arXiv preprint arXiv:2102.08942, 2021. 3

    2021 arXiv

  6. [14]

    Towards mitigating llm hallucination via self reflection

    Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating llm hallucination via self reflection. In Findings of the Association for Computa- tional Linguistics: EMNLP 2023 , pages 1827–1843, 2023. 3

    2023

  7. [15]

    Hallucination augmented contrastive learn- ing for multimodal large language model

    Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learn- ing for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2024

  8. [16]

    V olcano: mitigating multimodal hallucina- tion through self-feedback guided revision

    Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Min- joon Seo. V olcano: mitigating multimodal hallucina- tion through self-feedback guided revision. arXiv preprint arXiv:2311.07362, 2023. 6, 7

    2023 arXiv

  9. [17]

    Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogn...

    2024

  10. [18]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 1, 2, 5, 6, 7

    2023 arXiv

  11. [19]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024. 3

    2024 arXiv

  12. [20]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representa- tions, 2023. 1, 2

    2023

  13. [21]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 7

    2024

  14. [22]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 6, 7

    2024

  15. [23]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 2, 6, 7

    2024

  16. [24]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 4

    2023 arXiv

  17. [25]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. 6

    2023 arXiv

  18. [26]

    Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer, 2025. 2

    2025

  19. [27]

    Deepseek-vl: towards real-world vision- language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision- language understanding. arXiv preprint arXiv:2403.05525,

    arXiv

  20. [28]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507–2521,

  21. [29]

    Hello gpt-4o

    Openai. Hello gpt-4o. https://openai.com/index/ hello-gpt-4o/, 2024. 2, 7

    2024

  22. [30]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  23. [31]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. 4

    2024

  24. [32]

    Object hallucination in image cap- tioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning. arXiv preprint arXiv:1809.02156, 2018. 1, 2, 6

    2018 arXiv

  25. [33]

    Scienceqa: A novel resource for question answering on scholarly articles

    Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3):289–301, 2022. 6

    2022

  26. [34]

    Proximal policy optimization algo- rithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization algo- rithms. arXiv preprint arXiv:1707.06347, 2017. 4

    2017 arXiv

  27. [35]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2018

  28. [36]

    Intervening anchor token: Decod- ing strategy in alleviating hallucinations for mllms

    Feilong Tang, Zile Huang, Chengzhi Liu, Qiang Sun, Harry Yang, and Ser-Nam Lim. Intervening anchor token: Decod- ing strategy in alleviating hallucinations for mllms. In The Thirteenth International Conference on Learning Represen- tations. 2

  29. [37]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 3

    2023 arXiv

  30. [38]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2, 6, 7

    2024 arXiv

  31. [39]

    Mitigating hallucinations in large vision-language models with instruction contrastive decoding

    Xintong Wang, Jingheng Pan, Liang Ding, and Chris Bie- mann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. arXiv preprint arXiv:2403.18715, 2024. 2

    2024 arXiv

  32. [40]

    A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more

    Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ram- nath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Mao, Sitaram Asur, et al. A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407.16216, 2024. 4

    2024 arXiv

  33. [41]

    Detecting and mitigating hallucination in large vision language models via fine-grained ai feedback.arXiv preprint arXiv:2404.14233, 2024

    Wenyi Xiao, Ziwei Huang, Leilei Gan, Wanggui He, Haoyuan Li, Zhelun Yu, Hao Jiang, Fei Wu, and Linchao Zhu. Detecting and mitigating hallucination in large vision language models via fine-grained ai feedback.arXiv preprint arXiv:2404.14233, 2024. 1, 3

    2024 arXiv

  34. [42]

    Sayself: Teaching llms to express confidence with self-reflective rationales

    Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao. Sayself: Teaching llms to express confidence with self-reflective rationales. arXiv preprint arXiv:2405.20974, 2024. 3

    2024 arXiv

  35. [43]

    Mmrc: A large-scale bench- mark for understanding multimodal large language model in real-world conversation

    Haochen Xue, Feilong Tang, Ming Hu, Yexin Liu, Qidong Huang, Yulong Li, Chengzhi Liu, Zhongxing Xu, Chong Zhang, Chun-Mei Feng, et al. Mmrc: A large-scale bench- mark for understanding multimodal large language model in real-world conversation. arXiv preprint arXiv:2502.11903 ,

    arXiv

  36. [44]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024. 3

    2024 arXiv

  37. [45]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 1, 6, 7

    2024 arXiv

  38. [46]

    Woodpecker: Hallucination correction for multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310.16045 ,

    arXiv

  39. [47]

    Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data

    Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wen- tao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2024

  40. [48]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 6

    2023 arXiv

  41. [49]

    Debiasing large visual language models

    Yi-Fan Zhang, Weichen Yu, Qingsong Wen, Xue Wang, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Debiasing large visual language models. arXiv preprint arXiv:2403.05262, 2024. 6

    2024 arXiv

  42. [50]

    Beyond hallucinations: Enhanc- ing lvlms through hallucination-aware direct preference op- timization

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Ji- aqi Wang, and Conghui He. Beyond hallucinations: Enhanc- ing lvlms through hallucination-aware direct preference op- timization. arXiv preprint arXiv:2311.16839, 2023. 1, 2, 3, 6, 7

    2023 arXiv

  43. [51]

    Self- supervised visual preference alignment

    Ke Zhu, Liang Zhao, Zheng Ge, and Xiangyu Zhang. Self- supervised visual preference alignment. arXiv preprint arXiv:2404.10501, 2024. 3, 6, 7

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.