Pith. sign in

REVIEW 4 major objections 4 minor 49 references

Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding

T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Editing a vision-language model's internal representations to amplify its own hallucinations, then subtracting those hallucinatory logits, reduces object and attribute hallucination at inference time.

desk verdict Clever internal-editing contrastive decoding, but the paper's own ablation contradicts the mechanism and the headline numbers don't match the tables. read the letter →

arxiv 2502.01056 v1 pith:HO4BZO2C submitted 2025-02-03 cs.CV cs.CL

Chao Wang , Xuancheng Zhou , Weiwei Fu , Yang Zhou This is my paper · ORCID
classification cs.CVcs.CL
keywords largevision-languagemodelsobjecthallucinationcontrastivedecodinginternalrepresentationeditingtruthfulnessprobePOPEbenchmarkMMEinference-timemitigation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a vision-language model's own hallucinations can be used against themselves: by editing the model's internal representations to deliberately amplify its hallucinatory tendencies, one can obtain a distribution of 'hallucinatory logits' that, when subtracted from the normal output, removes false objects and attributes at inference time. The method, called Internal Fact-based Contrastive Decoding (IFCD), requires only a small trained probe and no fine-tuning of the base model or external knowledge sources. On the POPE and MME object-hallucination benchmarks the authors report average accuracy gains of about 9% and 8% over direct decoding across two open models, LLaVA 1.5 and InstructBLIP. If correct, this offers a cheap, model-agnostic route to reducing hallucinations in deployed vision-language systems.

What carries the argument

The central mechanism is a learned truthfulness-direction edit combined with contrastive decoding. TruthX is an autoencoder with two multi-layer-perceptron encoders, TruthEnc and SemEnc, and a decoder; it maps an LVLM's internal hidden states into separate truthfulness and semantic latent spaces, reconstructs them, and the difference between reconstructed and original states gives an editing vector $\Delta$. Editing the hidden state as $\hat{x} = x + \gamma \Delta$ with positive $\gamma$ is claimed to make the model's output more truthful, while negative $\gamma$ amplifies language-prior hallucinations such as calling black strawberries red. Contrastive decoding then subtracts the hallucinated distribution from the positive one using the formula above, and an adaptive plausibility constraint restricts the penalty to tokens whose normal probability is above a fraction $\beta$ of the maximum, keeping the contrast from distorting already-correct tokens.

What would settle it

Measure token-level probabilities from p^+ and p^- on a diverse set of images with known ground-truth objects: if p^- does not assign systematically higher probability than p^+ to hallucinated objects (the very tokens that should be suppressed), the contrastive subtraction cannot be the cause of the reported gains. A simpler check would be to run the reported ablation 'w/o POS' (negative editing contrasted against the original, unedited distribution) across all POPE and MME settings: on InstructBLIP's CHAIR_i the paper already reports it beating full IFCD, so if that pattern holds across benchmarks the positive direction is not doing the claimed work.

Watch

Extended reading notes

Core claim

IFCD builds two token distributions from the same LVLM: a positive distribution $p^+$ obtained by editing the model's internal representations in a 'truthful' direction, and a negative distribution $p^-$ obtained by editing in the opposite, hallucination-amplifying direction. The final decoding distribution is computed as $p_{\text{IFCD}}(y_t) = \sigma((1+\alpha)p^+(y_t) - \alpha p^-(y_t))$, so tokens that are boosted by the hallucinated distribution are penalised. The editing direction comes from an autoencoder, TruthX, trained with contrastive learning on a few hundred image-caption pairs to separate truthful from untruthful internal states; the difference between the average truthful and untruthful latent positions defines the editing vector $\delta$, and the edited representation is $\hat{x} = x + \gamma \Delta$. The paper's experiments claim this reduces object-existence and attribute hallucinations on POPE, MME, and long-form captioning (CHAIR), and that the method degrades less than baselines when POPE's sampling becomes adversarial.

Load-bearing premise

The load-bearing premise is that editing a model's internal representation along the learned 'truthfulness' direction reliably makes its outputs more truthful, and editing in the opposite direction reliably makes them more hallucinated, so that the subtracted distribution p^- really consists of hallucinatory logits rather than random noise.

Editorial extensions

If this is right

  • IFCD can be applied to any open-source LVLM with a small trained probe, requiring no fine-tuning of the base model and no external knowledge sources.
  • The method reduces both object-existence and attribute-level hallucinations, with reported average accuracy gains of 9% on POPE and 8% on the MME object-hallucination subset across LLaVA 1.5 and InstructBLIP.
  • In long-form captioning on MSCOCO, IFCD lowers the proportion of hallucinated objects (CHAIR_i) and hallucinated sentences (CHAIR_s) compared with direct decoding, while keeping BLEU roughly unchanged.
  • IFCD's accuracy drop from POPE's random to adversarial sampling settings is smaller than that of direct decoding, VCD, and ICD, suggesting it is more robust to statistical bias.
  • Training the TruthX probe requires only about 300 image-caption pairs; larger training sets did not improve POPE accuracy in the reported experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the method is model-agnostic and the probe is small, the same two-model contrastive decoding recipe could transfer to other modalities, such as editing internal states of audio or video-language models to amplify modality-specific priors.
  • The reported ablation that contrasting the original distribution with the negative-edited distribution alone ('w/o POS') matches or exceeds the full IFCD on some CHAIR scores suggests that the positive editing direction may not be strictly necessary; a focused study of when the positive edit helps would clarify the mechanism.
  • The performance peak at 300 training samples, with declines at larger sizes, hints that the truthfulness direction is learned from a narrow data distribution; testing on diverse domains (medical, scientific imagery) would reveal whether the direction generalises.
  • One could test the method's central premise directly by measuring, at the token level, whether $p^-$ assigns higher probability to known hallucinated objects than $p^+$; such a probe would make the internal mechanism falsifiable without a full benchmark run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Internal Fact-based Contrastive Decoding (IFCD), an inference-time method for reducing object hallucinations in LVLMs. IFCD trains a TruthX-style probe to identify a truthful/untruthful editing direction in hidden representations, then forms two distributions p+ and p− by editing internal representations in opposite directions and decodes with Eq. (5): p_IFCD = σ((1+α)p+ − αp−). The authors evaluate on POPE, MME, MSCOCO captioning, and LLaVA-Bench with LLaVA 1.5 and InstructBLIP, reporting gains over regular decoding and the VCD/ICD baselines.

Significance. Hallucination mitigation at inference time with a small trained probe is a practically important and timely goal, and the paper has the right experimental skeleton: two backbone LVLMs, several benchmarks, and an ablation study. The central mechanistic claim, however, depends on the direction δ in Eq. (3) being a reliable hallucination amplifier; the paper's own Table 4 and Figure 3 contradict that premise. Because the advertised gains are also obtained under hyperparameters selected on the evaluation benchmarks themselves (Appendix A, Section 5), the paper does not currently establish that IFCD works through 'internal facts' rather than through generic contrastive effects or benchmark-specific tuning.

major comments (4)
  1. [Section 5, Table 4] The ablation undermines the central premise of Eq. (5). The method assumes that negative editing produces a hallucination-amplified distribution p− whose subtraction removes hallucinatory logits. On InstructBLIP, however, 'EDITING' (positive editing alone) yields CHAIRs/CHAIRi = 57/15, which is worse than Regular decoding (48/13.9), so positive editing does not act as an anti-hallucination operation in this backbone. Moreover, 'IFCD w/o POS' (contrasting negative editing with the original distribution) achieves 28/7.6, better than full IFCD (39.6/11.2), so the component identified as the core contrast is also not the one responsible for the best result. The table therefore does not establish that p− is a hallucination distribution or that Eq. (5) calibrates outputs according to internal facts.
  2. [Figure 3] Figure 3 contradicts its own caption. For black strawberries, the panel labeled 'Hallucinated Editing' reports logits Red 3.2, Black 14.5, Gray 2.4, which still favor 'Black' over 'Red'; the caption instead claims that editing amplifies the language prior toward the conventional color 'red'. Since this figure is the direct evidence that internal representation editing amplifies hallucination, the direction assumption behind Eqs. (3)–(5) is not supported by the displayed numbers.
  3. [Appendix A and Section 5] The headline results are not independent predictions because the hyperparameters are selected on the evaluation benchmarks. Appendix A selects the TruthX training size (300) by POPE accuracy on the MSCOCO subset, and Section 5 selects editing strength (0.5), the number of edited layers (15), and contrast strength α (0.1) through CHAIR-style sweeps on MSCOCO (Figures 6 and 7). The same POPE and CHAIR/MSCOCO settings are then reported as the main results in Tables 1 and 3. The paper needs a separate validation split or a sensitivity analysis showing that the gains are not artifacts of this selection.
  4. [Abstract and Table 1] The abstract claims an average 9% accuracy improvement on POPE, but Table 1 does not contain an average gain of 9 points: the largest per-setting gain over 'Regular' is 7.44 points (InstructBLIP, GQA adversarial), and many gains are between 1.6 and 7.3 points. If the '9%' refers to a different statistic (for example, a relative improvement or a subset of settings), the text should state that explicitly rather than presenting it as the average POPE accuracy gain.
minor comments (4)
  1. [Appendix A] The sentence 'As shown in the left part of Figure 5' appears to refer to the training-size comparison, but Figure 5 is the max-token CHAIR experiment; the reference should point to Table 5.
  2. [Appendix B] There are several typos: 'Regrading' should be 'Regarding', 'DIRTURBANCE' should be 'DISTURBANCE' (also in Figure 3), and Table 3's caption says 'resutls' instead of 'results'.
  3. [Section 4.1 and Tables 1–3] No standard errors, multiple seeds, or significance tests are reported for the POPE/CHAIR numbers. Given that temperatures are set to 1, the generation is stochastic and the reported differences between close configurations may be within noise.
  4. [Section 3.2] The sign convention for editing is confusing: Eq. (3) defines δ = H_pos_truth − H_neg_truth, while Figure 2 labels −δ as 'Neg Editing hallucinated' and +δ as 'Pos Editing anti-hallucinated'; these two conventions should be reconciled in one place.

Circularity Check

3 steps flagged · score 6.0 of 10

Main CHAIR and MSCOCO-POPE improvements are fitted by selecting hyperparameters and TruthX training size on the same benchmark rows; P−'s 'hallucination-inducing' label is asserted, making Eq. 5's hallucination removal definitional.

  1. fitted input called prediction [Section 5 (Figures 6 and 7), Appendix B Table 7, Table 3 (MSCOCO CHAIR)]
    "Figure 6 presents the impact of editing strength and layers when editing the internal representation on the effectiveness of IFCD in hallucinations mitigation. ... As shown in the left part of Figure 7, the small α leads top performance, denoting the gap of distributions involved in contrastive decoding is striking. ... we modify only the top 15 most important layers, with the editing strength s = 0.5."

    Editing strength (0.5), number of layers (15), and contrastive strength (α=0.1) are selected as the CHAIR-optimal values on MSCOCO (Figures 6–7, Table 7). Table 3 then reports CHAIRs and CHAIRi on MSCOCO as the main long-generation result. The reported improvement is therefore the selection criterion itself, not an independent prediction: Table 3 is a fit to the same metric and dataset used for tuning.

  2. fitted input called prediction [Appendix A (Table 5), Section 4.1 Implementation Details, Table 1 (MSCOCO POPE rows)]
    "We compare the performance of IFCD with varying training sizes on the MSCOCO subset of POPE, utilizing three different POPE sampling strategies. ... the overall best performance is achieved when the training data size reaches 300 in POPE. ... We use 300 MSCOCO images paired with both correct and incorrect responses as the training dataset for TruthX."

    The 300-sample TruthX training size is chosen by maximizing accuracy on the MSCOCO subset of POPE (Appendix A). The same MSCOCO POPE rows in Table 1 are then reported as evidence of IFCD's improvement (e.g., 89.17 vs 83.29 on random sampling for LLaVA 1.5). Thus the MSCOCO POPE results are a selected maximum, not a held-out prediction; only the A-OKVQA and GQA POPE rows are partly independent of this selection.

1 more flagged steps
  1. self definitional [Section 3.3 Eq. 5, Section 5 Table 4, Figure 3]
    "In IFCD, we designate the distribution that undergoes anti-hallucinations as P+ and the distribution outputted by hallucination-inducing as P−. ... pIFCD(yt|xv, xq) = σ((1+α)p+(yt|∗) − αp−(yt|∗))."

    The claim that Eq. 5 removes hallucinatory logits is a restatement of the label assigned to P− ('hallucination-inducing'). The paper's own evidence fails to establish that label: Table 4 shows positive editing alone (EDITING) is worse than Regular on InstructBLIP (57/15 vs 48/13.9), and 'w/o POS' (negative editing + original) outperforms full IFCD (28/7.6 vs 39.6/11.2), so P+ is not reliably anti-hallucinating and the named mechanism is not the best variant. Figure 3's 'Hallucinated Editing' panel also raises Black (14.5) rather than the conventional Red (3.2). Hence the 'hallucination removal' outcome is definitional, not derived from an independently established P−.

full rationale

The paper does not rely on self-citation or imported uniqueness theorems: TruthX is an external prior work, and the contrastive-decoding idea is standard. However, the two strongest empirical claims are partially circular by construction. First, the CHAIR improvements on MSCOCO (Table 3) are produced with hyperparameters (editing strength, layer count, contrast strength) explicitly chosen by CHAIR sweeps on the same MSCOCO benchmark (Figures 6–7), so those numbers are fitted rather than predicted. Second, the TruthX training size of 300 is selected as the best performer on the MSCOCO subset of POPE (Appendix A), and the same MSCOCO POPE rows appear in Table 1 as headline results. The MME results and the A-OKVQA/GQA POPE rows provide some independent support, which prevents a higher score. Additionally, the central mechanism is defined rather than demonstrated: P− is called 'hallucination-inducing' by fiat, while Table 4 and Figure 3 contradict the assumed direction, making Eq. 5's 'removal of hallucinatory logits' a definitional restatement of that label. Overall score 6: important reported predictions reduce to benchmark selection or to a labeled input, while some independent evaluation remains.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The method rests on a small set of hand-chosen hyperparameters, several of which are tuned on the evaluation benchmarks, and on the domain assumption that internal-state editing produces a clean hallucination direction. The truthful and untruthful training pairs are an external anchor, but their construction is under-specified. No new physical or architectural entities are introduced.

free parameters (5)
  • Editing strength gamma = 0.5
    Selected from the CHAIR sweep in Figure 6 on MSCOCO captions, the same benchmark used for the main MSCOCO results; it controls how far hidden states are moved along the truthfulness direction.
  • Number of edited layers = 15
    Selected from the top-layer sweep in Figure 6; the paper does not define how layers are ranked by importance.
  • Contrastive decoding strength alpha = 0.1
    Selected from Figure 7 as the small alpha that yields best CHAIR on MSCOCO; the paper notes only a minimal contrastive strength is needed.
  • Adaptive plausibility threshold beta = 0.1
    Taken from prior contrastive decoding work (Li et al., 2023c) and fixed for all experiments; still a hand-chosen hyperparameter.
  • TruthX training set size = 300 image-text pairs
    Appendix A (Table 5) selects 300 as the size with the best POPE MSCOCO accuracy; larger training sets perform worse, so this is a test-set-tuned value.
assumptions (6)
  • domain assumption The TruthX-derived direction delta = H_pos_truth - H_neg_truth (Eq. 3) separates truthful from untruthful LVLM hidden states.
    Section 3.2 adopts an LLM-oriented truthfulness editor for LVLM representations; no independent validation beyond the PCA plot in Figure 7 is provided.
  • domain assumption Editing hidden states by +/- gamma*Delta amplifies or suppresses object hallucinations in the intended direction.
    Section 3.2 and Figure 3; the figure's numbers appear inconsistent with this assumption, and Table 4's positive-editing-only results worsen hallucinations.
  • domain assumption Subtracting the hallucination-amplified distribution from the normal distribution removes hallucinatory logits without harming factual tokens.
    Section 3.3, Eq. 5; this is the core mechanism, asserted rather than derived or ablated to a definitive conclusion.
  • domain assumption The truthful and untruthful labels for the 300 training pairs in Appendix A are correct.
    The paper states correct and incorrect responses are paired with MSCOCO images but does not describe how incorrect responses are generated or verified.
  • domain assumption POPE, MME, and CHAIR metrics faithfully measure object hallucination.
    Standard benchmarks adopted without discussion of their known limitations, such as POPE's simple existence questions.
  • standard math Standard probability and contrastive decoding operations, such as softmax and head truncation, behave as expected.
    Eqs. 5-7 assume standard contrastive decoding mechanics from Li et al. 2023.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding." pith.science (2026). https://pith.science/paper/HO4BZO2C

@misc{pith2026250201056,
  author       = {Pith},
  title        = {Pith review of: Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HO4BZO2C}},
  note         = {Machine review of arXiv:2502.01056}
}
read the original abstract

Large Visual Language Models (LVLMs) integrate visual and linguistic modalities, exhibiting exceptional performance across various multimodal tasks. Nevertheless, LVLMs remain vulnerable to the issue of object hallucinations. Previous efforts to mitigate this issue focus on supervised fine-tuning (SFT) or incorporating external knowledge, both of which entail significant costs related to training and the acquisition of external data. To address these challenges, we propose a novel model-agnostic approach termed Internal Fact-based Contrastive Decoding (IFCD), designed to mitigate and suppress hallucinations during the inference process of LVLMs by exploiting the LVLMs' own hallucinations. IFCD is grounded in experimental observations that alterations to the LVLMs' internal representations tend to amplify hallucinations caused by language bias. By contrasting disturbed distribution, IFCD calibrates the LVLMs' output and effectively removes the hallucinatory logits from the final predictions. Experimental results validate that IFCD significantly alleviates both object-level and attribute-level hallucinations while achieving an average 9% accuracy improvement on POPE and 8% accuracy improvement on MME object hallucinations subset compared with direct decoding, respectively.

Figures

Figures reproduced from arXiv: 2502.01056 by the authors.

Figure 1
Figure 1. Cases of object hallucinations and effect of IFCD on LLaVA 1.5. Given two images, an LLaVA 1.5 outputs responses with attribute and category hallucinations which IFCD fixes. 2023). These models are increasingly viewed as a step to￾ward achieving artificial general intelligence (Sanderson, 2023). LVLMs are capable of extracting intricate complex visual information and transforming it into continuous lan￾guage represe… view at source ↗
Figure 2
Figure 2. An overview of IFCD. IFCD first edits the internal representation of the LVLMs to construct counterfactual logits for comparison by deliberately injecting hallucinations into the model trained by contrastive learning. These counterfactual logits are utilized to reveal potential hallucinatory tendencies of the LVLMs. Furthermore, the internal representation editing model is employed to actively attenuate a portion of… view at source ↗
Figure 3
Figure 3. An illustration of editing internal representation amplify￾ing language priors. Given an image depicting three black straw￾berries, LVLMs assign more preference for more conventional strawberry color, such as “red”, with increasing editing strength. 3.3. Internal Fact-based Contrastive Decoding Contrasting the Predictions with Disturbance The find￾ings from our previous analyses substantiate the hypothesis that mani… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Perception subset of MME results on LLaVA 1.5. Reg￾ular denotes the direct sampling method, whereas ICD refers to the Instruction Contrastive Decoding, VCD refers to the Visual Contrastive Decoding baseline and IFCD is a sampling from our proposed contrastive decoding.…
Figure 5
Figure 5. Figure 5: Comparison IFCD and regular decoding on the ratio of hallucination objects (CHAIRi) with respect to the number of max tokens. IFCD maintains a low ratio of hallucination objects while increasing the number of objects [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: IFCD performance with different contrast strengths and the capacity of identifying truthfulness. The order of magnitude of the PCA figure is 1e-7. The Capacity of Editing Internal Representation To in￾vestigate the effect of editing internal representations, we ex￾plor…
Figure 6
Figure 6. Figure 6: CHAIR scores vary with editing strength and layers [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: LLaVA-Bench results comparing direct decoding, ICD, VCD, and IFCD with InstructBLIP backbone. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: LLaVA-Bench results comparing direct decoding, ICD, VCD, and IFCD with LLaVA 1.5 backbone. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 21 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Self-rag: Self-reflective retrieval augmented generation

    Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. Self-rag: Self-reflective retrieval augmented generation. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023

    work page 2023

  4. [4]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  5. [5]

    F., G \'o mez, L., and Karatzas, D

    Biten, A. F., G \'o mez, L., and Karatzas, D. Let there be a clock on the beach: Reducing object hallucination in image captioning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.\ 1381--1390, 2022

    work page 2022

  6. [6]

    In-context sharpness as alerts: An inner representation perspective for hallucination mitigation

    Chen, S., Xiong, M., Liu, J., Wu, Z., Xiao, T., Gao, S., and He, J. In-context sharpness as alerts: An inner representation perspective for hallucination mitigation. In ICLR 2024 Workshop on Reliable and Responsible Foundation Models, 2024 a

    work page 2024

  7. [7]

    In-context sharpness as alerts: An inner representation perspective for hallucination mitigation

    Chen, S., Xiong, M., Liu, J., Wu, Z., Xiao, T., Gao, S., and He, J. In-context sharpness as alerts: An inner representation perspective for hallucination mitigation. In ICLR 2024 Workshop on Reliable and Responsible Foundation Models, 2024 b . URL https://openreview.net/forum?id=24U6vAHnYM

    work page 2024

  8. [8]

    Halc: Object hallucination reduction via adaptive focal-contrast decoding

    Chen, Z., Zhao, Z., Luo, H., Yao, H., Li, B., and Zhou, J. Halc: Object hallucination reduction via adaptive focal-contrast decoding. In Forty-first International Conference on Machine Learning, 2024 c

    work page 2024

Show all 49 references
  1. [9]

    E., et al

    Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., et al. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2 0 (3): 0 6, 2023

    2023

  2. [10]

    Chain-of-verification reduces hallucination in large language models

    Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., and Weston, J. Chain-of-verification reduces hallucination in large language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2...

    2024 doi

  3. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on...

    2021

  4. [12]

    Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., and Florence...

    2023

  5. [13]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  6. [14]

    A., and Gal, Y

    Han, J., Kossen, J., Razzak, M., Schut, L., Malik, S. A., and Gal, Y. Semantic entropy probes: Robust and cheap hallucination detection in llms. In ICML 2024 Workshop on Foundation Models in the Wild, 2024

    2024

  7. [15]

    Knowledge-centric hallucination detection

    Hu, X., Ru, D., Qiu, L., Guo, Q., Zhang, T., Xu, Y., Luo, Y., Liu, P., Zhang, Y., and Zhang, Z. Knowledge-centric hallucination detection. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process...

    2024 doi

  8. [16]

    J., Madotto, A., and Fung, P

    Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. Survey of hallucination in natural language generation. ACM Computing Surveys, 55 0 (12): 0 1--38, 2023

    2023

  9. [17]

    Mitigating object hallucinations in large vision-language models through visual contrastive decoding

    Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., and Bing, L. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 13872--13882, 2024

    2024

  10. [18]

    Li, D., Li, J., Le, H., Wang, G., Savarese, S., and Hoi, S. C. LAVIS : A one-stop library for language-vision intelligence. In Bollegala, D., Huang, R., and Ritter, A. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: Sy...

    2023 doi

  11. [19]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp.\ 19730--19742. PMLR, 2023 b

    2023

  12. [20]

    L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, L., and Lewis, M

    Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, L., and Lewis, M. Contrastive decoding: Open-ended text generation as optimization. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Asso...

    2023 doi

  13. [21]

    Evaluating object hallucination in large vision-language models

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, X., and Wen, J.-R. Evaluating object hallucination in large vision-language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 292--305, S...

    2023 doi

  14. [22]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (eds.), Computer Vision -- ECCV 2014, pp.\ 740--755, Cham, 2014. Spri...

    2014

  15. [23]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36, 2024

    2024

  16. [24]

    Factual confidence of LLM s: on reliability and robustness of current estimators

    Mahaut, M., Aina, L., Czarnowska, P., Hardalov, M., M \"u ller, T., and Marquez, L. Factual confidence of LLM s: on reliability and robustness of current estimators. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association f...

    2024 doi

  17. [25]

    Mitigating hallucinations in lvlms via summary-guided decoding

    Min, K., Kim, M., Lee, K.-i., Lee, D., and Jung, K. Mitigating hallucinations in lvlms via summary-guided decoding. In Neurips Safe Generative AI Workshop 2024, 2024

    2024

  18. [26]

    K., and Sankarasubbu, M

    Pal, A., Umapathi, L. K., and Sankarasubbu, M. M ed- HALT : Medical domain hallucination test for large language models. In Jiang, J., Reitter, D., and Deng, S. (eds.), Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pp.\ 314--334, Singap...

    2023 doi

  19. [27]

    Towards unified multimodal editing with enhanced knowledge collaboration

    Pan, K., Fan, Z., Li, J., Yu, Q., Fei, H., Tang, S., Hong, R., Zhang, H., and Sun, Q. Towards unified multimodal editing with enhanced knowledge collaboration. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/for...

    2024

  20. [28]

    B leu: a method for automatic evaluation of machine translation

    Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. B leu: a method for automatic evaluation of machine translation. In Isabelle, P., Charniak, E., and Lin, D. (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp.\ 311--318, Philad...

    2002

  21. [29]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PMLR, 2021

    2021

  22. [30]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  23. [31]

    A., Burns, K., Darrell, T., and Saenko, K

    Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., and Saenko, K. Object hallucination in image captioning. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.\ 4035...

    2018 doi

  24. [32]

    A comprehensive survey of hallucination in large language, image, video and audio foundation models

    Sahoo, P., Meharia, P., Ghosh, A., Saha, S., Jain, V., and Chadha, A. A comprehensive survey of hallucination in large language, image, video and audio foundation models. Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 11709--11724, 2024

    2024

  25. [33]

    Gpt-4 is here: what scientists think

    Sanderson, K. Gpt-4 is here: what scientists think. Nature, 615 0 (7954): 0 773, 2023

    2023

  26. [34]

    Multilingual fact-checking using llms

    Singhal, A., Law, T., Kassner, C., Gupta, A., Duan, E., Damle, A., and Li, R. Multilingual fact-checking using llms. In Proceedings of the Third Workshop on NLP for Positive Impact, pp.\ 13--31, 2024

    2024

  27. [35]

    and Huang, S

    Song, Z. and Huang, S. Hscl-rl: Mitigating hallucinations in multimodal large language models. In NeurIPS 2024 Workshop on Open-World Agents, 2024

    2024

  28. [36]

    Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33: 0 3008--3021, 2020

    2020

  29. [37]

    Mitigating hallucinations in large vision-language models with instruction contrastive decoding

    Wang, X., Pan, J., Ding, L., and Biemann, C. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 15840--15...

    2024 doi

  30. [38]

    W., Lester, B., Du, N., Dai, A

    Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=gEZrGCozdqR

    2022

  31. [39]

    On the road with gpt-4v (ision): Explorations of utilizing visual-language model as autonomous driving agent

    Wen, L., Yang, X., Fu, D., Wang, X., Cai, P., Li, X., Tao, M., Li, Y., Linran, X., Shang, D., et al. On the road with gpt-4v (ision): Explorations of utilizing visual-language model as autonomous driving agent. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024

    2024

  32. [40]

    Transformers: State-of-the-art natural language processing

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Transformers: State-of-the-art natural language processing. EMNLP 2020, pp.\ 38, 2020

    2020

  33. [41]

    Mm-react: Prompting chatgpt for multimodal reasoning and action

    Yang, Z., Li, L., Wang, J., Lin, K., Azarnasab, E., Ahmed, F., Liu, Z., Liu, C., Zeng, M., and Wang, L. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, 2023

    2023 arXiv

  34. [42]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Ye, Q., Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q., Zhang, J., and Huang, F. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 1304...

    2024

  35. [43]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Ye, Q., Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q., Zhang, J., and Huang, F. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 13040--1305...

    2024

  36. [44]

    Truthx: Alleviating hallucinations by editing large language models in truthful space

    Zhang, S., Yu, T., and Feng, Y. Truthx: Alleviating hallucinations by editing large language models in truthful space. In Proceedings of the 62th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics,...

    2024 arXiv

  37. [45]

    Siren's song in the ai ocean: a survey on hallucination in large language models

    Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. Siren's song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023

    2023 arXiv

  38. [46]

    Mitigating object hallucination in large vision-language models via image-grounded guidance

    Zhao, L., Deng, Y., Zhang, W., and Gu, Q. Mitigating object hallucination in large vision-language models via image-grounded guidance. In Neurips Safe Generative AI Workshop 2024, 2024

    2024

  39. [47]

    X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al

    Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023

    2023 arXiv

  40. [48]

    Analyzing and mitigating object hallucination in large vision-language models

    Zhou, Y., Cui, C., Yoon, J., Zhang, L., Deng, Z., Finn, C., Bansal, M., and Yao, H. Analyzing and mitigating object hallucination in large vision-language models. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023

    2023

  41. [49]

    Mini GPT -4: Enhancing vision-language understanding with advanced large language models

    Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Mini GPT -4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=1tZbq88f27

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.