Pith. sign in

REVIEW 3 major objections 7 minor 72 references

Hallucinated captions boost VLM accuracy up to 14.68%

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 08:36 UTC pith:FOPD4DAX

load-bearing objection Hallucinated captions can improve VLM task accuracy — a real finding, but the causal story is underdetermined by a confound the paper doesn't fully close. the 3 major comments →

arxiv 2607.07507 v1 pith:FOPD4DAX submitted 2026-07-08 cs.CV cs.AI

HIVE: Understanding Post-Hallucination Reasoning in Vision Language Models

Feng He , Zhenting Wang , Qifan Wang , Qiang Guan , Dongfang Liu , Ruixiang Tang , Qiankun Li This is my paper
classification cs.CV cs.AI
keywords reasoninghallucinatedinferencehallucinationhivelanguagemodelsstage
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper identifies and systematically studies Post-Hallucination Reasoning (PHR), the stage in a vision-language model's inference where hallucinated semantics have already entered the model's context and begin influencing downstream predictions. The authors introduce HIVE, an evaluation infrastructure that generates paired faithful and hallucinated captions under matched conditions, classifies them with an ensemble of three factuality verifiers using majority voting, and then measures downstream task accuracy under each condition. Across nine tasks and nine models, the central finding is that hallucinated captions systematically improve accuracy on vision-language tasks, with gains reaching +14.68% on PlantVillage and +11.76% on ISIC when using GPT-4o, while text-only tasks show limited or unstable effects. The paper argues this improvement is not noise: hallucinated captions broaden semantic coverage in the embedding space, modulate reasoning entropy in a task-dependent manner (reducing entropy on reasoning-heavy tasks to promote convergence, increasing it on open-ended tasks to support exploration), and preserve stable inference trajectories with high inter-chain consistency. An inverted-U relationship between hallucination intensity and accuracy is observed, with moderate hallucination at temperature 0.6 and 256-token budgets producing the strongest gains. Token-level ablation confirms that masking key hallucinated evidence tokens causes accuracy to drop, indicating the model genuinely relies on these cues rather than treating them as redundant.

Core claim

The paper's central discovery is that hallucinated captions function as speculative semantic anchors that expand a vision-language model's hypothesis space under partial observability, and that this expansion is causally linked to improved downstream accuracy rather than being incidental noise. The mechanism operates at three levels: at the input level, hallucinated captions produce wider variance and heavier tails in embedding distributions compared to faithful captions; at the process level, they modulate reasoning trajectory entropy in task-dependent directions; and at the output level, correct predictions correlate with higher caption entropy than incorrect predictions across all four VL

What carries the argument

HIVE (Hallucination Inference and Verification Engine): a three-module evaluation pipeline. (1) Caption Generator produces candidate captions from a unified source with identical prompts, temperature, and token budget. (2) Caption Discriminator classifies captions as faithful or hallucinated using an ensemble of three complementary verifiers (fine-grained factuality verifier, self-evaluation factuality verifier, paraphrase-consistency verifier) with majority voting, achieving approximately 82-84% accuracy. (3) Task Solver concatenates the original input, one caption, and a task instruction to produce predictions under three conditions: Raw, +Faithful, and +Hallucinatory. The isolated halluc

Load-bearing premise

The paper assumes that its hallucination discriminator, an ensemble of three verifiers with majority voting achieving roughly 82-84% accuracy, cleanly separates faithful from hallucinated captions, and that the observed accuracy differences are causally attributable to hallucination content rather than to systematic differences in caption length, specificity, or semantic richness that co-occur with the hallucination label. The discriminator's 16-18% error rate means a non-tr

What would settle it

Mask the specific hallucinated tokens within captions that the model relied on for correct predictions, then re-evaluate. If accuracy drops substantially after masking, hallucinated content is causally necessary. The paper performs this ablation and reports accuracy drops from near-perfect to 0.244-0.700 across datasets, which is the strongest single piece of evidence that hallucinated cues are functionally load-bearing rather than incidental noise.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If hallucinated semantics genuinely improve reasoning under partial observability, then the dominant strategy of detecting and suppressing all hallucinations at generation time may be discarding useful speculative cues that could aid downstream task performance.
  • The modality-dependent pattern, where VL tasks benefit substantially but text-only tasks do not, suggests that hallucination utility is tied to the perceptual gap between what a model can observe and what it needs to know, making hallucination a form of speculative perception-filling rather than pure error.
  • The inverted-U relationship between hallucination intensity and accuracy implies that a principled mechanism for calibrating hallucination strength, rather than binary suppression, could become a standard inference-time tuning lever alongside temperature and token budget.
  • The finding that intermediate-scale models (e.g., Qwen2.5-VL-32B at +9.4%) benefit more from hallucinated cues than very large models (Qwen2.5-VL-72B at -4.9%) suggests that hallucination utility interacts with model capacity, with larger models potentially saturating on faithful inputs and finding hallucinations distracting.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's hallucination discriminator operates at 82-84% accuracy, meaning roughly 16-18% of captions are misclassified. If misclassified captions systematically differ in length, specificity, or semantic richness from correctly classified ones, the observed accuracy gap between 'hallucinated' and 'faithful' conditions could be partially or fully attributable to these confounds rather than to ha
  • The observation that hallucinated captions broaden semantic coverage and that broader coverage correlates with correct predictions suggests a simpler hypothesis the paper does not fully rule out: any caption that introduces additional task-relevant semantic content, whether hallucinated or merely more detailed faithful description, might produce similar gains. A controlled comparison between hallu
  • The inverted-U pattern and the temperature optimum at 0.6 are consistent with a general exploration-exploitation tradeoff in stochastic decoding rather than a phenomenon specific to hallucination. If so, similar accuracy curves might be obtained by interpolating between two faithful captions of varying detail, without any hallucination component at all.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper introduces Post-Hallucination Reasoning (PHR), the stage in which hallucinated semantics from VLM captions enter the inference context and influence downstream predictions. The authors present HIVE, an evaluation infrastructure that generates paired faithful (F) and hallucinated (H) captions, classifies them via an ensemble discriminator, and measures downstream accuracy differences across 9 tasks and 9 models. The central empirical finding is that hallucinated captions systematically improve accuracy on vision-language tasks (e.g., +14.68% on PlantVillage, +11.76% on ISIC with GPT-4o), while text-only tasks show limited or unstable effects. Mechanistic analyses examine embedding distributions, reasoning-chain entropy, and token-level ablations to argue that hallucinated cues broaden semantic coverage rather than acting as noise. Controls include a random-checker ablation, prompt paraphrase robustness, 5-run statistics with t-tests, and a token-level masking experiment.

Significance. The paper addresses a genuinely underexplored question: what happens after hallucinated semantics enter a model's reasoning context, rather than how to detect or suppress them at generation time. The experimental design includes several commendable controls: the random-checker ablation (Table 3) rules out arbitrary filtering, the prompt robustness test (Table 4) checks paraphrase sensitivity, and 5-run statistics with paired t-tests (Table S1) provide reliability. The token-level ablation (Table 5) is a good-faith attempt at causal isolation. Public code availability and the breadth of tasks and models tested add value. The inverted-U hallucination intensity analysis (Fig. 5) provides a falsifiable prediction. The core observation—that hallucinated captions can improve VLM task accuracy—is surprising and worth investigating.

major comments (3)
  1. §4.2, Table 5 (Token-level ablation): The token-level ablation masks hallucinated evidence tokens and observes accuracy drops, but lacks a critical control condition. There is no experiment masking an equal number of non-hallucinated tokens from faithful captions or from the same hallucinated captions. Without this control, the observed accuracy drop could partly reflect generic token removal (reducing information density) rather than the specific contribution of hallucinated content. This is load-bearing for the claim that 'key hallucinated tokens act as informative cues rather than redundant noise' (§4.2). Adding a matched-length random-token masking baseline would substantially strengthen the causal attribution.
  2. §3.1, Principle ① and §4.1: The F-vs-H comparison controls generation-time parameters (prompt, temperature, token budget) but does not control for selection-time confounds. The discriminator sorts captions into F and H buckets based on factual content, and this sorting is itself a function of caption length, specificity, number of claims, and semantic richness. A caption that makes more specific claims is more likely to contain an unverifiable element and thus be labeled H, but it may also be more informative regardless of hallucination status. The paper does not report caption length distributions or specificity measures across the F and H buckets. Without controlling for these co-varying properties, the observed Δ(H−F) could partly reflect a richness/specificity confound. Reporting mean caption lengths and token counts for F vs. H captions, or matching on length, would address this.
  3. §4.2 and §S7: The discriminator operates at 81.76% (TruthfulQA) and 83.72% (manual annotation) accuracy, meaning ~16–18% of captions are misclassified. The paper acknowledges this but does not analyze the sensitivity of the main results to discriminator noise. A simple robustness check—e.g., re-running the main Table 2 comparison with only captions where all three discriminators agree (a stricter, higher-precision subset)—would demonstrate that the observed gains are not driven by the misclassified subset. This is important because the central claim depends on the F/H separation being meaningful.
minor comments (7)
  1. §4.4, Fig. 2 (right): The claim that 'correct predictions consistently exhibit higher caption entropy' is stated universally, but Table S5 shows this holds for only 4 of 9 datasets at p<0.05. On BBBP, the effect is reversed (incorrect predictions have higher entropy, p=0.032). The text should state this more precisely.
  2. §4.6, Table 6: The temperature ablation reports Δ values but not absolute F and H accuracies, making it hard to assess whether baseline accuracy shifts with temperature independently of the hallucination effect. Reporting absolute values or at least the F baseline per temperature would be appropriate.
  3. §S8: The KDE bandwidth is fixed at 0.5 and PCA reduces to 3 dimensions. These are free parameters that could affect the entropy-based claims in §4.4. A brief sensitivity analysis or justification for these choices would strengthen the analysis.
  4. Table 2: The Δ values for GQA on Claude 3 Sonnet (−0.99) and Qwen-VL-Max (−1.98) are negative, which is inconsistent with the claim that 'VLMs reliably benefit from hallucinated semantics across multiple tasks.' The text should acknowledge these exceptions more clearly rather than attributing them to 'occasional drops.'
  5. Fig. 3 (right): The inter-chain similarity analysis is reported only on PlantVillage. It would be useful to note whether this holds on other datasets or whether PlantVillage is representative.
  6. §S3, Table S3: The scaling results show non-monotonic effects (7B: −6.22%, 32B: +9.39%, 72B: −4.93%), but the std for 7B and 72B is 0.0000, suggesting only a single run. If so, this should be noted; if not, the reporting should be clarified.
  7. The term 'Post-Hallucination Reasoning' is introduced as a new concept, but the boundary between PHR and existing concepts like 'unfaithful chain-of-thought' is not sharply delineated. A brief comparison positioning PHR relative to these existing observations would help.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive report. The three major comments are all well-taken and identify genuine gaps in our experimental controls. We address each below and commit to revisions for all three.

read point-by-point responses

  1. Referee: §4.2, Table 5 (Token-level ablation): The token-level ablation masks hallucinated evidence tokens and observes accuracy drops, but lacks a critical control condition. There is no experiment masking an equal number of non-hallucinated tokens from faithful captions or from the same hallucinated captions. Without this control, the observed accuracy drop could partly reflect generic token removal (reducing information density) rather than the specific contribution of hallucinated content. This is load-bearing for the claim that 'key hallucinated tokens act as informative cues rather than redundant noise' (§4.2). Adding a matched-length random-token masking baseline would substantially strengthen the causal attribution.

    Authors: The referee is correct. The current token-level ablation (Table 5) demonstrates that masking hallucinated evidence tokens causes accuracy to drop, but without a matched-length random-token masking control, we cannot fully rule out the alternative explanation that the drop reflects generic information reduction rather than the specific contribution of hallucinated content. This is a genuine gap in our causal attribution. We will add the requested control experiment in the revision. Specifically, we will implement two additional masking conditions: (1) masking an equal number of randomly selected non-hallucinated tokens from the same hallucinated captions, and (2) masking an equal number of randomly selected tokens from faithful captions. If the accuracy drop is substantially larger when masking hallucinated evidence tokens than under these matched-length random masking baselines, this would strengthen the claim that the hallucinated tokens specifically contribute to correct predictions. If the drops are comparable, we will appropriately temper our claim. We will revise the language in §4.2 to reflect the results of this control and adjust the strength of the causal claim accordingly. revision: yes

  2. Referee: §3.1, Principle ① and §4.1: The F-vs-H comparison controls generation-time parameters (prompt, temperature, token budget) but does not control for selection-time confounds. The discriminator sorts captions into F and H buckets based on factual content, and this sorting is itself a function of caption length, specificity, number of claims, and semantic richness. A caption that makes more specific claims is more likely to contain an unverifiable element and thus be labeled H, but it may also be more informative regardless of hallucination status. The paper does not report caption length distributions or specificity measures across the F and H buckets. Without controlling for these co-varying properties, the observed Δ(H−F) could partly reflect a richness/specificity confound. Reporting mean caption lengths and token counts for F vs. H captions, or matching on length, would address this.

    Authors: This is a valid and important concern. Our experimental design controls generation-time parameters (prompt, temperature, token budget) but does not explicitly control for selection-time confounds introduced by the discriminator. As the referee notes, captions labeled H may systematically differ from those labeled F in length, specificity, or number of claims, and these properties could independently affect downstream accuracy. We agree that this confound is not addressed by our current controls (random checker, prompt robustness, t-tests). In the revision, we will: (1) report mean caption lengths (in tokens) and token-count distributions for F vs. H captions across all datasets, so readers can assess the magnitude of any length/specificity difference; (2) if meaningful differences exist, conduct a length-matched sub-analysis where we subsample F and H captions to have comparable token counts and re-compute Δ(H−F); and (3) discuss the potential confound explicitly in the limitations or discussion section. We note that our existing random-checker ablation (Table 3) partially addresses the concern that arbitrary filtering drives the gains—random filtering yields negligible and non-significant changes—but we agree it does not specifically control for the richness/specificity confound. The length-matched analysis will provide a more direct test. revision: yes

  3. Referee: §4.2 and §S7: The discriminator operates at 81.76% (TruthfulQA) and 83.72% (manual annotation) accuracy, meaning ~16–18% of captions are misclassified. The paper acknowledges this but does not analyze the sensitivity of the main results to discriminator noise. A simple robustness check—e.g., re-running the main Table 2 comparison with only captions where all three discriminators agree (a stricter, higher-precision subset)—would demonstrate that the observed gains are not driven by the misclassified subset. This is important because the central claim depends on the F/H separation being meaningful.

    Authors: The referee is right that we should demonstrate sensitivity to discriminator noise more directly. While our ensemble majority-voting approach is designed to be robust under noisy classifiers, we have not shown that the main results hold when we restrict to higher-confidence classifications. We will add the suggested robustness check: re-running the main vision-language comparison (Table 2) using only caption pairs where all three discriminators agree (unanimous agreement), which should yield a higher-precision subset with fewer misclassifications. If the Δ(H−F) pattern persists on this stricter subset, it would confirm that the gains are not artifacts of discriminator errors. We will report these results as an additional table or in the appendix. We will also report the fraction of caption pairs that achieve unanimous agreement, so readers can assess how much of the data this stricter filter retains. If the gains attenuate on the unanimous subset, we will discuss this transparently and adjust our claims accordingly. revision: yes

Circularity Check

0 steps flagged

No circularity found; the derivation chain is self-contained against external benchmarks, external models, and external detection tools

full rationale

The paper's derivation chain proceeds: (1) generate captions using external LLMs with unified decoding parameters; (2) discriminate faithful vs. hallucinated captions using an ensemble of three external, established detectors (SelfCheckGPT, Factcheck-Bench, paraphrase consistency) with majority voting; (3) measure accuracy differences Δ(H−F) on external benchmarks (GQA, ISIC, PlantVillage, etc.) using external models (GPT-4o, Claude, Gemini, Qwen); (4) perform mechanistic analyses (entropy, convergence, token ablation) using independent tools (OpenCLIP embeddings, KDE). No step reduces to its inputs by construction. There are no load-bearing self-citations — all cited methods are external and independently verifiable. The token-level ablation (Table 5) selects samples where H succeeds and F fails, and the paper transparently acknowledges 'H-before is near-perfect by construction,' but the ablation result itself (accuracy drops after masking hallucinated tokens) is a genuine intervention, not a definitional identity. The discriminator's 16–18% error rate and the potential confounding of caption length/specificity with the hallucination label are validity and correctness concerns, not circularity — the paper does not define its output in terms of its input, nor does it rename a fitted parameter as a prediction. The random-checker control (Table 3) and prompt robustness test (Table 4) are independent controls that further demonstrate the results are not artifacts of the evaluation pipeline. This is a standard experimental paper with no circular derivation.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The paper introduces two conceptual entities (PHR, HIVE) that are well-grounded in the experimental design. The free parameters (T=0.6, token budget=256, KDE bandwidth=0.5, PCA dims=3) are selected post-hoc or fixed without sensitivity analysis, which is a moderate concern. The core axiom — that the discriminator reliably separates F from H — is the most load-bearing assumption and is only partially validated (82-84% accuracy).

free parameters (4)
  • Sampling temperature T = 0.6 (optimal)
    Selected by sweeping T ∈ {0.0, 0.3, 0.6, 0.9} and choosing the value that maximizes hallucination-induced gain (Table 6). This is a post-hoc parameter selection.
  • Token budget = 256 (optimal)
    Selected by sweeping {128, 256, 512, 1024} and choosing the best-performing value (Table 6).
  • KDE bandwidth = 0.5
    Fixed for entropy estimation (Appendix S8); sensitivity to this choice is not analyzed.
  • PCA dimensions for entropy analysis = 3
    Embeddings projected to 3D before KDE (Appendix S8); sensitivity not analyzed.
axioms (4)
  • domain assumption Hallucinations can be reliably separated from faithful captions by an ensemble of LLM-based verifiers with majority voting.
    The entire experimental design depends on this separation. The discriminator achieves 82-84% accuracy (Section 4.2), meaning ~16-18% of captions are misclassified, but the paper treats the F/H labels as ground truth for all downstream analysis.
  • domain assumption Accuracy differences between F and H conditions are causally attributable to hallucination content, not to confounds in caption length, specificity, or semantic richness.
    The paper does not control for caption length or information density as confounds. Hallucinated captions may systematically differ in these properties.
  • domain assumption Reasoning chain embeddings (via OpenCLIP) and their KDE-based entropy are meaningful proxies for reasoning dynamics.
    The mechanistic analysis (Section 4.3-4.4) relies on this proxy. The choice of 3D PCA projection and fixed KDE bandwidth are not validated against alternative embeddings or dimensions.
  • standard math Binary classification accuracy is a sufficient metric for evaluating reasoning quality across all nine tasks.
    All tasks are reduced to binary yes/no decisions (Appendix S2). This constrains the scope of claims about 'reasoning improvement' to binary classification, not open-ended generation.
invented entities (2)
  • Post-Hallucination Reasoning (PHR) independent evidence
    purpose: Names the stage of inference after hallucinated semantics enter the model's context.
    PHR is a conceptual framing rather than a new physical entity. It is supported by the controlled experiments showing measurable accuracy differences. The term is falsifiable: if no systematic difference existed between F and H conditions, PHR would have no empirical content.
  • HIVE (Hallucination Inference and Verification Engine) independent evidence
    purpose: Evaluation infrastructure for controlled F/H caption comparison.
    HIVE is a software pipeline, not a theoretical construct. Its code is publicly available, and its components (caption generator, discriminator, task solver) are independently reproducible.

pith-pipeline@v1.1.0-glm · 25894 in / 3403 out tokens · 666194 ms · 2026-07-09T08:36:43.904964+00:00 · methodology

0 comments
read the original abstract

Hallucinations in vision language models (VLMs) are commonly treated as semantic errors, yet they often arise from partial or ambiguous visual evidence. Prior work mainly focuses on detecting or suppressing hallucinations at generation time, leaving the subsequent reasoning stage largely unexplored. In this work, we study Post Hallucination Reasoning (PHR), the stage in which hallucinated semantics enter the model's inference context and influence downstream predictions. To systematically investigate PHR, we introduce HIVE, Hallucination Inference and Verification Engine, an evaluation infrastructure that enables controlled comparisons between faithful and hallucinated captions. Across nine tasks and nine models, we observe structured modality dependent patterns: hallucinated captions often improve accuracy on vision language tasks, while text only tasks exhibit limited or unstable effects. Further analyses show that hallucinated cues broaden semantic coverage and reshape reasoning dynamics while preserving stable inference. These findings highlight that hallucinated semantics may influence downstream reasoning once they enter the model's inference context. Understanding this post hallucination stage is important for improving the reliability and interpretability of multimodal reasoning systems. Code is publicly available at https://github.com/hefengcs/HIVE.

Figures

Figures reproduced from arXiv: 2607.07507 by Dongfang Liu, Feng He, Qiang Guan, Qiankun Li, Qifan Wang, Ruixiang Tang, Zhenting Wang.

Figure 1
Figure 1. Figure 1: Post–hallucination reasoning in VLMs. (a) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Process- and output-level analysis of hallucination effects. Left [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Inter-chain stability on the PlantVillage dataset. Left: Step-wise co￾sine similarity shows reasoning chains progressively converge during inference. Right: Hallucinated (H) and faithful (F) captions exhibit highly overlapping similarity distri￾butions, indicating that hallucinations preserve stability across sampling runs. right). These analyses provide a fine-grained view of how hallucinations affect con… view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of caption embeddings (Faithful (F) vs. Hallucinated [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Downstream task accuracy as a function of hallucination strength. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Case study on ISIC. A hallucinated (H) caption introduces a spurious vascu￾lar cue that anchors the reasoning toward a seborrheic keratosis (SK) frame, ultimately yielding the correct benign diagnosis. In contrast, the faithful (F) caption confines rea￾soning to superficial features, leading to a malignant misclassification, highlighting hallucination’s potential as constructive guidance. −5.00% on AntiCP2… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

72 extracted references · 72 canonical work pages · 8 internal anchors

  1. [1]

    Briefings in bioinformatics 22(3) (2021) 25

    Agrawal, P., Bhagat, D., Mahalwal, M., Sharma, N., Raghava, G.P.: Anticp 2.0: an updated model for predicting anticancer peptides. Briefings in bioinformatics 22(3) (2021) 25

    work page 2021

  2. [2]

    In: NAACL (2025) 3

    Bendahman, N., Pinel-Sauvagnat, K., Hubert, G., Billami, M.B.: Not all halluci- nations are good to throw away when it comes to legal abstractive summarization. In: NAACL (2025) 3

    work page 2025

  3. [3]

    IJCV134(3), 131 (2026) 4

    Chen, Z., Min, Y., Zhang, J., Yan, B., Wang, J., Wang, X., Shan, S.: A survey of multimodal hallucination evaluation and detection. IJCV134(3), 131 (2026) 4

    work page 2026

  4. [4]

    In: EMNLP (2023) 3

    Choi, S., Fang, T., Wang, Z., Song, Y.: Kcts: Knowledge-constrained tree search decoding with token-level hallucination detection. In: EMNLP (2023) 3

    work page 2023

  5. [5]

    Faithful Reasoning Using Large Language Models

    Creswell, A., Shanahan, M.: Faithful reasoning using large language models. arXiv preprint arXiv:2208.14271 (2022) 2

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  6. [6]

    In: NeurIPS

    Dong, B., Ni, M., Huang, Z., Yang, G., Zuo, W., Zhang, L.: Mirage: Assessing hallucination in multimodal reasoning chains of mllm. In: NeurIPS. vol. 38, pp. 122910–122955 (2026) 4

    work page 2026

  7. [7]

    In: NeurIPS (2024) 3

    Du, X., Xiao, C., Li, Y.: Haloscope: harnessing unlabeled llm generations for hal- lucination detection. In: NeurIPS (2024) 3

    work page 2024

  8. [8]

    In: EMNLP

    Gong, X., Ming, T., Wang, X., Wei, Z.: Damro: Dive into the attention mechanism of lvlm to reduce object hallucination. In: EMNLP. pp. 7696–7712 (2024) 1

    work page 2024

  9. [9]

    In: NeurIPS (2023) 3

    Guha, N., Nyarko, J., Ho, D.E., Ré, C., Chilton, A., Narayana, A., Chohlas- Wood, A., Peters, A., Waldon, B., Rockmore, D.N., et al.: Legalbench: a collabo- ratively built benchmark for measuring legal reasoning in large language models. In: NeurIPS (2023) 3

    work page 2023

  10. [10]

    In: CVPR

    Han, G., Lim, S.N.: Few-shot object detection with foundation models. In: CVPR. pp. 28608–28618 (2024) 1

    work page 2024

  11. [11]

    In: AAAI

    Han, J., Ren, Y., Ding, J., Yan, K., Xia, G.S.: Few-shot object detection via vari- ational feature aggregation. In: AAAI. vol. 37, pp. 755–763 (2023) 1

    work page 2023

  12. [12]

    In: NeurIPS (2022) 25

    Henderson, P., Krass, M.S., Zheng, L., Guha, N., Manning, C.D., Jurafsky, D., Ho, D.E.: Pile of law: learning responsible data filtering from the law and a 256gb open-source legal dataset. In: NeurIPS (2022) 25

    work page 2022

  13. [13]

    In: CVPR

    Hong, Y., Wu, Q., Qi, Y., Rodriguez-Opazo, C., Gould, S.: Vln bert: A recurrent vision-and-language bert for navigation. In: CVPR. pp. 1643–1653 (2021) 1

    work page 2021

  14. [14]

    In: EMNLP (2021) 3

    Honovich, O., Choshen, L., Aharoni, R., Neeman, E., Szpektor, I., Abend, O.: Q2: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. In: EMNLP (2021) 3

    work page 2021

  15. [15]

    In: EMNLP

    Hu, X., Ru, D., Qiu, L., Guo, Q., Zhang, T., Xu, Y., Luo, Y., Liu, P., Zhang, Y., Zhang, Z.: Knowledge-centric hallucination detection. In: EMNLP. pp. 6953–6975 (2024) 2

    work page 2024

  16. [16]

    In: CVPR (2019) 25

    Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: CVPR (2019) 25

    work page 2019

  17. [17]

    An open access repository of images on plant health to enable the development of mobile disease diagnostics

    Hughes, D., Salathé, M., et al.: An open access repository of images on plant health to enable the development of mobile disease diagnostics. arXiv preprint arXiv:1511.08060 (2015) 25

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  18. [18]

    ACM computing surveys55(12), 1–38 (2023) 1

    Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y.J., Madotto, A., Fung, P.: Survey of hallucination in natural language generation. ACM computing surveys55(12), 1–38 (2023) 1

    work page 2023

  19. [19]

    Language Models (Mostly) Know What They Know

    Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al.: Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221 (2022) 4 HIVE 17

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  20. [20]

    In: NeurIPS (2012) 1

    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep con- volutional neural networks. In: NeurIPS (2012) 1

    work page 2012

  21. [21]

    In: EMNLP (2020) 3

    Kryściński, W., McCann, B., Xiong, C., Socher, R.: Evaluating the factual consis- tency of abstractive text summarization. In: EMNLP (2020) 3

    work page 2020

  22. [22]

    Measuring Faithfulness in Chain-of-Thought Reasoning

    Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., et al.: Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702 (2023) 4

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  23. [23]

    Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models

    Lee, Y.L., Tsai, Y.H., Chiu, W.C.: Delve into visual contrastive decoding for hallucination mitigation of large vision-language models. arXiv preprint arXiv:2412.06775 (2024) 3

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  24. [24]

    Lehman, E., Jain, S., Pichotta, K., Goldberg, Y., Wallace, B.C.: Does bert pre- trained on clinical notes reveal sensitive data? In: NAACL (2021) 3

    work page 2021

  25. [25]

    In: NeurIPS (2020) 4

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. In: NeurIPS (2020) 4

    work page 2020

  26. [26]

    Li, J., Cao, P., Chen, Y., Xu, J., Li, H., Jiang, X., Liu, K., Zhao, J.: Towards better chain-of-thought: A reflection on effectiveness and faithfulness. In: ACL. pp. 10747–10765 (2025) 2

    work page 2025

  27. [27]

    In: EMNLP (2023) 1, 2

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hallucination in large vision-language models. In: EMNLP (2023) 1, 2

    work page 2023

  28. [28]

    In: CVPR

    Li, Z., Wu, X., Du, H., Liu, F., Nghiem, H., Shi, G.: A survey of state of the art large vision language models: Benchmark evaluations and challenges. In: CVPR. pp. 1587–1606 (2025) 1

    work page 2025

  29. [29]

    In: ACL (2022) 3

    Lin, S., Hilton, J., Evans, O.: Truthfulqa: Measuring how models mimic human falsehoods. In: ACL (2022) 3

    work page 2022

  30. [30]

    In: WWW (2020) 26

    Liu, B., Wei, H., Niu, D., Chen, H., He, Y.: Asking questions the human way: Scalable question-answer generation from text corpus. In: WWW (2020) 26

    work page 2020

  31. [31]

    In: ICLR (2024) 3

    Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., Wang, L.: Mitigating hallucination in large multi-modal models via robust instruction tuning. In: ICLR (2024) 3

    work page 2024

  32. [32]

    In: NAACL

    Liu, S., Halder, K., Qi, Z., Xiao, W., Pappas, N., Htut, P.M., John, N.A., Benajiba, Y., Roth, D.: Towards long context hallucination detection. In: NAACL. pp. 7827– 7835 (2025) 2

    work page 2025

  33. [33]

    In: NeurIPS (2021) 25

    Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al.: Codexglue: A machine learning benchmark dataset for code understanding and generation. In: NeurIPS (2021) 25

    work page 2021

  34. [34]

    Hallucination Detection and Hallucination Mitigation: An Investigation

    Luo, J., Li, T., Wu, D., Jenkin, M., Liu, S., Dudek, G.: Hallucination detection and hallucination mitigation: An investigation. arXiv preprint arXiv:2401.08358 (2024) 2

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [35]

    In: IJCNLP

    Lyu, Q., Havaldar, S., Stein, A., Zhang, L., Rao, D., Wong, E., Apidianaki, M., Callison-Burch, C.: Faithful chain-of-thought reasoning. In: IJCNLP. pp. 305–329 (2023) 2

    work page 2023

  36. [36]

    In: CVPR

    Ma, J., Suo, W., Wang, P., Zhang, Y.: Understanding and mitigating hallucinations in multimodal chain-of-thought models. In: CVPR. pp. 40224–40234 (2026) 4

    work page 2026

  37. [37]

    In: ICLR (2024) 1

    Ma, Y.J., Hejna, J., Fu, C., Shah, D., Liang, J., Xu, Z., Kirmani, S., Xu, P., Driess, D., Xiao, T., et al.: Vision language models are in-context value learners. In: ICLR (2024) 1

    work page 2024

  38. [38]

    Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics

    Mahler,J.,Liang,J.,Niyaz,S.,Laskey,M.,Doan,R.,Liu,X.,Ojea,J.A.,Goldberg, K.: Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint arXiv:1703.09312 (2017) 25 18 F. He et al

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  39. [39]

    In: EMNLP (2023) 2, 3, 26

    Manakul, P., Liusie, A., Gales, M.: Selfcheckgpt: Zero-resource black-box halluci- nation detection for generative large language models. In: EMNLP (2023) 2, 3, 26

    work page 2023

  40. [40]

    In: ACL (2024) 4

    Manevich, A., Tsarfaty, R.: Mitigating hallucinations in large vision-language mod- els (lvlms) via language-contrastive decoding (lcd). In: ACL (2024) 4

    work page 2024

  41. [41]

    Journal of chemical information and modeling52(6), 1686–1697 (2012) 25

    Martins, I.F., Teixeira, A.L., Pinheiro, L., Falcao, A.O.: A bayesian approach to in silico blood-brain barrier penetration modeling. Journal of chemical information and modeling52(6), 1686–1697 (2012) 25

    work page 2012

  42. [42]

    In: ICML (2024) 3

    Mudgal, S., Lee, J., Ganapathy, H., Li, Y., Wang, T., Huang, Y., Chen, Z., Cheng, H.T., Collins, M., Strohman, T., et al.: Controlled decoding from language models. In: ICML (2024) 3

    work page 2024

  43. [43]

    Capabilities of GPT-4 on Medical Challenge Problems

    Nori, H., King, N., McKinney, S.M., Carignan, D., Horvitz, E.: Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375 (2023) 3

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  44. [44]

    In: ACL (2024) 2

    Parcalabescu, L., Frank, A.: On measuring faithfulness or self-consistency of nat- ural language explanations. In: ACL (2024) 2

    work page 2024

  45. [45]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Peng, S., Yang, S., Jiang, L., Tian, Z.: Mitigating object hallucinations via sentence-level early intervention. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 635–646 (2025) 2

    work page 2025

  46. [46]

    In: EMNLP (2023) 2

    Qiu, Y., Ziser, Y., Korhonen, A., Ponti, E., Cohen, S.B.: Detecting and mitigating hallucinations in multilingual summarisation. In: EMNLP (2023) 2

    work page 2023

  47. [47]

    In: ACL (2018) 4

    Rajpurkar, P., Jia, R., Liang, P.: Know what you don’t know: Unanswerable ques- tions for squad. In: ACL (2018) 4

    work page 2018

  48. [48]

    In: EMNLP (2018) 1

    Rohrbach, A., Hendricks, L.A., Burns, K., Darrell, T., Saenko, K.: Object halluci- nation in image captioning. In: EMNLP (2018) 1

    work page 2018

  49. [49]

    In: EMNLP (2023) 3

    Sadat, M., Zhou, Z., Lange, L., Araki, J., Gundroo, A., Wang, B., Menon, R.R., Parvez, M., Feng, Z.: Delucionqa: Detecting hallucinations in domain-specific ques- tion answering. In: EMNLP (2023) 3

    work page 2023

  50. [50]

    In: EACL (2024) 4

    Sennrich, R., Vamvas, J., Mohammadshahi, A.: Mitigating hallucinations and off- target machine translation with source-contrastive and language-contrastive de- coding. In: EACL (2024) 4

    work page 2024

  51. [51]

    In: ACL (2024) 2, 3

    Su, W., Wang, C., Ai, Q., Hu, Y., Wu, Z., Zhou, Y., Liu, Y.: Unsupervised real- time hallucination detection based on the internal states of large language models. In: ACL (2024) 2, 3

    work page 2024

  52. [52]

    In: Findings of the Association for Computational Linguistics: ACL-IJCNLP (2021) 25

    Tafjord, O., Dalvi, B., Clark, P.: Proofwriter: Generating implications, proofs, and abductive statements over natural language. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP (2021) 25

    work page 2021

  53. [53]

    In: ICLR (2026),https://openreview.net/forum?id= XpL5eqjCjF4

    Tian, X., Zou, S., Yang, Z., He, M., Waschkowski, F., Wesemann, L., Tu, P.H., Zhang, J.: More thought, less accuracy? on the dual nature of reasoning in vision-language models. In: ICLR (2026),https://openreview.net/forum?id= XpL5eqjCjF4

    work page 2026

  54. [54]

    Scientific data5(1), 1–9 (2018) 25

    Tschandl, P., Rosendahl, C., Kittler, H.: The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data5(1), 1–9 (2018) 25

    work page 2018

  55. [55]

    In: ACL (2020) 3

    Wang, A., Cho, K., Lewis, M.: Asking and answering questions to evaluate the factual consistency of summaries. In: ACL (2020) 3

    work page 2020

  56. [56]

    In: EMNLP

    Wang, F., Zhou, W., Huang, J.Y., Xu, N., Zhang, S., Poon, H., Chen, M.: mdpo: Conditional preference optimization for multimodal large language models. In: EMNLP. pp. 8078–8088 (2024) 1

    work page 2024

  57. [57]

    In: ACL (2024) 2 HIVE 19

    Wang, X., Pan, J., Ding, L., Biemann, C.: Mitigating hallucinations in large vision- language models with instruction contrastive decoding. In: ACL (2024) 2 HIVE 19

    work page 2024

  58. [58]

    In: NeurIPS (2024) 4

    Wang, X., Zhou, D.: Chain-of-thought reasoning without prompting. In: NeurIPS (2024) 4

    work page 2024

  59. [59]

    In: EMNLP (2024) 26

    Wang, Y., Reddy, R.G., Mujahid, Z., Arora, A., Rubashevskii, A., Geng, J., Afzal, O.M., Pan, L., Borenstein, N., Pillai, A., et al.: Factcheck-bench: Fine-grained evaluation benchmark for automatic fact-checkers. In: EMNLP (2024) 26

    work page 2024

  60. [60]

    TMLR (2022) 4

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al.: Emergent abilities of large language models. TMLR (2022) 4

    work page 2022

  61. [61]

    In: NeurIPS (2022) 2

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E.H., Le, Q.V., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models. In: NeurIPS (2022) 2

    work page 2022

  62. [62]

    In: ACL (2024) 2

    Xu, J., Fei, H., Pan, L., Liu, Q., Lee, M.L., Hsu, W.: Faithful logical reasoning via symbolic chain-of-thought. In: ACL (2024) 2

    work page 2024

  63. [63]

    In: NeurIPS

    Xu,Z.,Liu,C.,Wei,Q.,Wu,J.,Zou,J.,Wang,X.,Zhou,Y.,Liu,S.:Morethinking, less seeing? assessing amplified hallucination in multimodal reasoning models. In: NeurIPS. vol. 38, pp. 82878–82905 (2026) 4

    work page 2026

  64. [64]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Yin, H., Si, G., Wang, Z.: Clearsight: Visual signal enhancement for object hallu- cination mitigation in multimodal large language models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 14625–14634 (2025) 2

    work page 2025

  65. [65]

    In: CVPR (2024) 3

    Yu, Q., Li, J., Wei, L., Pang, L., Ye, W., Qin, B., Tang, S., Tian, Q., Zhuang, Y.: Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In: CVPR (2024) 3

    work page 2024

  66. [66]

    In: CVPR (2019) 1

    Zellers, R., Bisk, Y., Farhadi, A., Choi, Y.: From recognition to cognition: Visual commonsense reasoning. In: CVPR (2019) 1

    work page 2019

  67. [67]

    TPAMI46(8), 5625–5644 (2024) 1

    Zhang, J., Huang, J., Jin, S., Lu, S.: Vision-language models for vision tasks: A survey. TPAMI46(8), 5625–5644 (2024) 1

    work page 2024

  68. [68]

    In: ECCV (2024) 3

    Zhang, J., Wang, T., Zhang, H., Lu, P., Zheng, F.: Reflective instruction tuning: Mitigating hallucinations in large vision-language models. In: ECCV (2024) 3

    work page 2024

  69. [69]

    In: EMNLP

    Zhang, T., Qiu, L., Guo, Q., Deng, C., Zhang, Y., Zhang, Z., Zhou, C., Wang, X., Fu, L.: Enhancing uncertainty-based hallucination detection with stronger focus. In: EMNLP. pp. 915–932 (2023) 2

    work page 2023

  70. [70]

    In: CVPR

    Zhang, W., Wang, Y.X.: Hallucination improves few-shot object detection. In: CVPR. pp. 13008–13017 (2021) 1

    work page 2021

  71. [71]

    In: EMNLP (2020) 3

    Zhao, Z., Cohen, S.B., Webber, B.: Reducing quantity hallucinations in abstractive summarization. In: EMNLP (2020) 3

    work page 2020

  72. [72]

    HIVE: Understanding Post Hallucination Reasoning in Vision Language Models

    Zhou, X., Zhang, M., Lee, Z., Ye, W., Zhang, S.: Hademif: Hallucination detection and mitigation in large language models. In: ICLR (2025) 2 20 F. He et al. SUMMARY OF THE APPENDIX This supplementary contains additional details for ECCV 2026, titled“HIVE: Understanding Post Hallucination Reasoning in Vision Language Models”. The supplementary is organized...

    work page arXiv 2025