Pith. sign in

REVIEW 4 major objections 5 minor 6 cited by

ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding precomputed activation shifts to selected attention heads during the forward pass reduces object hallucination in large vision-language models without adding inference latency.

desk verdict A plausible training-free intervention for LVLM hallucination with real cross-dataset gains, but the headline POPE numbers are partly circular and the 'trusted/untrusted' mechanism is never directly validated. read the letter →

arxiv 2411.15268 v1 pith:X3MK3DA4 submitted 2024-11-22 cs.CV cs.CL

Junzhe Chen , Tianshu Zhang , Shiyu Huang , Yuwei Niu , Linfeng Zhang , Lijie Wen , Xuming Hu This is my paper · ORCID
classification cs.CVcs.CL
keywords objecthallucinationlargevision-languagemodelsactivationinterventiontraining-freeforwardpassattentionheadsPOPEbenchmarkMME
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that object hallucination in open-weight large vision-language models can be reduced during the forward pass, rather than at decoding time, by adding precomputed activation shifts to attention heads that encode image-level and object-level visual information. The shifts come from comparing model activations on original images against activations on Gaussian-blurred 'untrusted' versions, using 1,500 'Yes' pairs from the POPE dataset. Applied together, the two intervention levels steer attention toward visual evidence while keeping useful language priors intact. If the claim holds, this yields a training-free, plug-in method that improves POPE and MME scores with no added inference latency, sidestepping the trade-off that contrastive decoding methods face.

What carries the argument

The central objects are the activation shift vectors $S^{(l)}_n$ and $S^{(l)}_{n,\text{obj}}$, defined as the mean difference between trusted and untrusted attention activations over the calibration samples at head $n$ and layer $l$. A binary SVM classifier trained on the trusted/untrusted pairs, with 2-fold cross-validation, identifies which heads encode image-level versus object-level visual information; the top-$K$ heads by classification accuracy receive the additive shift during the forward pass. Because the shift vectors are precomputed, the intervention adds no computational cost at inference time. The mechanism's role is to push attention toward visual evidence at two granularities without removing language priors.

What would settle it

Compute activation shift vectors from several corruptions of the same images (Gaussian blur, cropping, rotation) and compare whether they point in a common direction or whether applying the shift systematically reduces hallucination on a held-out set. If the shifts disagree across corruptions, or if applying the shift to inputs where the model is already correct degrades accuracy, the claim that the shift is a generic trustworthiness direction is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that object hallucination can be mitigated by a two-level activation intervention applied during the forward pass. For image-level information, whole-image Gaussian blur creates untrusted activations, and the mean trusted-minus-untrusted activation difference across 1,500 POPE 'Yes' pairs defines an image-level shift vector $S^{(l)}_n$ for each attention head and layer. For object-level information, Gaussian blur applied only to the detected object region yields an object-level shift vector $S^{(l)}_{n,\text{obj}}$. A binary SVM classifier per head picks the top-$K$ heads that best separate trusted from untrusted activations, and the shifts are added to those heads' outputs with intensities $\alpha$ and $\beta$ during generation. In tandem the interventions give the model a balanced attention mechanism that reduces over-reliance on language priors while increasing sensitivity to visual cues. The authors report average POPE F1 gains of 7.09 percentage points for LLaVA-v1.5 and 5.44 for Qwen-VL, MME gains of 80.51 and 34.67 points respectively, and essentially unchanged inference time.

Load-bearing premise

Gaussian blur applied to the whole image or to a detected object region creates an 'untrusted' model state whose activation difference from the original points along the object-hallucination direction; if blur mainly removes generic image fidelity rather than inducing the specific failure mode, the precomputed shift vector may not transfer to new images, prompts, or datasets.

Editorial extensions

If this is right

  • Applying ICT improves average POPE F1 by 7.09 percentage points on LLaVA-v1.5 and 5.44 on Qwen-VL across nine subsets, exceeding the gains of the OPERA baseline.
  • ICT raises MME hallucination-subset scores by 80.51 points on LLaVA-v1.5 and 34.67 on Qwen-VL, and also improves commonsense QA scores.
  • Inference time stays essentially unchanged (for example, 415.9 ms versus 405.3 ms for 20 generated tokens on LLaVA-v1.5), whereas contrastive decoding methods are 2.1 to 3.6 times slower.
  • Shift vectors trained on the MSCOCO random subset transfer to A-OKVQA and GQA and even to a different model, with LLaVA-v1.5-derived vectors improving Qwen-VL on GQA by about 4.6 points in F1 and accuracy.
  • Because the intervention happens in the forward pass, it is orthogonal to decoding-stage methods and can be combined with them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: if the blur-derived direction is a genuine trustworthiness direction, the same recipe should transfer to attribute and relation hallucinations, not just object presence, which is a testable extension.
  • Inference: the per-head SVM selection step implies that hallucination control is localizable to a small set of heads; architectures with similar head specialization should benefit similarly, while models without such specialization may not.
  • Inference: the method's dependence on open weights suggests a practical extension: learning the shift directions from a surrogate model of the same family and applying them to a larger sibling model.
  • Inference: the reported 1,500-sample calibration budget may be much larger than necessary; ablating the calibration sample count could reveal a data-efficiency frontier for activation-based intervention.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ICT (Image-Object Cross-Level Trusted Intervention), a training-free, forward-pass intervention method for reducing object hallucination in open-weight LVLMs. For a set of calibrated image-question pairs with 'Yes' answers, the authors blur the whole image (image-level) or only a detected object region (object-level) via a diffusion-style Gaussian noise process, and compute per-head activation shift vectors as the mean difference between clean and blurred activations. SVMs are trained per attention head to identify heads that distinguish clean from blurred states, and the top-K such heads are shifted by scaled versions of the shift vectors during generation. Experiments on LLaVA-v1.5 and Qwen-VL report consistent gains on POPE (nine subsets across MSCOCO, A-OKVQA, GQA), on the MME hallucination subsets, and a zero-latency advantage over contrastive decoding baselines. Cross-dataset and cross-model transfer experiments (e.g., shift vectors from LLaVA applied to Qwen on GQA) are also reported. The authors claim that the method mitigates over-reliance on language priors while preserving useful priors, and that it generalizes across datasets and models.

Significance. If the mechanism and evaluation were fully validated, ICT would be a useful contribution: it is lightweight, adds no inference latency, and is orthogonal to decoding-time contrastive methods, while the cross-dataset and cross-model results suggest the computed shift directions capture something transferable rather than dataset-specific. The paper's core idea of steering attention-head activations during the forward pass, using a cheap calibration set, is novel and practically appealing. However, the paper is currently weakened by a circular evaluation on the POPE COCO Random subset, a lack of reported variance, and, most importantly, an unvalidated mechanistic premise that Gaussian-blur-induced activation differences point along the object-hallucination direction. The out-of-sample gains on A-OKVQA, GQA, and MME provide genuine independent support, but the central claim that ICT specifically counteracts language-prior over-reliance while preserving beneficial priors is not yet established by the evidence presented.

major comments (4)
  1. [Sec. 5.1, Appendix A, Table 1] The POPE COCO Random results are partly circular: the activation shift vectors are computed from 1,500 'Yes' pairs from the COCO Random subset, the per-head SVM classifiers are trained on the same subset, and the hyperparameters α (= β) and K are selected by grid search on the POPE COCO Random subset (Appendix A). Reporting the test-set F1 gain on that same subset as a headline result (§5.2, first bullet, and Table 1) is therefore a test-set-tuned number, not a fair evaluation. Please either tune on a separate validation split and then report the COCO Random test row, or clearly relegate the COCO Random row to a development-set result and treat the other eight POPE subsets, plus MME, as the primary out-of-sample evidence. The paper should also report variance or significance (e.g., multiple seeds or bootstrap intervals) for at least the main rows, since no uncertainty is currently given.
  2. [Secs. 4.1, 4.2, Eq. (4), Eq. (6), Sec. 7 Limitations] The load-bearing premise that a Gaussian-blurred image constitutes an 'untrusted' state whose activation difference from the clean state points along the object-hallucination direction is not validated. The calibration pairs are (q + O_i, V_i) versus (q + O_i, V'_i), where q is 'What is it in the image?' and O_i is the correct object name; thus the trusted/untrusted contrast isolates the model's sensitivity to image fidelity, not its tendency to hallucinate a specific object. The SVM classifiers in Secs. 4.1-4.2 only measure whether heads change activation under blur, which any visual degradation would satisfy, and the Limitations section itself notes that only Gaussian blur was used. As a result, S_n and S_obj,n could be generic 'amplify visual signal' directions rather than hallucination-specific directions, leaving the paper's core mechanistic claim ('mitigates excessive reliance on language priors while enhancing sensitivity to essential visual cues') unestablished. Please add a direct test that the shift direction is tied to hallucination, for example: (i) compare against other perturbations (rotation, cropping, color jitter, or a different noise type) and show the intervention transfers only when the perturbation genuinely induces hallucination errors; (ii) measure whether applying the shift to a hallucinating sample moves its activations toward a correct sample, using the paper's own trusted/untrusted labels; (iii) or at minimum, characterize what fraction of the 1,500 POPE 'Yes' pairs would be answered correctly or incorrectly after blur, to show that the untrusted state corresponds to actual error.
  3. [Sec. 5.1 Baselines; Tables 1 and 3] The baseline numbers for VCD and OPERA are reported without stating whether they are re-run under the same code, model version, and decoding configuration, or taken from the original papers. Since the paper's central contribution is a comparison against these decoding-time methods, please clarify the provenance of every baseline number and, if they are not re-run, re-run them with the same evaluation harness to ensure a fair comparison. Additionally, all reported numbers in Tables 1 and 3 are point estimates; without any measure of variability across runs or evaluation subsets, it is difficult to assess whether the claimed improvements over VCD/OPERA are statistically meaningful, particularly on A-OKVQA and GQA where some gaps are as small as 0.1-0.4 points.
  4. [Sec. 6.2 and Figure 4] The cross-model generalization claim (Sec. 6.2, Table 3) is interesting but the t-SNE visualization in Figure 4 is used to conclude that the shift vectors are 'universal shift vectors' (Sec. 6.2, final sentence). t-SNE is a nonlinear, stochastic embedding, and the visual proximity of points in t-SNE does not directly demonstrate that the shift vectors can be transferred across models; the actual transfer results in Table 3 are the relevant evidence, and they show a performance drop relative to the same-model ICT (e.g., Qwen GQA Random: ICT 86.38 vs ICT-LLaVA 83.27 F1). Please temper the claim of universality and instead report a quantitative similarity measure (e.g., cosine similarity between shift vectors after alignment) or a head-selection overlap statistic.
minor comments (5)
  1. [Throughout] There is inconsistent naming ('OPERA' vs 'Opera') and numerous duplicated or near-duplicate references in the bibliography, e.g., refs [3] and [4] are the same AGLA paper, and refs [54] and [55] are the same survey. Please deduplicate the reference list.
  2. [Eq. (3) and Eq. (4)] Equation (3) writes a full forward diffusion process, but only the final blurred image V'_i is used; the intermediate steps and noise schedule β_t are never specified or used. Either specify the exact blur/noise parameters used (number of diffusion steps, β_t values, or equivalently the Gaussian kernel and standard deviation) or simplify the equation to a direct Gaussian blur. Without these details, the calibration procedure is not reproducible.
  3. [Figure 3] The radar chart in Figure 3 is difficult to read, and the text under the model names ('CSQA 119.28', 'Existence 190.00', etc.) is not clearly labeled as referring to which variant (vanilla, VCD, or ICT). The caption should state explicitly which curve and which set of numbers correspond to each method, and the claimed total improvement of '67.37 points' (Introduction, contribution bullet) is not directly derivable from the numbers in the figure; please provide a clear numerical summary table.
  4. [Sec. 5.2, POPE results] The first bullet in Sec. 5.2 says ICT improves average F1 by 7.09% for LLaVA and 5.44% for Qwen, and that these are 'respectively higher than the previous contrastive decoding sota baseline (Opera) 2.19% and 1.14%.' The comparison is confusing because the 7.09/5.44 numbers are absolute F1 improvements over Regular, while the 2.19/1.14 numbers appear to be the difference between Opera's improvement and ICT's improvement. Please state the comparison on the same scale and clarify whether these are absolute percentage points or relative improvements.
  5. [Sec. 6.5 and case studies] The case studies in Figure 5 and Figure 8 are anecdotal. They are useful for illustration, but the paper should make clear that they are selected examples, not a systematic error analysis. The 'undercut fruit' example also shows a failure mode where ICT misses the 'uncut' attribute, which is a useful negative case; please discuss more explicitly whether and how the method can be expected to handle attribute-level reasoning.

Circularity Check

1 steps flagged · score 5.0 of 10

COCO Random results are in-sample: the intervention shift and hyperparameters are derived from the same POPE COCO Random "Yes" pairs used in the headline evaluation; cross-dataset gains keep the central claim partially independent.

  1. fitted input called prediction [Section 5.1 Implementation Details; Table 1 (COCO Random rows); Appendix A and Section 6.3 hyperparameter search]
    "we utilized 1,500 QA pairs with “Yes” responses from the COCO Random subset of the POPE dataset to train the intervention shift vector. Subsequently, we evaluated the ICT approach on two datasets with significant distributional differences: POPE, and MME. ... we set α = β and determined the optimal values of α, β, and K through a grid search."

    The intervention shift vectors S_n and S_obj,n are computed by averaging A_clean − A_blurred over exactly the 1,500 POPE COCO Random "Yes" pairs (Eqs. 4 and 6), and the intervention strength α=β and head count K are grid-searched on the same COCO Random subset (Appendix A; Fig. 6). Table 1 then reports the COCO Random row as a headline result. That row is therefore in-sample: the direction of the intervention and its intensity are both derived from the very benchmark split being scored. The substantial gains on the other eight POPE subsets and on MME are out-of-sample and provide independent support, so the circularity is partial rather than total.

full rationale

The central circularity is confined to the flagship COCO Random result. The paper computes S_n and S_obj,n from 1,500 POPE COCO Random "Yes" pairs and tunes α, β, and K on the same COCO Random subset, then reports the COCO Random row of Table 1 as a main result; that specific prediction reduces to an in-sample fit. The cross-distribution POPE subsets (A-OKVQA, GQA), the MME benchmark, and the cross-model transfer experiment are genuinely out-of-sample, so the method has independent empirical content and the score is moderate, not extreme. The Limitations section's admission that only Gaussian blur was used, and the absence of a direct check that the blurred state is actually a hallucination state, are correctness risks about what the shift vector measures rather than circular reductions; they do not by themselves make the derivation circular under the strict evidence standard. No load-bearing self-citation or imported uniqueness theorem appears in the paper.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central method rests on a calibration step that fits shift vectors and head selections to labeled POPE data, plus grid-searched intervention strengths. The core novelty is the intervention recipe, while the benchmark gains depend on several domain assumptions about blur directionality and head selectivity.

free parameters (4)
  • alpha (and beta, with alpha = beta) = chosen by grid search over {8,16,24,32}; exact optimum not stated
    Intervention intensity for image- and object-level shifts; tuned directly on POPE COCO Random evaluation set without a held-out validation split.
  • K (number of intervened heads) = chosen by grid search over {32,64,128,256}; exact optimum not stated
    Number of attention heads receiving the activation shift; tuned on the POPE COCO Random evaluation set.
  • B (calibration set size) = 1500
    Number of POPE COCO Random 'Yes' QA pairs used to compute shift vectors; chosen by hand, not justified by ablations.
  • SVM hyperparameters = unspecified (likely defaults)
    Binary classifiers per head are trained on the calibration pairs; kernel and regularization details are not reported.
assumptions (5)
  • domain assumption Gaussian blur creates an untrusted image whose model activations differ from the trusted original along the hallucination direction.
    Used to define S_n in Eq. (4) and S_obj,n in Eq. (6); the entire intervention direction is this difference.
  • domain assumption A binary classifier's accuracy at separating trusted from untrusted activations identifies heads that encode image-level or object-level visual information.
    Head selection in Sections 4.1 and 4.2 and Figure 7 relies on this.
  • domain assumption The last-token activation is a fused multimodal representation, and a constant shift vector added to selected heads affects all token positions equivalently.
    Shift vectors are computed at position s = Nx + Nv + 1 but applied to the whole sequence in Eqs. (5), (7), and (8).
  • domain assumption POPE 'Yes' ground truth labels and Grounding DINO object boxes are accurate enough to define trusted data.
    Trusted pairs use these annotations; errors would propagate into the shift vectors.
  • ad hoc to paper The grid search over alpha and K on the POPE COCO Random subset does not overfit the reported results.
    No separate validation set is described; the evaluation set is used for hyperparameter selection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models." pith.science (2026). https://pith.science/paper/X3MK3DA4

@misc{pith2026241115268,
  author       = {Pith},
  title        = {Pith review of: ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X3MK3DA4}},
  note         = {Machine review of arXiv:2411.15268}
}
read the original abstract

Despite the recent breakthroughs achieved by Large Vision Language Models (LVLMs) in understanding and responding to complex visual-textual contexts, their inherent hallucination tendencies limit their practical application in real-world scenarios that demand high levels of precision. Existing methods typically either fine-tune the LVLMs using additional data, which incurs extra costs in manual annotation and computational resources or perform comparisons at the decoding stage, which may eliminate useful language priors for reasoning while introducing inference time overhead. Therefore, we propose ICT, a lightweight, training-free method that calculates an intervention direction to shift the model's focus towards different levels of visual information, enhancing its attention to high-level and fine-grained visual details. During the forward pass stage, the intervention is applied to the attention heads that encode the overall image information and the fine-grained object details, effectively mitigating the phenomenon of overly language priors, and thereby alleviating hallucinations. Extensive experiments demonstrate that ICT achieves strong performance with a small amount of data and generalizes well across different datasets and models. Our code will be public.

Figures

Figures reproduced from arXiv: 2411.15268 by the authors.

Figure 1
Figure 1. Comparison between Contrastive Decoding (top) and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our proposed ICT method. ICT consists of two levels of intervention: Image-Level and Object-Level. The Image [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. t-SNE visualization of Object-Level and Image-Level [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Case Study and Error Analysis of ICT. 20 0 20 40 60 80 100 120 Alpha 72.5 75.0 77.5 80.0 82.5 85.0 87.5 F1 Score % (LLaVA-v1.5) LLaVA-v1.5 Qwen-VL 30 60 90 120 150 180 210 240 K 55 60 65 70 75 80 85 F1 Score % (LLaVA-v1.5) LLaVA-v1.5 Qwen-VL 80 81 82 83 84 F1 Score % (…
Figure 6
Figure 6. Figure 6: Effect of hyperparameters α (left) and K (right) on the F1 score for LLaVA-v1.5 and Qwen-VL models on the POPE COCO random subset. equately intervened upon, resulting in suboptimal perfor￾mance. Conversely, when K is too large, attention heads that encode irrelevant in…
Figure 7
Figure 7. Figure 7: Heatmap of classification accuracy for Image-Level [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: More case studies of ICT. In the first example, our ICT avoids captioning non-existent objects in the image while keeping the [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    Large multimodal models mostly fail to proactively detect flawed textual premises, and their performance depends on error type and on how they weight text versus images.

  2. Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens

    cs.CV 2025-08 conditional novelty 6.0 of 10

    Modality bias, an imbalanced attention to text or image during hallucinated outputs, is shown to be mitigated by a training-free attention intervention plus contrastive decoding.

  3. GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs

    cs.CL 2025-07 conditional novelty 6.0 of 10

    GrAInS uses Integrated Gradients to identify the most influential tokens, then builds layer-wise steering vectors that improve truthfulness, reduce hallucination, and preserve general capabilities in LLMs and VLMs.

  4. CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention

    cs.CL 2025-06 conditional novelty 6.0 of 10

    An inference-time attention-shift intervention aligns non-English queries' cross-modal attention with English, cutting multilingual object hallucination in LVLMs on POPE and MME.

  5. Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations

    cs.CV 2025-05 conditional novelty 6.0 of 10

    VaLSe uses attention-based visual contribution maps to steer an LVLM's latent features toward visually grounded content, reducing object hallucinations on several benchmarks while exposing flaws in CHAIR-style evaluation.

  6. Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A temporal-aware activation steering method for VideoLLMs reduces hallucination on VidHalluc and EventHallusion across three models, with the key finding that hallucination-sensitive modules are tied to temporal varia...

Reference graph

Works this paper leans on

109 extracted references · 40 canonical work pages · cited by 6 Pith papers

  1. [1]

    Characterizing attri- bution and fluency tradeoffs for retrieval-augmented large language models, 2023

    Renat Aksitov, Chung-Ching Chang, David Reitter, Sia- mak Shakeri, and Yunhsuan Sung. Characterizing attri- bution and fluency tradeoffs for retrieval-augmented large language models, 2023. 3

    2023

  2. [2]

    Flamingo: A visual language model for few-shot learn- ing

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: A visual language model for few-shot learn- ing. In Advances in Neural Information Processing Sys- tems, pages 23716–23736, 2022. 2

    2022

  3. [3]

    Agla: Mitigating object hallucinations in large vision- language models with assembly of global and local atten- tion

    Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large vision- language models with assembly of global and local atten- tion. arXiv preprint, 2024. 3

    2024

  4. [4]

    Agla: Mitigating object hallucinations in large vision- language models with assembly of global and local atten- tion

    Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large vision- language models with assembly of global and local atten- tion. arXiv preprint arXiv:2406.12718, 2024. 1

    arXiv 2024

  5. [5]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. 2023. 1, 2, 5

    2023

  6. [6]

    Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, pages 1877–1901,

    1901

  7. [7]

    Alleviating hallucinations in large vision- language models through hallucination-induced optimiza- tion, 2024

    Beitao Chen, Xinyu Lyu, Lianli Gao, Jingkuan Song, and Heng Tao Shen. Alleviating hallucinations in large vision- language models through hallucination-induced optimiza- tion, 2024. 3

    2024

  8. [8]

    Selfie: Self-interpretation of large language model embeddings,

    Haozhe Chen, Carl V ondrick, and Chengzhi Mao. Selfie: Self-interpretation of large language model embeddings,

Show all 109 references
  1. [9]

    Halc: Object hallucination re- duction via adaptive focal-contrast decoding

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: Object hallucination re- duction via adaptive focal-contrast decoding. In Forty-first International Conference on Machine Learning. 1

  2. [10]

    Halc: Object hallucination reduc- tion via adaptive focal-contrast decoding, 2024

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: Object hallucination reduc- tion via adaptive focal-contrast decoding, 2024. http: //arxiv.org/abs/2403.00425. 2, 3

    2024 arXiv

  3. [11]

    Factool: Factuality detection in gener- ative ai – a tool augmented framework for multi-task and multi-domain scenarios, 2023

    I.-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Ke- hua Feng, Chunting Zhou, Junxian He, Graham Neubig, and Pengfei Liu. Factool: Factuality detection in gener- ative ai – a tool augmented framework for multi-task and multi-domain scenarios, 2023. http://arxiv.org/ abs/2...

    2023 arXiv

  4. [12]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023),...

    2023

  5. [13]

    Fine-grained im- age captioning with clip reward

    Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Der- noncourt, Trung Bui, and Mohit Bansal. Fine-grained im- age captioning with clip reward. In Findings of the Associ- ation for Computational Linguistics: NAACL 2022 , pages 517–527, 2022. 1

    2022

  6. [14]

    Dola: Decoding by con- trasting layers improves factuality in large language mod- els, 2024

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by con- trasting layers improves factuality in large language mod- els, 2024. http://arxiv.org/abs/2309.03883 . 2, 3

    2024 arXiv

  7. [15]

    Support-vector networks

    Corinna Cortes. Support-vector networks. Machine Learn- ing, 1995. 5

    1995

  8. [16]

    Large language mod- els with controllable working memory

    Li D., Rawat A.S., Zaheer M., Wang X., Lukasik M., Veit A., Yu F., and Kumar S. Large language mod- els with controllable working memory. arXiv preprint arXiv:2211.05110, 2022. 3

    2022 arXiv

  9. [17]

    Retrieve only when it needs: Adaptive re- trieval augmentation for hallucination mitigation in large language models, 2024

    Hanxing Ding, Liang Pang, Zihao Wei, Huawei Shen, and Xueqi Cheng. Retrieve only when it needs: Adaptive re- trieval augmentation for hallucination mitigation in large language models, 2024. http://arxiv.org/abs/ 2402.10612. 3

    2024

  10. [18]

    Detecting hallucinations in large language mod- els using semantic entropy

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language mod- els using semantic entropy. Nature, 630(8017):625–630,

  11. [19]

    Costa-juss `a

    Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R. Costa-juss `a. A primer on the inner workings of transformer-based language models, 2024. 2

    2024

  12. [20]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 5

    2023 arXiv

  13. [21]

    Textbooks are all you need, 2023

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C´esar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harki- rat Singh Behl, Xin Wang, S ´ebastien Bubeck, Ronen Eldan, Adam Tauman Kal...

    2023

  14. [22]

    Detecting and preventing hallucinations in large vision language models

    Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intel- ligence, pages 18135–18143, 2024. 3

    2024

  15. [23]

    Detecting and preventing hallucinations in large vision language models

    Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intel- ligence, pages 18135–18143, 2024. 2

    2024

  16. [24]

    Swapmix: Diagnosing and regularizing the over-reliance on visual context in vi- sual question answering

    Vipul Gupta, Zhuowan Li, Adam Kortylewski, Chenyu Zhang, Yingwei Li, and Alan Yuille. Swapmix: Diagnosing and regularizing the over-reliance on visual context in vi- sual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition ,...

    2022

  17. [25]

    Visual perturbation-aware collaborative learning for overcoming the language prior problem

    Yudong Han, Liqiang Nie, Jianhua Yin, Jianlong Wu, and Yan Yan. Visual perturbation-aware collaborative learning for overcoming the language prior problem. arXiv preprint arXiv:2207.11850, 2022. 1

    2022 arXiv

  18. [26]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural informa- tion processing systems, 33:6840–6851, 2020. 3

    2020

  19. [27]

    Cogvlm2: Visual language mod- els for image and video understanding

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Jun- hui Ji, Zhao Xue, et al. Cogvlm2: Visual language mod- els for image and video understanding. arXiv preprint arXiv:2408.16500, 2024. 1

    2024 arXiv

  20. [28]

    Lrp4rag: Detecting hallucinations in retrieval-augmented generation via layer-wise relevance propagation, 2024

    Haichuan Hu, Yuhan Sun, and Quanjun Zhang. Lrp4rag: Detecting hallucinations in retrieval-augmented generation via layer-wise relevance propagation, 2024. http:// arxiv.org/abs/2408.15533. 3

    2024 arXiv

  21. [29]

    Look before you leap: Unveiling the power of gpt- 4v in robotic vision-language planning

    Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao. Look before you leap: Unveiling the power of gpt- 4v in robotic vision-language planning. arXiv preprint arXiv:2311.17842, 2023. 1

    2023 arXiv

  22. [30]

    A survey on hallucination in large language models: Principles, tax- onomy, challenges, and open questions, 2023

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, tax- onomy, challenges, and open questions, 2023. 3

    2023

  23. [31]

    Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Confer...

    2024

  24. [32]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 5

    2019

  25. [33]

    Self-introspective decod- ing: Alleviating hallucinations for large vision-language models, 2024

    Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self-introspective decod- ing: Alleviating hallucinations for large vision-language models, 2024. 3

    2024

  26. [34]

    Vcoder: Ver- satile vision encoders for multimodal large language mod- els

    Jitesh Jain, Jianwei Yang, and Humphrey Shi. Vcoder: Ver- satile vision encoders for multimodal large language mod- els. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 27992–28002,

  27. [35]

    Surgical-llava: To- ward surgical scenario understanding via large language and vision models

    Juseong Jin and Chang Wook Jeong. Surgical-llava: To- ward surgical scenario understanding via large language and vision models. arXiv preprint arXiv:2410.09750, 2024. 1

    2024 arXiv

  28. [36]

    Code: Contrasting self-generated description to combat hallucination in large multi-modal models, 2024

    Junho Kim, Hyunjun Kim, Yeonju Kim, and Yong Man Ro. Code: Contrasting self-generated description to combat hallucination in large multi-modal models, 2024. 2, 3

    2024

  29. [37]

    Semantic uncertainty: Linguistic invariances for uncertainty estima- tion in natural language generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estima- tion in natural language generation. arXiv, 2023. 3

    2023

  30. [38]

    Multi- modal reasoning with multimodal knowledge graph

    Junlin Lee, Yequan Wang, Jing Li, and Min Zhang. Multi- modal reasoning with multimodal knowledge graph. arXiv preprint arXiv:2406.02030, 2024. 2

    2024 arXiv

  31. [39]

    Deduplicating training data makes lan- guage models better

    Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes lan- guage models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volum...

    2022

  32. [40]

    Vlind-bench: Measuring language priors in large vision- language models

    Kang-il Lee, Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongryeol Lee, Hyukhun Koh, and Kyomin Jung. Vlind-bench: Measuring language priors in large vision- language models. arXiv preprint arXiv:2406.08702, 2024. 1

    2024 arXiv

  33. [41]

    Factuality enhanced language models for open-ended text generation

    Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pas- cale N Fung, Mohammad Shoeybi, and Bryan Catanzaro. Factuality enhanced language models for open-ended text generation. In Advances in Neural Information Processing Systems, pages 34586–34599, 2022. 3

    2022

  34. [42]

    V olcano: Mitigating multimodal hallucination through self-feedback guided revision

    Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Minjoon Seo. V olcano: Mitigating multimodal hallucination through self-feedback guided revision. arXiv, 2024. 3

    2024

  35. [43]

    Mitigating object hallucinations in large vision-language models through vi- sual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shi- jian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through vi- sual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  36. [44]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the International Conference on Machine Learning, pages 19730–19742. PMLR, 2023. 2

    2023

  37. [45]

    Inference-time intervention: Elic- iting truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Vi´egas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Elic- iting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2024. 3

    2024

  38. [46]

    Vlfeedback: A large-scale ai feedback dataset for large vision-language models alignment

    Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, Lingpeng Kong, and Qi Liu. Vlfeedback: A large-scale ai feedback dataset for large vision-language models alignment. InPro- ceedings of the 2024 Conference on Empirical Methods in Na...

    2024

  39. [47]

    Contrastive decoding: Open-ended text gen- eration as optimization

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text gen- eration as optimization. arXiv preprint, 2023. 3

    2023

  40. [48]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Em- pirical Methods in Natural Language Processing. 5

    2023

  41. [49]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedin...

    2014

  42. [50]

    Ash, Surbhi Goel, Akshay Krishna- murthy, and Cyril Zhang

    Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishna- murthy, and Cyril Zhang. Exposing attention glitches with flip-flop language modeling, 2023. 3

    2023

  43. [51]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representa- tions, 2023. 3

    2023

  44. [52]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023. 1, 2, 5

    2023

  45. [53]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Infor- mation Processing Systems, 36, 2024. 3

    2024

  46. [54]

    A survey on hallucination in large vision-language models

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint, 2024. 2

    2024

  47. [55]

    A survey on hallucination in large vision-language models

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253, 2024. 1

    2024 arXiv

  48. [56]

    Kangaroo: A powerful video-language model supporting long-context video input

    Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408.15542, 2024. 1

    2024 arXiv

  49. [57]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 4

    2023 arXiv

  50. [58]

    Paying more at- tention to image: A training-free method for alleviating hal- lucination in lvlms, 2024

    Shi Liu, Kecheng Zheng, and Wei Chen. Paying more at- tention to image: A training-free method for alleviating hal- lucination in lvlms, 2024. 3

    2024

  51. [59]

    Kernel language entropy: Fine-grained uncer- tainty quantification for llms from semantic similarities

    Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. Kernel language entropy: Fine-grained uncer- tainty quantification for llms from semantic similarities. arXiv, 2024. 3

    2024

  52. [60]

    OpenAI. GPT-4. https://openai.com/gpt- 4 ,

  53. [61]

    Training language models to follow instructions with hu- man feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, et al. Training language models to follow instructions with hu- man feedback. In Advances in Neural Information Process- ing Systems, pages 27730–27744, 2022. 3

    2022

  54. [62]

    Bender, Emily Denton, and Alex Hanna

    Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns, 2(11):100336, 2021. 3

    2021

  55. [63]

    Distillation con- trastive decoding: Improving llms reasoning with con- trastive decoding and distillation, 2024

    Phuc Phan, Hieu Tran, and Long Phan. Distillation con- trastive decoding: Improving llms reasoning with con- trastive decoding and distillation, 2024. 2, 3

    2024

  56. [64]

    Nationality bias in text generation.arXiv preprint arXiv:2302.02463, 2023

    Venkit P.N., Gautam S., Panchanadikar R., Huang T.H., and Wilson S. Nationality bias in text generation.arXiv preprint arXiv:2302.02463, 2023. 3

    2023 arXiv

  57. [65]

    Alleviating hallucination in large vision- language models with active retrieval augmentation, 2024

    Xiaoye Qu, Qiyuan Chen, Wei Wei, Jishuo Sun, and Jianfeng Dong. Alleviating hallucination in large vision- language models with active retrieval augmentation, 2024. http://arxiv.org/abs/2408.00555. 3

    2024 arXiv

  58. [66]

    Look, compare, decide: Alleviating hallucination in large vision- language models via multi-view multi-path reasoning,

    Xiaoye Qu, Jiashuo Sun, Wei Wei, and Yu Cheng. Look, compare, decide: Alleviating hallucination in large vision- language models via multi-view multi-path reasoning,

  59. [67]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International conference on machine learning...

    2021

  60. [68]

    The curious case of hallucinations in neural ma- chine translation

    Vikas Raunak, Arul Menezes, and Marcin Junczys- Dowmunt. The curious case of hallucinations in neural ma- chine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, pag...

    2021

  61. [69]

    Vacode: Visual augmented contrastive decoding

    Kim S., Cho B., Bae S., Ahn S., and Yun S.Y . Vacode: Visual augmented contrastive decoding. arXiv preprint arXiv:2408.05337, 2024. 2, 3

    2024 arXiv

  62. [70]

    Truthfulqa: Measur- ing how models mimic human falsehoods

    Lin S., Hilton J., and Evans O. Truthfulqa: Measur- ing how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021. 3

    2021 arXiv

  63. [71]

    A survey on multimodal large language models

    Yin S., Fu C., Zhao S., Li K., Sun X., Xu T., and Chen E. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023. 3

    2023 arXiv

  64. [72]

    A comprehensive survey of hallucination in large language, image, video and audio foundation models

    Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, and Aman Chadha. A comprehensive survey of hallucination in large language, image, video and audio foundation models. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 11709– 1...

    2024

  65. [73]

    A-okvqa: A benchmark for visual question answering using world knowledge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision , pages 146–162. Springer, 2022. 5

    2022

  66. [74]

    Identifying untrustwor- thy samples: Data filtering for open-domain dialogues with bayesian optimization

    Lei Shen, Haolan Zhan, Xin Shen, Hongshen Chen, Xi- aofang Zhao, and Xiaodan Zhu. Identifying untrustwor- thy samples: Data filtering for open-domain dialogues with bayesian optimization. In Proceedings of the 30th ACM In- ternational Conference on Information & Knowledge Man-...

    2021

  67. [75]

    Trusting your evidence: Hallucinate less with context-aware decoding,

    Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding,

  68. [76]

    Smith, Luke Zettlemoyer, Scott Yih, and Mike Lewis

    Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Rich James, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Scott Yih, and Mike Lewis. In-context pretraining: Language modeling beyond document bound- aries, 2023. 3

    2023

  69. [77]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools

    GLM T., Zeng A., Xu B., Wang B., Zhang C., Yin D., Rojas D., Feng G., Zhao H., Lai H., and Yu H. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, 2024. 2

    2024 arXiv

  70. [78]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi`ere, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, L´eonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Al...

    2024

  71. [79]

    Llama: Open and efficient foundation language mod- els

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Bap- tiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language mod- els. arXiv preprint, 2023. 2

    2023

  72. [80]

    Analyzing multi-head self-attention: Spe- cialized heads do the heavy lifting, the rest can be pruned

    Elena V oita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Spe- cialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics , pages...

  73. [81]

    Investigating and mitigating the mul- timodal hallucination snowballing in large vision-language models

    Zhong W., Feng X., Zhao L., Li Q., Huang L., Gu Y ., Ma W., Xu Y ., and Qin B. Investigating and mitigating the mul- timodal hallucination snowballing in large vision-language models. arXiv preprint arXiv:2407.00569, 2024. 2, 3

    2024 arXiv

  74. [82]

    Surgical-lvlm: Learning to adapt large vision-language model for grounded visual question answering in robotic surgery

    Guankun Wang, Long Bai, Wan Jun Nah, Jie Wang, Zhaoxi Zhang, Zhen Chen, Jinlin Wu, Mobarakol Islam, Hong- bin Liu, and Hongliang Ren. Surgical-lvlm: Learning to adapt large vision-language model for grounded visual question answering in robotic surgery. arXiv preprint arXiv:24...

    2024 arXiv

  75. [83]

    Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution

    P Wang, S Bai, S Tan, S Wang, Z Fan, J Bai, K Chen, X Liu, J Wang, W Ge, and Y Fan. Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2

    2024 arXiv

  76. [84]

    Mitigating hallucinations in large vision-language models with instruction contrastive decoding, 2024

    Xintong Wang, Jingheng Pan, Liang Ding, and Chris Bie- mann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding, 2024. 3

    2024

  77. [85]

    Ritual: Random image transforma- tions as a universal anti-hallucination lever in lvlms, 2024

    Sangmin Woo, Jaehyuk Jang, Donguk Kim, Yubin Choi, and Changick Kim. Ritual: Random image transforma- tions as a universal anti-hallucination lever in lvlms, 2024. http://arxiv.org/abs/2405.17821. 3

    2024 arXiv

  78. [86]

    Don’t miss the forest for the trees: At- tentional vision calibration for large vision language mod- els, 2024

    Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi, and Changick Kim. Don’t miss the forest for the trees: At- tentional vision calibration for large vision language mod- els, 2024. 2, 3

    2024

  79. [87]

    Logical closed loop: Uncov- ering object hallucinations in large vision-language models,

    Junfei Wu, Qiang Liu, Ding Wang, Jinghao Zhang, Shu Wu, Liang Wang, and Tieniu Tan. Logical closed loop: Uncov- ering object hallucinations in large vision-language models,

  80. [88]

    Noiseboost: Alleviating hallucination with noise perturbation for multimodal large language models, 2024

    Kai Wu, Boyuan Jiang, Zhengkai Jiang, Qingdong He, Donghao Luo, Shengzhi Wang, Qingwen Liu, and Chengjie Wang. Noiseboost: Alleviating hallucination with noise perturbation for multimodal large language models, 2024. http://arxiv.org/abs/2405.20081. 3

    2024 arXiv

  81. [89]

    http://arxiv.org/abs/2402.11622. 3

    arXiv

  82. [90]

    Re-reading im- proves reasoning in large language models, 2024

    Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, and Jian guang Lou. Re-reading im- proves reasoning in large language models, 2024. http: //arxiv.org/abs/2309.06275. 3

    2024 arXiv

  83. [91]

    Seeing the image: Prioritizing visual correlation by contrastive alignment, 2024

    Xin Xiao, Bohong Wu, Jiacong Wang, Chunyuan Li, Xun Zhou, and Haoyuan Guo. Seeing the image: Prioritizing visual correlation by contrastive alignment, 2024. 3

    2024

  84. [92]

    Entity cloze by date: What lms know about unseen entities

    Onoe Y ., Zhang M.J., Choi E., and Durrett G. Entity cloze by date: What lms know about unseen entities. arXiv preprint arXiv:2205.02832, 2022. 3

    2022 arXiv

  85. [93]

    xgen-mm (blip-3): A family of open large mul- timodal models

    L Xue, M Shu, A Awadalla, J Wang, A Yan, S Purush- walkam, H Zhou, V Prabhu, Y Dai, M S Ryoo, and S Kendre. xgen-mm (blip-3): A family of open large mul- timodal models. arXiv preprint arXiv:2408.08872 , 2024. 2

    2024

  86. [94]

    Llm4drive: A survey of large language models for au- tonomous driving

    Zhenjie Yang, Xiaosong Jia, Hongyang Li, and Junchi Yan. Llm4drive: A survey of large language models for au- tonomous driving. In NeurIPS 2024 Workshop on Open- World Agents, 2023. 1

    2024

  87. [95]

    Convis: Con- trastive decoding with hallucination visualization for miti- gating hallucinations in multimodal large language models

    Park Y ., Lee D., Choe J., and Chang B. Convis: Con- trastive decoding with hallucination visualization for miti- gating hallucinations in multimodal large language models. arXiv preprint arXiv:2408.13906, 2024. 3

    2024 arXiv

  88. [96]

    Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data

    Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wen- tao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2024

  89. [97]

    Woodpecker: Hallucination correction for multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint, 2023. 3

    2023

  90. [98]

    Less is more: Mit- igating multimodal hallucination from an eos decision per- spective, 2024

    Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mit- igating multimodal hallucination from an eos decision per- spective, 2024. http://arxiv.org/abs/2402. 14545. 3

    2024

  91. [99]

    Rlhf-v: Towards trustwor- thy mllms via behavior alignment from fine-grained correc- tional human feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, et al. Rlhf-v: Towards trustwor- thy mllms via behavior alignment from fine-grained correc- tional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  92. [100]

    Skip: A simple method to reduce hallucination in large vision-language models

    Han Z., Bai Z., Mei H., Xu Q., Zhang C., and Shou M.Z. Skip: A simple method to reduce hallucination in large vision-language models. arXiv preprint arXiv:2402.01345,

    arXiv

  93. [101]

    Hallucination of multimodal large language models: A survey

    Bai Z., Wang P., Xiao T., He T., Han Z., Zhang Z., and Shou MZ. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930, 2024. 3

    2024 arXiv

  94. [102]

    De- biasing multimodal large language models

    Yi-Fan Zhang, Weichen Yu, Qingsong Wen, Xue Wang, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. De- biasing multimodal large language models. arXiv preprint,

  95. [103]

    Truthx: Alle- viating hallucinations by editing large language models in truthful space, 2024

    Shaolei Zhang, Tian Yu, and Yang Feng. Truthx: Alle- viating hallucinations by editing large language models in truthful space, 2024. 3

    2024

  96. [104]

    Beyond hallucinations: En- hancing lvlms through hallucination-aware direct prefer- ence optimization

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Ji- aqi Wang, and Conghui He. Beyond hallucinations: En- hancing lvlms through hallucination-aware direct prefer- ence optimization. arXiv preprint arXiv:2311.16839, 2023. 2

    2023 arXiv

  97. [105]

    Mitigating object hallucination in large vision-language models via classifier- free guidance

    L Zhao, Y Deng, W Zhang, and Q Gu. Mitigating object hallucination in large vision-language models via classifier- free guidance. 2024. 3

    2024

  98. [106]

    Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding

    Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding. arXiv preprint, 2024. 3

    2024

  99. [107]

    Enhancing contextual understanding in large lan- guage models through contrastive decoding, 2024

    Zheng Zhao, Emilio Monti, Jens Lehmann, and Haytham Assem. Enhancing contextual understanding in large lan- guage models through contrastive decoding, 2024. 3

    2024

  100. [109]

    Multilingual contrastive decoding via language-agnostic layers skipping, 2024

    Wenhao Zhu, Sizhe Liu, Shujian Huang, Shuaijie She, Chris Wendler, and Jiajun Chen. Multilingual contrastive decoding via language-agnostic layers skipping, 2024. http://arxiv.org/abs/2407.10795. 2, 3 ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hal...

    2024 arXiv

  101. [2024]

    http://arxiv.org/abs/2408.17150. 3

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.