Pith. sign in

REVIEW 6 major objections 6 minor 1 cited by

Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs

T0 review · 6 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A bounded adversarial noise pattern added to input images reduces object hallucinations in eight vision-language models, without retraining.

desk verdict A genuinely new combination of inference-time adversarial noise and the model's own text anchors, but the evidence mostly supports a threshold shift, not real grounding. read the letter →

arxiv 2501.19164 v2 pith:T7V4OOCA submitted 2025-01-31 cs.CV

Kejia Zhang , Keda Tao , Jiasheng Tang , Huan Wang This is my paper · ORCID
classification cs.CV
keywords objecthallucinationlargevision-languagemodelsvisualadversarialperturbationmitigationzero-gradientoptimizationblack-boxPOPEevaluationBEAF
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a carefully optimized pattern of visual noise, added to an input image before it reaches a large vision-language model (LVM), can suppress object hallucinations without retraining or modifying the model. The method, called visual adversarial perturbation (VAP), treats hallucination as a mismatch between what the model says and what the image actually shows, and searches for a small perturbation that pulls the model's answer toward its own neutral visual description of the image and away from language-driven priors. Across eight LVMs, the perturbed images consistently improve scores on the POPE, BEAF, and CHAIR hallucination benchmarks, while equal-strength Gaussian noise degrades performance. If the effect holds, it gives a practical, model-agnostic input filter for reducing one of the main reliability failures of vision-language systems.

What carries the argument

The mechanism is a three-term adversarial objective computed in text-embedding space. The first term, $L_{s1}$, anchors the answer to visual content by maximizing similarity between the prompted response and the model's null-prompt response; the second and third terms, $L_{s2}$ and $L_{s3}$, reduce parametric knowledge bias by minimizing similarity between responses to the perturbed image and a noised contrastive negative image, under prompted and null conditions. The combined loss is optimized by a zeroth-order (zero-gradient) estimator, so the perturbation is generated in a fully black-box fashion and requires no access to or modification of the LVM's internals. The bounded perturbation itself is the entire intervention.

What would settle it

Run VAP on a matched pair of image sets, one whose null-text descriptions are verified accurate and one whose null-text descriptions contain an object absent from the image; if hallucination scores improve on the second set as much as on the first, the claim that null-text responses are faithful visual anchors is falsified.

Watch

Extended reading notes

Core claim

The central claim is that adversarial perturbation, usually treated as a threat to model reliability, can be repurposed as a corrective input filter. For a frozen LVM $f_\theta$, an image $x$, and a question $c$, VAP maximizes the semantic similarity between the model's prompted answer $f_\theta(x+\delta,c)$ and the model's own null-text description $f_\theta(x+\delta,\emptyset)$, while minimizing similarity between responses to the perturbed image and a heavily distorted contrastive image $\bar{x}$, in both prompted and unprompted settings. The optimized perturbation $\delta$, bounded to $\epsilon = 2$ and produced by a zero-gradient estimator that uses a CLIP text encoder as the similarity surrogate, is added to the image before inference. The paper reports consistent hallucination reductions across eight state-of-the-art LVMs under three evaluation settings, with the largest gains under adversarial sampling where parametric knowledge bias is most exposed.

Load-bearing premise

The load-bearing premise is that the model's null-text response to an image is a faithful description of what is actually in the image; if that neutral response is itself biased or hallucinated, maximizing the prompted answer's similarity to it could push the model deeper into the same error.

Editorial extensions

If this is right

  • On the POPE text-axis benchmark, VAP improves accuracy by up to +2.81 percentage points and F1 by up to +2.09 points across the eight tested LVMs, with the largest gains under adversarial sampling.
  • On the BEAF before-after benchmark, VAP improves the change-aware True Understanding metric by up to +2.31 points and F1TUID by up to +1.74 points, indicating better sensitivity to object presence changes.
  • On the CHAIR captioning benchmark, VAP lowers both CHAIRI and CHAIRS for all eight models; Intern-VL2, for example, drops CHAIRI by 0.68 and CHAIRS by 0.90.
  • VAP requires no retraining or decoding changes and stacks with the decoding-time method VCD for additional CHAIR reductions.
  • Perturbations optimized on a smaller proxy model transfer to a larger model, giving Intern-VL2-1B to 8B a +1.78 F1 gain at one-eighth the inference cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension follows from the paper's yes-ratio shifts: because VAP sharply reduces affirmative responses, much of its POPE gain may come from de-biasing a yes-prior rather than from visual grounding, so a decomposition experiment should separate recalibration from grounding.
  • Because the null-text response is the anchor, the method inherits any bias in that response; on images where the model's neutral description is itself hallucinated, the optimizer would pull the prompted answer toward that hallucination, making the quality of the unprompted visual summary the real ceiling on improvement.
  • The input-level, black-box design suggests a deployment reading the paper does not test: a single small proxy model could generate VAP noise once per image, and that noise could be applied before sending images to any commercial LVM API, at a small fraction of the target model's inference cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper proposes Visual Adversarial Perturbation (VAP), a data-centric method that adds optimized noise to input images to reduce object hallucinations in large vision-language models (LVMs) without modifying the base model. The perturbation is optimized with three similarity-based losses computed in CLIP text-embedding space: maximizing agreement between the prompted response and the null-text response of the perturbed image (Ls1), and minimizing agreement with responses to a distorted version of the image under prompted and unprompted conditions (Ls2 and Ls3). The optimization uses zeroth-order gradient estimation. Experiments on eight LVMs report consistent improvements on POPE, BEAF, and CHAIR benchmarks, with per-model hyperparameters and a proxy-model variant discussed in the appendix.

Significance. The idea of repurposing adversarial perturbations as a data-centric cure for hallucination is novel and timely, and the breadth of the evaluation across eight LVMs and three benchmarks is a clear strength. If the effect is real and reproducible, VAP would be a practical, model-agnostic intervention that is orthogonal to decoding-based methods and requires no model modification. However, the current evidence is weakened by a sign inconsistency in the core objective, a missing term in the reported optimization algorithm, the lack of statistical significance testing, and an unvalidated assumption about the null-text anchor. These issues must be addressed before the central claim can be accepted.

major comments (6)
  1. [§3.2–3.3, Eqs. (7)–(10)] The objective in Eq. (7) sums a maximization term (Ls1, Eq. (3)) with two minimization terms (Ls2, Eq. (5); Ls3, Eq. (6)) under a single loss LS, and Eq. (9) then sets δ = ∇x LS with the perturbed image updated as x̂ = x + αδ. As written, gradient ascent on LS increases Ls2 and Ls3, which is the opposite of the stated goal of reducing similarity to the distorted anchors. Please clarify the sign convention; for example, if LS is to be maximized, it should be LS = Ls1/σ1^2 − Ls2/σ2^2 − Ls3/σ3^2, or the minimization terms should be negated. This is a load-bearing inconsistency that affects every experiment.
  2. [Appendix G, Algorithm 1, Eq. (42)] The zeroth-order gradient estimator in Algorithm 1 (Eq. 42) omits the random direction γn: δ is written as (1/(Nβ)) Σ_n (L^(n)_S − L^(0)_S), which is a scalar, not a vector. Equation (11) correctly includes γn. As written, the algorithm is not a valid gradient estimate and cannot be reproduced. Please correct to δ = (1/(Nβ)) Σ_n (L^(n)_S − L^(0)_S) γ_n.
  3. [§3.2, Eq. (3)] The Ls1 term maximizes similarity between the prompted response and the same model's null-text response fθ(x+δ,∅). The paper assumes this null-text response is a faithful visual description, but no evidence is provided; if the null-text response itself contains hallucinations or generic priors, Ls1 could reinforce them. Please report the hallucination rates of the null-text responses on the same benchmarks, and perform a control experiment where the null-text anchor is replaced by a human-written or independently generated caption. Without this, the claimed grounding mechanism is not established.
  4. [§4.1–4.2, Tables 1–3] No statistical significance tests or confidence intervals are reported. Many improvements are small (e.g., +0.42 F1 for LLaVA-OV in Table 1; +0.18 TU for LLaVA-OV in Table 2), and with 500 POPE samples and 1,000 CHAIR samples these differences may be within sampling noise. Please provide standard deviations over multiple runs or seeds, and paired tests (e.g., bootstrap or McNemar) for the headline metrics.
  5. [§4.1, Table 5] The balancing coefficients σ1, σ2, σ3 and the distortion timestep T are manually tuned per model. The selection procedure and validation set are not described. Since the reported gains are small, please show that a fixed generic configuration (e.g., all σ=1, T=500) still yields consistent improvements across the eight models, or provide a per-model sensitivity analysis on a held-out split.
  6. [§4.1, Eq. (4) and Algorithm 1] The noise schedule μ_T is not defined; the paper only references 'the noise scheduling coefficient at timestep T'. Without specifying the schedule (e.g., cosine, linear), the distorted image generation is not reproducible. Moreover, the claim that the distorted image 'quantifies parametric knowledge bias' is not validated independently. Please provide the schedule and an ablation over T showing its effect on the final hallucination metrics.
minor comments (6)
  1. [Table 1] Table 1 uses '+AVP' instead of '+VAP' in the LVM column; please make the acronym consistent.
  2. [Figure 8] Figure 8(f) contains a typo: 'Is there a bickcle in the image?' should be 'bicycle'.
  3. [§4.2] Section 4.2 refers to 'Figure 5 presents results from specific examples', but Figure 5 shows perturbation strength; the examples are in Figure 4. Please correct the cross-reference.
  4. [§4.2] In Section 4.2, the phrase 'beneficial visual nosise' should be 'beneficial visual noise'.
  5. [§4.1, CHAIR setup] The CHAIR evaluation restricts responses to 30 characters, which deviates from the standard CHAIR protocol; please justify this choice and report full-caption results if available.
  6. [References] The citation 'Anonymous, 2025' is under review and appears in the related work; please update or anonymize appropriately if allowed by the venue.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the VAP objective is a surrogate loss, and the reported hallucination reductions are measured on external benchmarks that are not part of the optimization.

full rationale

The paper's derivation chain does not reduce to its own inputs by construction. The three loss terms (Ls1, Ls2, Ls3) are heuristic objectives that align or contrast the LVM's own text responses using CLIP text-embedding similarity; they are not the evaluation metrics. The central claim that VAP reduces object hallucinations is tested against POPE, BEAF, and CHAIR, all of which use external ground-truth annotations (object presence, before/after scene changes, and caption object matches) that are absent from the loss. No equation in the paper equates the optimization objective with the reported accuracy, F1, TU, or CHAIR scores. The per-model coefficients sigma_i and timestep T in Table 5 are hyperparameters accompanied by a sensitivity analysis (Table 7), not fitted parameters renamed as predictions; the paper does not describe fitting them to the evaluation benchmarks. The assumption that the null-text response f(x+delta, empty) is a faithful visual description is an unvalidated correctness concern, not a circularity, because the paper's empirical evaluation would still be meaningful even if that assumption were false. The only notable self-citation is the anonymized 'PerturboLLaVA' reference used for the linguistic-prior premise, but that premise is also supported by multiple independent prior works cited in the paper, so the self-citation is not load-bearing. Overall, the claimed derivation is self-contained with respect to the external benchmarks, and no prediction reduces to its own inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or ontological entities; the 'beneficial noise' is a perturbation of existing inputs. The central claim rests on several domain assumptions about the validity of CLIP similarity and the null-text response as grounding signals, plus per-model hyperparameters that are tuned without public procedure.

free parameters (3)
  • balancing coefficients σ1, σ2, σ3 = Per model (Table 5), e.g., LLaVA-v1.5: 1.0, 1.0, 1.0; Intern-VL2: 1.0, 0.5, 0.5
    Assigned per model without a disclosed tuning procedure. Parameter sensitivity analysis is only provided for Intern-VL2 (Table 7), and the paper states that 'certain parameters yield competitive performance even when set to zero', suggesting redundancy and instability in the choices.
  • distortion timestep T = Per model (Table 5), ranging from 100 to 800
    Controls the contrastive negative image in Eq. 4. T is chosen per model with no public criterion, and Figure 7 shows a strong dependence of performance on T.
  • perturbation magnitude ϵ and step size α = ϵ = 2, α = 1/255, β = 8/255, N = 10
    Fixed across models, but the choice is not derived. Figure 5 shows performance peaks at intermediate strengths and drops below baseline for ϵ ≥ 16, so the fixed value is sensitive.
assumptions (4)
  • domain assumption CLIP text encoder embedding similarity gψ(·)ᵀgψ(·) is a valid proxy for semantic alignment between LVM responses
    Used to define all loss terms (Eq. 8) and the objective; no validation is provided that this similarity correlates with hallucination or with factual grounding.
  • domain assumption The null-text response fθ(x+δ, ∅) represents a faithful visual semantic description of the image
    Eq. 3 makes this the target for alignment in Ls1. If the null-text response is generic, biased, or hallucinatory, the optimization could reinforce rather than reduce hallucination.
  • domain assumption Zeroth-order gradient estimation with N=10 samples and β=8/255 provides a sufficiently accurate optimization direction
    Eq. 11 and Algorithm 1 rely on this approximation, but no convergence test, variance analysis, or comparison with first-order optimization is provided.
  • ad hoc to paper The distorted image ¯x drawn from the noise schedule in Eq. 4 quantifies parametric knowledge bias
    The connection between this specific Gaussian noise schedule and 'parametric knowledge bias' is asserted in Section 3.2, not derived or independently supported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs." pith.science (2026). https://pith.science/paper/T7V4OOCA

@misc{pith2026250119164,
  author       = {Pith},
  title        = {Pith review of: Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T7V4OOCA}},
  note         = {Machine review of arXiv:2501.19164}
}
read the original abstract

Large vision-language models (LVMs) extend large language models (LLMs) with visual perception capabilities, enabling them to process and interpret visual information. A major challenge compromising their reliability is object hallucination that LVMs may generate plausible but factually inaccurate information. We propose a novel visual adversarial perturbation (VAP) method to mitigate this hallucination issue. VAP alleviates LVM hallucination by applying strategically optimized visual noise without altering the base model. Our approach formulates hallucination suppression as an optimization problem, leveraging adversarial strategies to generate beneficial visual perturbations that enhance the model's factual grounding and reduce parametric knowledge bias. Extensive experimental results demonstrate that our method consistently reduces object hallucinations across 8 state-of-the-art LVMs, validating its efficacy across diverse evaluations.

Figures

Figures reproduced from arXiv: 2501.19164 by the authors.

Figure 1
Figure 1. We introduce VAP (visual adversarial perturbation), a novel approach that strategically injects beneficial visual noise to mitigate object hallucination in LVMs without altering the complex base model. Our method consistently improves performance across 8 state-of-the-art LVMs under the POPE hallucination evaluation setting (Li et al., 2023). Abstract Large vision-language models (LVMs) extend large language models … view at source ↗
Figure 2
Figure 2. Detailed Overview of our proposed method. The VAP method generates beneficial visual noise by leveraging adversarial knowledge through the optimization of three strategies: (1) maximizing the semantic alignment between the LVM’s response and the visual content to preserve the semantic consistency of the image, (2) minimizing the response similarity between the original and distorted visual content through noise-indu… view at source ↗
Figure 3
Figure 3. Comparison of the original images with our pro [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Examples of the vision-question-answer (VQA) tasks before and after applying our proposed method to the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Performance of the Intern-VL2 model (Chen et al., 2024b) under varying levels of perturbation strength in the POPE adversarial setting. We test the model’s performance with varying perturbations applied to the original images. alleviates object hallucination by introdu…
Figure 6
Figure 6. Figure 6: Comparison of the original images with our proposed VAP and Gaussian noise of equal strength ( [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: presents the performance variations of LVMs with increasing levels of distortion applied to negative distorted images, as described in Equation (4). As the distortion level T increases, the model’s hallucination initially decreases but subsequently rises, with performa…
Figure 8
Figure 8. Figure 8: Illustrative examples from the POPE hallucination evaluation across eight large vision-language models: (a) [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MissingBench-Verified: Probing Vision-Language Models' Inability to Detect Missing Object Parts

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Ten leading VLMs mostly fail to report removed essential object parts as missing, and simulated detector evidence, image tools, longer reasoning, and an easier fine-tune barely improve accuracy.

Reference graph

Works this paper leans on

44 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Perturbo LL a VA : Reducing multimodal hallucinations with perturbative visual training

    Anonymous. Perturbo LL a VA : Reducing multimodal hallucinations with perturbative visual training. In Submitted to ICLR, 2025. Under review

    work page 2025

  3. [3]

    Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., and Shou, M. Z. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930, 2024

    arXiv 2024

  4. [4]

    F., G\'omez, L., and Karatzas, D

    Biten, A. F., G\'omez, L., and Karatzas, D. Let there be a clock on the beach: Reducing object hallucination in image captioning. In WACV, 2022

    work page 2022

  5. [5]

    Sharegpt4v: Improving large multi-modal models with better captions

    Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi-modal models with better captions. In ECCV, 2024 a

    work page 2024

  6. [6]

    Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models

    Chen, P.-Y., Zhang, H., Sharma, Y., Yi, J., and Hsieh, C.-J. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, 2017

    work page 2017

  7. [7]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., Li, B., Luo, P., Lu, T., Qiao, Y., and Dai, J. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, 2024 b

    work page 2024

  8. [8]

    Halc: Object hallucination reduction via adaptive focal-contrast decoding

    Chen, Z., Zhao, Z., Luo, H., Yao, H., Li, B., and Zhou, J. Halc: Object hallucination reduction via adaptive focal-contrast decoding. In ICML, 2024 c

    work page 2024

Show all 44 references
  1. [9]

    K., and Lim, S.-N

    Cui, X., Aparcedo, A., Jang, Y. K., and Lim, S.-N. On the robustness of large multimodal models against image adversarial attacks. In CVPR, 2024

    2024

  2. [10]

    N., and Hoi, S

    Dai, W., Li, J., LI, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P. N., and Hoi, S. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurIPS, 2023

    2023

  3. [11]

    K., Aitchison, M., Orseau, L., et al

    Deletang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., et al. Language modeling is compression. In ICLR, 2024

    2024

  4. [12]

    Multi-modal hallucination control by visual information grounding

    Favero, A., Zancato, L., Trager, M., Choudhary, S., Perera, P., Achille, A., Swaminathan, A., and Soatto, S. Multi-modal hallucination control by visual information grounding. In CVPR, 2024

    2024

  5. [13]

    Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., Manocha, D., and Zhou, T. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In CVPR, 2024

    2024

  6. [14]

    Detecting and preventing hallucinations in large vision language models

    Gunjal, A., Yin, J., and Bas, E. Detecting and preventing hallucinations in large vision language models. In AAAI, 2024

    2024

  7. [15]

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation

    Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., and Yu, N. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In CVPR, 2024

    2024

  8. [16]

    Exploiting semantic reconstruction to mitigate hallucinations in vision-language models

    Kim, M., Kim, M., Bae, J., Choi, S., Kim, S., and Chang, B. Exploiting semantic reconstruction to mitigate hallucinations in vision-language models. In ECCV, 2024

    2024

  9. [17]

    S., Naseer, M., Das, A., Khan, S., and Khan, F

    Kuckreja, K., Danish, M. S., Naseer, M., Das, A., Khan, S., and Khan, F. S. Geochat: Grounded large vision-language model for remote sensing. In CVPR, 2024

    2024

  10. [18]

    What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024

    Lauren c on, H., Tronchon, L., Cord, M., and Sanh, V. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024

    2024 arXiv

  11. [19]

    Mitigating object hallucinations in large vision-language models through visual contrastive decoding

    Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., and Bing, L. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In CVPR, 2024

    2024

  12. [20]

    Llava-onevision: Easy visual task transfer

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  13. [21]

    Evaluating object hallucination in large vision-language models

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, X., and Wen, J.-R. Evaluating object hallucination in large vision-language models. In EMNLP, 2023

    2023

  14. [22]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L. Mitigating hallucination in large multi-modal models via robust instruction tuning. In ICLR, 2023 a

    2023

  15. [23]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In NeurIPS, 2023 b

    2023

  16. [24]

    A survey on hallucination in large vision-language models

    Liu, H., Xue, W., Chen, Y., Chen, D., Zhao, X., Wang, K., Hou, L., Li, R., and Peng, W. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253, 2024 a

    2024 arXiv

  17. [25]

    Paying more attention to image: A training-free method for alleviating hallucination in lvlms

    Liu, S., Zheng, K., and Chen, W. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. In ECCV, 2024 b

    2024

  18. [26]

    Ovis: Structural embedding alignment for multimodal large language model

    Lu, S., Li, Y., Chen, Q.-G., Xu, Z., Luo, W., Zhang, K., and Ye, H.-J. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797, 2024

    2024 arXiv

  19. [27]

    Cheap and quick: Efficient vision-language instruction tuning for large language models

    Luo, G., Zhou, Y., Ren, T., Chen, S., Sun, X., and Ji, R. Cheap and quick: Efficient vision-language instruction tuning for large language models. In NeurIPS, 2023

    2023

  20. [28]

    P., and Vondrick, C

    Menon, S., Chandratreya, I. P., and Vondrick, C. Task bias in contrastive vision-language models. IJCV, 132 0 (6): 0 2026--2040, 2024

    2026

  21. [29]

    and Spokoiny, V

    Nesterov, Y. and Spokoiny, V. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17 0 (2): 0 527--566, 2017

    2017

  22. [30]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, 2021

    2021

  23. [31]

    A., Burns, K., Darrell, T., and Saenko, K

    Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., and Saenko, K. Object hallucination in image captioning. In EMNLP, 2018

    2018

  24. [32]

    In NeurIPS, 2022

    Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S., Crowson, K., Schmidt, L., Kaczmarczyk, R., and Jitsev, J. In NeurIPS, 2022

    2022

  25. [33]

    What does clip know about a red circle? visual prompt engineering for vlms

    Shtedritski, A., Rupprecht, C., and Vedaldi, A. What does clip know about a red circle? visual prompt engineering for vlms. In ICCV, 2023

    2023

  26. [34]

    Winoground: Probing vision and language models for visio-linguistic compositionality

    Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., and Ross, C. Winoground: Probing vision and language models for visio-linguistic compositionality. In CVPR, 2022

    2022

  27. [35]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024 a

    2024 arXiv

  28. [36]

    Visionllm: Large language model is also an open-ended decoder for vision-centric tasks

    Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo, P., Lu, T., Zhou, J., Qiao, Y., et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. In NeurIPS, 2024 b

    2024

  29. [37]

    Llm2clip: Powerful language model unlock richer visual representation

    Wu, A., Yang, Y., Luo, X., Yang, Y., Wang, C., Hu, L., Dai, X., Chen, D., Luo, C., Qiu, L., et al. Llm2clip: Powerful language model unlock richer visual representation. In NeurIPS Workshop, 2024 a

    2024

  30. [38]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding

    Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024 b

    2024 arXiv

  31. [39]

    Can i trust your answer? visually grounded video question answering

    Xiao, J., Yao, A., Li, Y., and Chua, T.-S. Can i trust your answer? visually grounded video question answering. In CVPR, 2024

    2024

  32. [40]

    Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models

    Xu, P., Shao, W., Zhang, K., Gao, P., Liu, S., Lei, M., Meng, F., Huang, S., Qiao, Y., and Luo, P. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. IEEE TPAMI, pp.\ 1--18, 2024

    2024

  33. [41]

    Beaf: Observing before-after changes to evaluate hallucination in vision-language models

    Ye-Bin, M., Hyeon-Woo, N., Choi, W., and Oh, T.-H. Beaf: Observing before-after changes to evaluate hallucination in vision-language models. In ECCV, 2024

    2024

  34. [42]

    Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data

    Yu, Q., Li, J., Wei, L., Pang, L., Ye, W., Qin, B., Tang, S., Tian, Q., and Zhuang, Y. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In CVPR, 2024

    2024

  35. [43]

    M., and Lin, M

    Zhao, Y., Pang, T., Du, C., Yang, X., LI, C., Cheung, N.-M. M., and Lin, M. On evaluating adversarial robustness of large vision-language models. In NeurIPS, 2023

    2023

  36. [44]

    Analyzing and mitigating object hallucination in large vision-language models

    Zhou, Y., Cui, C., Yoon, J., Zhang, L., Deng, Z., Finn, C., Bansal, M., and Yao, H. Analyzing and mitigating object hallucination in large vision-language models. In ICLR, 2024

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.