Pith. sign in

REVIEW 10 cited by

CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.02301 v2 pith:MAAWNH57 submitted 2023-09-05 cs.CV

Hongyu Hu , Jiyuan Zhang , Minyi Zhao , Zhenbang Sun This is my paper
classification cs.CV
keywords hallucinationvlmsciemcontrastiveinstructiontuningevaluationmethod
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Nowadays, the research on Large Vision-Language Models (LVLMs) has been significantly promoted thanks to the success of Large Language Models (LLM). Nevertheless, these Vision-Language Models (VLMs) are suffering from the drawback of hallucination -- due to insufficient understanding of vision and language modalities, VLMs may generate incorrect perception information when doing downstream applications, for example, captioning a non-existent entity. To address the hallucination phenomenon, on the one hand, we introduce a Contrastive Instruction Evaluation Method (CIEM), which is an automatic pipeline that leverages an annotated image-text dataset coupled with an LLM to generate factual/contrastive question-answer pairs for the evaluation of the hallucination of VLMs. On the other hand, based on CIEM, we further propose a new instruction tuning method called CIT (the abbreviation of Contrastive Instruction Tuning) to alleviate the hallucination of VLMs by automatically producing high-quality factual/contrastive question-answer pairs and corresponding justifications for model tuning. Through extensive experiments on CIEM and CIT, we pinpoint the hallucination issues commonly present in existing VLMs, the disability of the current instruction-tuning dataset to handle the hallucination phenomenon and the superiority of CIT-tuned VLMs over both CIEM and public datasets.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Context-guided multi-turn interviews reveal more VLM hallucinations than static benchmarks, and those hallucinations increase with conversational history and false-premise questions.

  2. VOPE: Revisiting Hallucination of Vision-Language Models in Voluntary Imagination Task

    cs.CV 2025-11 conditional novelty 7.0 of 10

    A recheck-based evaluation shows LVLMs often fail to correctly judge the presence of objects they themselves generated, and current mitigation methods do not fix this.

  3. ChartCap: Mitigating Hallucination of Dense Chart Captioning

    cs.CV 2025-08 conditional novelty 7.0 of 10

    A new 565K-pair chart-caption dataset with schema-based dense captions and a reference-free visual consistency metric improves VLM captioning and reduces hallucination.

  4. ARGUS: Hallucination and Omission Evaluation in Video-LLMs

    cs.CV 2025-06 conditional novelty 7.0 of 10

    ARGUS measures hallucination and omission in free-form video captions using LLM-based entailment and temporal alignment, finding that even the best video-LLM still produces roughly 40% hallucinated content.

  5. C-PTQ: Fisher-weighted Channel-wise Sensitivity for Post-training Quantization of MLLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    C-PTQ weights quantization error by per-channel Fisher information of the task loss, improving low-bit accuracy of multimodal LLMs by small margins over existing channel-wise scaling methods.

  6. Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?

    cs.CL 2025-09 conditional novelty 6.0 of 10

    A new 1,012-question benchmark shows LLMs often fail instructions that deliberately invert common training conventions, revealing a measurable gap in counterintuitive instruction following.

  7. VFaith: Do Large Multimodal Models Really Reason on Seen Images Rather than Previous Memories?

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A new benchmark with edited-image question pairs shows that multimodal reasoning models lose accuracy when visual cues change, suggesting their reasoning is often not faithfully tied to the image.

  8. Adaptive Sparse Softmax: An Effective and Efficient Softmax Variant

    cs.LG 2025-08 unverdicted novelty 4.0 of 10

    The preprint's abstract claims a sparse softmax variant that masks non-competitive classes and accelerates training, but the provided body contains an unrelated chart-captioning paper and none of the claimed method.

  9. Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Models

    cs.CV 2025-07 reject novelty 4.0 of 10

    Expert-CFG combines entropy-based uncertainty selection with classifier-free guidance over expert-highlighted text to refine MedVLM outputs, reporting gains on VQA-RAD, SLAKE, and PathVQA.

  10. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools