Pith. sign in

REVIEW 5 major objections 6 minor 78 references

VORD: Visual Ordinal Calibration for Mitigating Object Hallucinations in Large Vision-Language Models

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VORD claims that enforcing one ordinal inequality on token probabilities suppresses object hallucinations in large vision-language models.

desk verdict A simple contrastive decoding method with consistent gains, but the paper's own motivating observation undercuts its core mechanism and needs to be resolved before I would trust it. read the letter →

arxiv 2412.15739 v1 pith:QMNB5HRP submitted 2024-12-20 cs.CV

Dexter Neo , Tsuhan Chen This is my paper · ORCID
classification cs.CV
keywords objecthallucinationlargevision-languagemodelscalibrationcontrastivedecodingordinalrankinglossMixupexpectederrorPOPEbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VORD claims that a single ordinal constraint can suppress object hallucinations in large vision-language models: for each candidate token, the probability the model assigns when viewing the true image should be at least as high as the probability it assigns when viewing a deliberately corrupted version of the same image, plus a margin. If this constraint is enforced at decoding time by setting violating tokens to zero, or trained into the model as a ranking penalty added to cross-entropy, the paper argues that generated text becomes more faithful and better calibrated. On the POPE hallucination benchmark, VORD decoding improves accuracy and F1 by up to about +2.2 and +2.3 points over regular decoding across three backbone models, while achieving lower expected calibration error. VORD loss fine-tuning raises POPE accuracy by up to +2.9 points, and combining both forms gives the best results. The practical interest is that hallucination control and confidence calibration are treated as one mechanism rather than separate add-ons.

What carries the argument

The load-bearing object is the ordinal mask δθ = Pθ(yt|v,x) + mθ >= Pθ(yt|v_hat,x), where v_hat is a corrupted copy of the image built by Mixup or diffusion noise and mθ is an adaptive margin mθ = (1/π) arccos( (f̄θ(v)·f̄θ(v_hat))/(||f̄θ(v)|| ||f̄θ(v_hat)||) ), the normalized angular distance between the averaged visual tokens of the clean and corrupted images. In VORD Decoding the mask is applied at each autoregressive step, zeroing any token that violates the inequality and then truncating further with an adaptive plausibility constraint; in VORD Loss the same inequality is converted into a convex, piecewise-quadratic ranking penalty max(·,0)^ψ whose gradients pull the clean-image probabilities above corrupted-image probabilities. The adaptive margin is what lets the method set its own strictness from the visual similarity of the image pair, which the ablations show is necessary because fixed margins either under-filter or over-penalize.

What would settle it

Take a set of images with known ground-truth object labels and corrupt each image with the same Mixup or diffusion recipe used in the paper; count how often a ground-truth object token has strictly lower probability on the clean image than on the corrupted image. If that violation rate is non-negligible on correct tokens, then the ordinal mask in Eq. (4) is deleting valid content, and the observed POPE gains would be explained by filtering rather than by a true ordinal regularity.

Watch

Extended reading notes

Core claim

The paper's central claim is that token probabilities in LVLMs are not naturally ordinal under visual corruption, and that enforcing ordinality fixes hallucination. Concretely, for a clean image v, a corrupted image v_hat, and a text prompt x, the paper posits that tokens consistent with the image should satisfy Pθ(yt|v,x) >= Pθ(yt|v_hat,x); VORD turns this into a hard mask P_VORD = Pθ(yt|v,x) if the inequality with margin mθ holds, and 0 otherwise, and into a convex loss L_vord = max(Pθ(y|v_hat,x) - Pθ(y|v,x) + mθ, 0)^ψ added to cross-entropy. The paper reports that this suppresses hallucinated objects like <person> or <loudspeaker>, improves accuracy, precision, recall and F1 on POPE, raises overall MME perception scores, and produces shorter, more detailed open-ended answers on LLaVA-Bench, all while lowering expected calibration error.

Load-bearing premise

VORD stands on the assumption that for every token that matters, the model's probability on the clean image exceeds its probability on the corrupted image by at least the margin; if a genuinely correct token ever scores higher on the corrupted image, the decoding rule sets its probability to zero and the correct answer is destroyed.

Editorial extensions

If this is right

  • A training-free decoder can lower object hallucination rates on POPE and MME without retraining, by comparing one forward pass on the image with one forward pass on a corrupted image.
  • The same ordinal principle works as a fine-tuning objective: adding max(Pθ(y|v_hat,x) - Pθ(y|v,x) + mθ, 0)^ψ to cross-entropy improves POPE accuracy and F1, with the squared variant (ψ=2) slightly ahead.
  • Calibration improves alongside accuracy: VORD reports the lowest expected calibration error among compared decoders on the tested backbone LVLMs, so hallucination suppression does not come at the cost of overconfident predictions.
  • Combining VORD loss with VORD decoding yields the best overall scores, so the training-time and decoding-time forms reinforce each other.
  • Because the method only needs image pairs and token probabilities, it is portable to any LVLM with a vision encoder, which is why the paper tests three backbone architectures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the ordinal rule is not tied to object categories; it should apply to attribute-level hallucinations such as color, position, or count whenever the corrupted image lowers the true attribute's probability, so VORD may generalize beyond the existence-focused POPE benchmark.
  • Editorial extension: a testable implication the paper does not pursue is that using a corruption which leaves semantically irrelevant tokens nearly unchanged could make VORD behave like a pure linguistic-prior filter; comparing Mixup against text-only corruptions would separate visual grounding from language-prior suppression.
  • Editorial extension: if the ordinal assumption holds only for severe corruptions, as the ablation suggests, a practical extension would be to schedule corruption intensity per image, adapting the margin so that mild corruptions still activate the constraint without deleting valid tokens.
  • Editorial extension: the authors note VORD could extend to LLMs and NLP via text modifications; an implicit consequence is that any input perturbation that selectively damages signal while preserving structure could support the same ordinal calibration, making the idea a general confidence-grounding recipe.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes VORD (Visual Ordinal Calibration), a method to reduce object hallucinations in large vision-language models by enforcing an ordinal relationship between token probabilities under clean and corrupted images. VORD has two forms: a training-free decoding filter that zeroes out tokens whose probability on the modified image exceeds their probability on the original image plus an adaptive margin, and a trainable loss that penalizes such violations. The adaptive margin is computed from the angular similarity between the visual features of the original and modified images. Experiments on POPE, MME, and LLaVA-Bench with LLaVA-v1.5, Qwen-VL, and InstructBLIP report consistent improvements in accuracy, F1, and ECE over regular decoding and against VCD and OPERA baselines, along with GPT-4oV-assessed gains in response accuracy and detailedness.

Significance. If the empirical claims hold, VORD offers a lightweight, training-free intervention and a complementary regularizer that improve both hallucination metrics and calibration across several backbone LVLMs. The code is publicly available, and the evaluation includes error bars over reruns, which is a strength. However, the significance is currently conditional because the central mechanistic justification—the ordinal assumption in Eq. (3)—is internally contradicted by the paper's own motivating observations, and the formal analysis of the loss function contains mathematical errors. The method may still work as an effective heuristic, but the paper does not yet establish the ordinal-calibration story it advertises.

major comments (5)
  1. [Sec. 3.4 and Eq. (4)] The motivating observation in Sec. 3.4 states that desired tokens (shown in green in Fig. 2) 'did not behave as expected' under Mixup, meaning they violate the ordinal relation in Eq. (3). Yet the decoding rule in Eq. (4) rejects exactly those tokens by setting their probability to zero when Pθ(yt|v,x) + mθ < Pθ(yt|v̂,x). The adaptive plausibility constraint in Eq. (5) only limits rejection to high-probability tokens; it does not repair the ordering. This is an internal inconsistency in the central mechanism: the paper argues from the failure of ordinality and then enforces ordinality as the acceptance criterion. The paper provides no quantitative measurement of how often ground-truth or human-judged correct tokens are rejected by Eq. (4). Without such a per-token or per-sequence analysis, the observed benchmark gains could result from aggressive suppression of all non-trivial tokens rather than from a sound ordinal property. This needs to be addressed with a direct measurement of correct-token survival rates or an equivalent analysis.
  2. [Sec. 4.2.1, Eqs. (8)–(9)] The convexity proof is mathematically unsupported. In Eq. (9), θ is the full parameter vector of the LVLM, so ∂²gθ/∂²θ is a Hessian matrix, not a scalar, and the statement that 'the second derivative is always positive' is not well-defined for a matrix. Moreover, (∂gθ/∂θ)² is an outer product, and the coefficient (ψ² − ψ) is negative when 0 < ψ < 1, so the second term is not necessarily positive semidefinite. The first term ψ g^{ψ−1} H is positive semidefinite only if H is positive semidefinite, which is not established. The claim that the VORD loss is convex in θ is therefore not proven. The authors should either provide a correct convexity argument (e.g., convexity in the functional output space) or remove the convexity claim and instead describe the loss as a heuristic penalty.
  3. [Sec. 4.2, Eq. (8) and Eq. (10)] The gradient analysis in Eq. (8) treats the margin mθ as a constant when differentiating gθ = Pθ(y|v̂,x) − Pθ(y|v,x) + mθ. However, Eq. (10) defines mθ as a function of the visual encoder parameters θ (through fθ(v) and fθ(v̂)). The correct gradient should include ∂mθ/∂θ, which is absent. Consequently, the reported gradient of Lvord is incomplete, and the same issue propagates to the second-derivative analysis. This is load-bearing for the training variant: the actual loss that is optimized differs from the analyzed one, and the claimed behavior (e.g., not penalizing when the ordinal property is obeyed) may not hold if the margin changes during optimization. The authors should correct the gradient derivation or explicitly state that they treat mθ as a fixed, non-differentiable scalar during backpropagation.
  4. [Appendix B, Table 8] The corruption ablation shows that VORD underperforms VCD on mild corruptions: Gaussian Blur (F1 80.83 vs. 82.18), Jpeg Compression (81.99 vs. 82.25), and Saturate (81.46 vs. 82.11). The authors attribute this to smaller margins and state that the corruption must be 'severe enough' for VORD to work. This is a direct acknowledgment that the ordinal assumption in Eq. (3) is corruption-dependent, and it weakens the paper's central claim that VORD calibrates based on a general ordinal relationship between modified image pairs. The paper should provide a quantitative characterization of when Eq. (3) holds and when it fails, and should report the survival rate of correct tokens under the recommended Mixup setting. Without such evidence, the choice of Mixup as the default corruption appears to be tuned to make the method work, rather than derived from the proposed principle.
  5. [Sec. 5.3 and Table 7] The abstract and Sec. 5.3 claim that VORD delivers 'better calibration,' but Table 7 shows that the VORD loss consistently increases ECE relative to the baseline: for example, on A-OKVQA Random with LLaVA-1.5-7B, ECE rises from 2.82 to 2.94 (ψ=1) and 3.01 (ψ=2); on MSCOCO Popular it rises from 3.14 to 3.31 and 3.56; and on GQA Adversarial from 5.63 to 5.82 and 5.85. The calibration improvement is only demonstrated for VORD decoding in Table 1, not for VORD loss. The claims should be qualified to distinguish between the two variants, and the ECE results for the loss variant should be discussed honestly.
minor comments (6)
  1. [Algorithm 1 and Algorithm 2] The algorithms use the notation 'ˆv = α ∗ v1 + (1 − α) ∗ v2', but the text defines the Mixup coefficient as λ drawn from Beta(α, α). Using α both for the Beta distribution shape parameter and for the mixing weight is confusing; please rename the mixing weight (e.g., λ) consistently.
  2. [Eq. (3)] The statement '1.0 ≥ λi ≥ λj ⇔ Pθ(yt|v,x) ≥ Pθ(yt|v̂,x)' uses λi and λj without a clear definition. Only a single λ is introduced in the preceding sentence; please clarify what λi and λj denote (e.g., two different mixing strengths) and whether the equivalence is an empirical claim or a definition.
  3. [Table 3 and Sec. 6] The text states that VORD generates shorter responses, but Table 3 shows that for Qwen-VL, VORD has a longer average length (13.13) than regular decoding (10.41) and VCD (10.36). Please reconcile this observation with the 'Short & Sweet' claim.
  4. [Sec. 5.3] Figure 5 is referenced as showing the 'squared variant (ψ = 2) highlighted in magenta,' but the manuscript appears to be in black-and-white; consider using a distinguishable marker pattern or a color-blind-safe palette.
  5. [Sec. 4.3] The paper claims the adaptive margin 'eliminates the need for hyperparameter tuning,' but the method still requires choosing the corruption type (e.g., Mixup), the Mixup strength α, and the plausibility threshold β. Please temper this claim to reflect the remaining hyperparameters.
  6. [Throughout] There are several minor grammatical issues: the abstract reads 'we present VORD a simple and effective method' (missing comma), and Sec. 4.1.2 has 'Higher values ofβ results' (grammar/spacing). A careful proofread is recommended.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; VORD's ordinal criterion is an applied hypothesis evaluated on held-out benchmarks, not a fitted or self-referential derivation.

full rationale

The paper's central mechanism Eq. (4) is a decoding rule, not a derived prediction: it filters tokens by comparing probabilities on clean and corrupted images plus a margin, and the claimed benefit is then measured on held-out POPE, MME, and LLaVA-Bench labels. The margin in Eq. (10) is a fixed functional form of the cosine similarity between the model's visual embeddings of v and v_hat; it is not tuned to benchmark outcomes, and no fitted parameter is renamed as a prediction. The VORD loss in Eq. (6) is likewise a fixed ordinal regularizer combined with cross-entropy, not an identity with the evaluation metric. No load-bearing self-citation appears: the paper cites VCD for adaptive plausibility constraints, but VCD is external prior work, and no uniqueness theorem or prior result by the same authors is invoked to forbid alternatives. The paper's own Sec. 3.4 admits that desired tokens 'did not behave as expected' under Mixup, and Appendix B shows that mild corruptions with small margins yield weaker gains than VCD (e.g., Gaussian Blur F1 80.83 vs 82.18). These are validity and robustness concerns about the ordinal assumption, not circular reductions: the assumption is externally testable, and the paper provides held-out evidence both for and against it. There is also an internal inconsistency in the margin semantics (Eq. (4) adds mθ to the clean probability, making larger mθ more lenient, while the Tab. 4 discussion says excessively large margins 'over-penalize'), but that is a correctness issue and is explicitly outside the circularity criterion. Overall, the central claim is empirically contingent rather than equivalent to its inputs by construction.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a small set of design choices: the ordinal ranking assumption, the adaptive margin formula, and the hyperparameters β, ψ, and mixup α. No new physical or conceptual entities are introduced. The main free parameters are the threshold and exponents, while the margin is a deterministic function of the model's own embeddings, shifting the tuning burden to the corruption choice.

free parameters (4)
  • Adaptive margin mθ = Not grid-searched; computed via Eq. (10), with observed values 0.027 to 0.131 across corruptions (Table 8)
    Controls the stringency of token acceptance. The scale is set by the chosen corruption type; the paper selects corruptions that yield large margins (Mixup/Diffusion) for the main results, making the effective operating point dependent on that choice.
  • β (adaptive plausibility threshold) = 0.2
    Threshold for retaining plausible tokens in Eq. (5), adopted from VCD [26]. It is a human-set hyperparameter controlling truncation strength.
  • ψ (power term) = 1.0 or 2.0
    Exponent in the VORD loss (Eq. 6); the paper recommends ψ = 2 based on empirical gains.
  • Mixup Beta parameter α = 1.0
    Parameter of the Beta distribution used to sample the mixing coefficient in image modification; Beta(1,1) is uniform on [0,1].
assumptions (3)
  • domain assumption Ordinal ranking assumption (Eq. 3): for relevant tokens, Pθ(yt|v,x) >= Pθ(yt|v_hat,x) as corruption strength increases
    Stated in Sec. 3.4 and used to derive the mask in Eq. (4) and the loss in Eq. (6). The paper notes LVLMs do not always obey this (Fig. 2), yet the method enforces it, making this the central premise.
  • domain assumption Adaptive margin mθ from cosine similarity (Eq. 10) is an appropriate penalty scale
    Used in both decoding and loss. The paper assumes that visual angular distance between original and modified embeddings correlates with the probability gap that should be required, without independent validation of this mapping.
  • domain assumption The corrupted image exposes hallucination-prone tokens that should be suppressed
    Sec. 3.3 and Appendix B. The paper relies on Mixup/Diffusion creating enough uncertainty that hallucinated tokens get higher probability on the corrupted image, but Table 8 shows this fails for mild corruptions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VORD: Visual Ordinal Calibration for Mitigating Object Hallucinations in Large Vision-Language Models." pith.science (2026). https://pith.science/paper/QMNB5HRP

@misc{pith2026241215739,
  author       = {Pith},
  title        = {Pith review of: VORD: Visual Ordinal Calibration for Mitigating Object Hallucinations in Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QMNB5HRP}},
  note         = {Machine review of arXiv:2412.15739}
}
read the original abstract

Large Vision-Language Models (LVLMs) have made remarkable developments along with the recent surge of large language models. Despite their advancements, LVLMs have a tendency to generate plausible yet inaccurate or inconsistent information based on the provided source content. This phenomenon, also known as ``hallucinations" can have serious downstream implications during the deployment of LVLMs. To address this, we present VORD a simple and effective method that alleviates hallucinations by calibrating token predictions based on ordinal relationships between modified image pairs. VORD is presented in two forms: 1.) a minimalist training-free variant which eliminates implausible tokens from modified image pairs, and 2.) a trainable objective function that penalizes unlikely tokens. Our experiments demonstrate that VORD delivers better calibration and effectively mitigates object hallucinations on a wide-range of LVLM benchmarks.

Figures

Figures reproduced from arXiv: 2412.15739 by the authors.

Figure 1
Figure 1. VORD suppresses hallucinated objects such as [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Visual corruptions, such as random noise and image mix [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. VORD penalizes tokens with higher conditional probabilities from [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Our experiments demonstrate that finetuning LLaVA [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: VORD produces accurate and detailed outputs, mitigating object hallucinations across a wide range of different visual cues. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Example of prompt-card used on GPT4o, evaluating the accuracy and detailedness of responses. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 65 canonical work pages

  1. [1]

    Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing

    Vedika Agarwal, Rakshith Shetty, and Mario Fritz. Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 3

    work page 2020

  2. [2]

    Ana- lyzing the behavior of visual question answering models

    Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Ana- lyzing the behavior of visual question answering models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1955–1960, Austin, Texas, 2016. Association for Computational Linguistics. 3

    work page 2016

  3. [3]

    Agla: Mitigating object hallucinations in large vision- language models with assembly of global and local attention

    Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, Qianying Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large vision- language models with assembly of global and local attention. ArXiv, abs/2406.12718, 2024. 3

    arXiv 2024

  4. [4]

    Let there be a clock on the beach: Reducing object hal- lucination in image captioning

    Ali Furkan Biten, Llu ´ıs G ´omez, and Dimosthenis Karatzas. Let there be a clock on the beach: Reducing object hal- lucination in image captioning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1381–1390, 2022. 3

    work page 2022

  5. [5]

    Driving with llms: Fusing object-level vec- tor modality for explainable autonomous driving

    Long Chen, Oleg Sinavski, Jan H ¨unermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with llms: Fusing object-level vec- tor modality for explainable autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024. 1

    2024

  6. [6]

    HALC: Object hallucination reduc- tion via adaptive focal-contrast decoding

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. HALC: Object hallucination reduc- tion via adaptive focal-contrast decoding. In Proceedings of the 41st International Conference on Machine Learning , pages 7824–7846. PMLR, 2024. 2, 3

    work page 2024

  7. [7]

    Calibrating deep neural networks by pairwise constraints

    Jiacheng Cheng and Nuno Vasconcelos. Calibrating deep neural networks by pairwise constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13709–13718, 2022. 2

    work page 2022

  8. [8]

    InstructBLIP: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Thirty- seventh Conference on Neural Information Processing Sys- tems, 2023. 6

    work page 2023

Show all 78 references
  1. [9]

    BERT: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human L...

    2019

  2. [10]

    Hierarchi- cal neural story generation

    Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchi- cal neural story generation. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers) , pages 889–898, Melbourne, Australia, 2018. Association for Computational L...

    2018

  3. [11]

    Multi-modal hal- lucination control by visual information grounding

    Alessandro Favero, Luca Zancato, Matthew Trager, Sid- dharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hal- lucination control by visual information grounding. In Pro- ceedings of the IEEE/CVF Conference on Computer V...

    2024

  4. [12]

    Beam search strate- gies for neural machine translation

    Markus Freitag and Yaser Al-Onaizan. Beam search strate- gies for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation , pages 56– 60, Vancouver, 2017. Association for Computational Lin- guistics. 4

    2017

  5. [13]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of The 33rd International Confer- ence on Machine Learning , pages 1050–1059, New York, New York, USA, 2016. PMLR. 2

    2016

  6. [14]

    Ido Galil, Mohammed Dabbah, and Ran El-Yaniv. What can we learn from the selective prediction and uncertainty esti- mation performance of 523 imagenet classifiers? In The Eleventh International Conference on Learning Representa- tions, 2023. 2

    2023

  7. [15]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 3

    2017

  8. [16]

    A stitch in time saves nine: A train-time reg- ularizing loss for improved neural network calibration

    Ramya Hebbalaguppe, Jatin Prakash, Neelabh Madan, and Chetan Arora. A stitch in time saves nine: A train-time reg- ularizing loss for improved neural network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 16081–16090,

  9. [17]

    Dietterich

    Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and per- turbations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. 3, 13

    2019

  10. [18]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In In- ternational Conference on Learning Representations , 2020. 4

    2020

  11. [19]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InIn- ternational Conference on Learning Representations , 2022. 6

    2022

  12. [20]

    Scaling up vision-language pre-training for image captioning

    Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17980–17989, 2022. 1

    2022

  13. [21]

    Movienet: A holistic dataset for movie un- derstanding

    Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie un- derstanding. In European Conference on Computer Vision,

  14. [22]

    Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Confer...

    2024

  15. [23]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and composi- tional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 6

    2019

  16. [24]

    Langsuit-e: Controlling, planning, and interacting with large language models in embodied text environments

    Zixia Jia, Mengmeng Wang, Baichen Tong, and Zilong Zheng. Langsuit-e: Controlling, planning, and interacting with large language models in embodied text environments. In Findings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024. 1

    2024

  17. [25]

    Adam Tauman Kalai and Santosh S. Vempala. Calibrated language models must hallucinate. Proceedings of the 56th Annual ACM Symposium on Theory of Computing, 2023. 2

    2023

  18. [26]

    Mitigating object hallucinations in large vision-language models through vi- sual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shi- jian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through vi- sual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  19. [27]

    BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceed- ings of the 39th International Conference on Machine Learn- ing, pages 12888–12900. PMLR, 2022. 2

    2022

  20. [28]

    What does BERT with vision look at? In Proceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics , pages 5265–5275, On- line, 2020

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. What does BERT with vision look at? In Proceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics , pages 5265–5275, On- line, 2020. Association for Computational Linguistics. 2

    2020

  21. [29]

    Contrastive decoding: Open-ended text genera- tion as optimization

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text genera- tion as optimization. InProceedings of the 61st Annual Meet- ing of the Association for Computational Ling...

  22. [30]

    Evaluating object hallucination in large vision- language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 292–305, Singapore, 2023. Association for ...

    2023

  23. [31]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 26689–26699, 2024. 2

    2024

  24. [32]

    TruthfulQA: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3214–3252, Dublin, Ireland, 2022. Association for...

    2022

  25. [33]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014. 6

    2014

  26. [34]

    The devil is in the margin: Margin-based label smooth- ing for network calibration

    Bingyuan Liu, Ismail Ben Ayed, Adrian Galdran, and Jose Dolz. The devil is in the margin: Margin-based label smooth- ing for network calibration. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 80–88, 2022. 2

    2022

  27. [35]

    Class adaptive network calibration

    Bingyuan Liu, J ´erˆome Rony, Adrian Galdran, Jose Dolz, and Ismail Ben Ayed. Class adaptive network calibration. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16070–16079, 2023. 2

    2023

  28. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural Information Processing Systems, pages 34892–34916. Curran Associates, Inc., 2023. 2, 6

    2023

  29. [37]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, 2024. 2, 6

    2024

  30. [38]

    SimCLS: A simple framework for contrastive learning of abstractive summarization

    Yixin Liu and Pengfei Liu. SimCLS: A simple framework for contrastive learning of abstractive summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V...

    2021

  31. [39]

    Curved scene text detection via transverse and longitudinal sequence connection

    Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Sheng Zhang. Curved scene text detection via transverse and longitudinal sequence connection. Pattern Recognition, 90: 337–345, 2019. 6

    2019

  32. [40]

    Roberta: A robustly optimized bert pretraining approach, 2019

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. 2

    2019

  33. [41]

    BRIO: Bringing order to abstractive summarization

    Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neu- big. BRIO: Bringing order to abstractive summarization. In Proceedings of the 60th Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Pa- pers), pages 2890–2903, Dublin, Ireland, 2022. Assoc...

    2022

  34. [42]

    Soft augmentation for image classifica- tion

    Yang Liu, Shen Yan, Laura Leal-Taix ´e, James Hays, and Deva Ramanan. Soft augmentation for image classifica- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 16241– 16250, 2023. 2

    2023

  35. [43]

    Dolphins: Multimodal language model for driving, 2023

    Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving, 2023. 1

    2023

  36. [44]

    Deepart: Learn- ing joint representations of visual arts

    Hui Mao, Ming Cheung, and James She. Deepart: Learn- ing joint representations of visual arts. In Proceedings of 10 the 25th ACM International Conference on Multimedia, page 1183–1191, New York, NY , USA, 2017. Association for Computing Machinery. 6

    2017

  37. [45]

    Revisiting the calibration of modern neu- ral networks

    Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neu- ral networks. In Advances in Neural Information Processing Systems, pages 15682–15694. Curran Associates, Inc., 2021. 2

    2021

  38. [46]

    Confidence-aware learning for deep neural net- works

    Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. Confidence-aware learning for deep neural net- works. In Proceedings of the 37th International Conference on Machine Learning, pages 7034–7044. PMLR, 2020. 2

    2020

  39. [47]

    When does label smoothing help? In Advances in Neu- ral Information Processing Systems

    Rafael M ¨uller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? In Advances in Neu- ral Information Processing Systems. Curran Associates, Inc.,

  40. [48]

    Cooper, and Milos Hauskrecht

    Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities us- ing bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, page 2901–2907. AAAI Press, 2015. 3

    2015

  41. [49]

    Rankmixup: Ranking-based mixup training for net- work calibration

    Jongyoun Noh, Hyekang Park, Junghyup Lee, and Bumsub Ham. Rankmixup: Ranking-based mixup training for net- work calibration. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pages 1358– 1368, 2023. 2

    2023

  42. [50]

    Gpt-4 technical report

    Sandhini Agarwal OpenAI: Josh Achiam, Steven Adler and Lama Ahmad et al. Gpt-4 technical report. In ChatGPT4 Report, 2023. 2

    2023

  43. [51]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...

    2021

  44. [52]

    SummaR- eranker: A multi-task mixture-of-experts re-ranking frame- work for abstractive summarization

    Mathieu Ravaut, Shafiq Joty, and Nancy Chen. SummaR- eranker: A multi-task mixture-of-experts re-ranking frame- work for abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 4504–4...

    2022

  45. [53]

    Distributionally robust ensemble of lottery tickets towards calibrated sparse network training

    Hitesh Sapkota, Dingrong Wang, Zhiqiang Tao, and Qi Yu. Distributionally robust ensemble of lottery tickets towards calibrated sparse network training. In Advances in Neural Information Processing Systems, pages 62657–62681. Cur- ran Associates, Inc., 2023. 2

    2023

  46. [54]

    A-okvqa: A benchmark for visual question answering using world knowl- edge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowl- edge. In European Conference on Computer Vision, 2022. 6

    2022

  47. [55]

    REPLUG: Retrieval-augmented black-box language models

    Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen- tau Yih. REPLUG: Retrieval-augmented black-box language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computationa...

    2024

  48. [56]

    Videobert: A joint model for video and language representation learning

    Chen Sun, Austin Myers, Carl V ondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 2

    2019

  49. [57]

    Sq-llava: Self-questioning for large vision-language assistant

    Guohao Sun, Can Qin, Jiamian Wang, Zeyuan Chen, Ran Xu, and Zhiqiang Tao. Sq-llava: Self-questioning for large vision-language assistant. In ECCV, 2024. 2

    2024

  50. [58]

    Llama: Open and efficient foundation lan- guage models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aure- lien Rodriguez, Armand Joulin, Edouard Grave, and Guil- laume Lample. Llama: Open and efficient foundation la...

    2023 arXiv

  51. [59]

    A closer look at the robustness of contrastive language-image pre-training (clip)

    Weijie Tu, Weijian Deng, and Tom Gedeon. A closer look at the robustness of contrastive language-image pre-training (clip). In Advances in Neural Information Processing Sys- tems, pages 13678–13691. Curran Associates, Inc., 2023. 2

    2023

  52. [60]

    An empirical study into what matters for calibrating vision-language models

    Weijie Tu, Weijian Deng, Dylan Campbell, Stephen Gould, and Tom Gedeon. An empirical study into what matters for calibrating vision-language models. In Proceedings of the 41st International Conference on Machine Learning , pages 48791–48808. PMLR, 2024. 2

    2024

  53. [61]

    GIT: A generative image-to-text transformer for vision and language

    Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A generative image-to-text transformer for vision and language. Transactions on Machine Learning Research ,

  54. [62]

    Chatcad: Interactive computer-aided diag- nosis on medical image using large language models

    Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. Chatcad: Interactive computer-aided diag- nosis on medical image using large language models. arXiv preprint arXiv:2302.07257, 2023. 1

    2023 arXiv

  55. [63]

    Editable scene simulation for autonomous driving via collaborative llm-agents

    Yuxi Wei, Zi Wang, Yifan Lu, Chenxin Xu, Changxing Liu, Hao Zhao, Siheng Chen, and Yanfeng Wang. Editable scene simulation for autonomous driving via collaborative llm-agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15...

    2024

  56. [64]

    Google landmarks dataset v2 - a large-scale benchmark for instance-level recognition and retrieval

    Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2 - a large-scale benchmark for instance-level recognition and retrieval. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 6

    2020

  57. [65]

    Precedent-enhanced legal judgment prediction with LLM and domain-model collaboration

    Yiquan Wu, Siying Zhou, Yifei Liu, Weiming Lu, Xiaozhong Liu, Yating Zhang, Changlong Sun, Fei Wu, and Kun Kuang. Precedent-enhanced legal judgment prediction with LLM and domain-model collaboration. InThe 2023 Conference on Empirical Methods in Natural Language Processing , 2023. 1

    2023

  58. [66]

    Martindale, and Marine Carpuat

    Weijia Xu, Sweta Agrawal, Eleftheria Briakou, Marianna J. Martindale, and Marine Carpuat. Understanding and detect- ing hallucinations in neural machine translation via model introspection. Transactions of the Association for Computa- tional Linguistics, 11:546–564, 2023. 2 11

    2023

  59. [67]

    mplug-owl: Modularization empowers large language models with mul- timodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yi Zhou, Junyan Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qiang Qi, Ji Zhang, and Feiyan Huang. mplug-owl: Modularization empowers large language models with mul- timoda...

    2023 arXiv

  60. [68]

    Mul- timodal healthcare ai: Identifying and designing clinically relevant vision-language applications for radiology

    Nur Yildirim, Hannah Richardson (nee Murfet), Maria T Wetscherek, Junaid Bajwa, Joseph Jacob, Mark A Pin- nock, Stephen Harris, Daniel Coelho de Castro, Shruthi Bannur, Stephanie Hyland, Pratik Ghosh, Mercy Ran- jit, Kenza Bouzid, Anton Schwaighofer, Fernando P ´erez- Garc´ıa,...

  61. [69]

    Woodpecker: Hallucination correction for multi- modal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xingguo Sun, and En- hong Chen. Woodpecker: Hallucination correction for multi- modal large language models. ArXiv, abs/2310.16045, 2023. 1, 2, 7

    arXiv

  62. [70]

    Dauphin, and David Lopez-Paz

    Hongyi Zhang, Moustapha Ciss ´e, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion. In 6th International Conference on Learning Represen- tations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net,

    2018

  63. [71]

    What if the tv was off? examining counterfactual reasoning abilities of multi-modal language models

    Letian Zhang, Xiaotong Zhai, Zhongkai Zhao, Yongshuo Zong, Xin Wen, and Bingchen Zhao. What if the tv was off? examining counterfactual reasoning abilities of multi-modal language models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (C...

    2024

  64. [72]

    Siren’s song in the ai ocean: A survey on hal- lucination in large language models

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren’s song in the ai ocean: A survey on hal- lucination in large language models. ArXiv, abs/2309.01219,

    arXiv

  65. [73]

    Calibrating sequence likelihood improves conditional language genera- tion

    Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. Calibrating sequence likelihood improves conditional language genera- tion. In The Eleventh International Conference on Learning Representations, 2023. 3

    2023

  66. [74]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Process- in...

  67. [75]

    Learning deep features for scene recognition using places database

    Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Tor- ralba, and Aude Oliva. Learning deep features for scene recognition using places database. In Advances in Neu- ral Information Processing Systems. Curran Associates, Inc.,

  68. [76]

    Detecting hallucinated content in conditional neural sequence generation

    Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzm ´an, Luke Zettlemoyer, and Marjan Ghazvininejad. Detecting hallucinated content in conditional neural sequence generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 1...

    2021

  69. [77]

    Jpeg Compression

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In The Twelfth International Con- ference on Learning Representations, 2024. 1, 2, 3 12 ...

    2024

  70. [2023]

    Association for Computational Linguistics. 4

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.