Pith. sign in

REVIEW 3 major objections 4 minor 59 references

Reward-guided decoding lets users dial an MLLM's object precision against recall at inference time, cutting hallucinations about 70%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

MRGD guides MLLM decoding with a learned hallucination reward and a detector-based recall reward, allowing users to trade off object precision, recall, and test-time compute while reducing object hallucinations on CHAIR and AMBER.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A genuinely useful inference-time control method for MLLMs, but the headline comparisons need stronger evaluation hygiene before the claims are fully trusted. the 3 major comments →

arxiv 2508.11616 v1 pith:IRVHYJEG submitted 2025-08-15 cs.CV cs.AIcs.CLcs.LG

Controlling Multimodal LLMs via Reward-guided Decoding

Oscar Ma\~nas , Pierluca D'Oro , Koustuv Sinha , Adriana Romero-Soriano , Michal Drozdzal , Aishwarya Agrawal This is my paper
classification cs.CV cs.AIcs.CLcs.LG
keywords reward-guided decodingmultimodal LLMobject hallucinationvisual groundingprecision-recall trade-offtest-time computeimage captioningpreference reward model
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the behavior of a multimodal LLM can be steered at inference time by searching over candidate sentence completions with a weighted reward signal. The authors build two small reward models—one trained from preference data to reward captions that name only objects actually in the image (object precision), one assembled from an object detector and word embeddings to reward captions that name objects present (object recall)—and combine them with a single weight $w$. On the COCO object-hallucination benchmark, the precision-guided setting ($w=1$) cuts the instance-level hallucination rate from 15.05% to 4.53%, while reducing object recall by only about 6.5%; a middle setting trades a little precision for substantially higher recall. The same reward models transfer to newer, architecturally different MLLMs without retraining, and varying the search breadth $k$ and evaluation period $T$ trades visual grounding against test-time compute.

Core claim

The central discovery is that MLLM decoding can be made controllable on the fly through multimodal reward-guided decoding (MRGD), and that this controllability exposes an inherent precision/recall trade-off in the base models. MRGD samples $k$ candidate sentence-level completions from the current partial response and keeps the one maximizing $s(x_v,x_q,y)=w\,r_{\mathrm{hal}}(x_v,x_q,y)+(1-w)\,r_{\mathrm{rec}}(x_v,x_q,y)$, repeating until the end-of-sentence token. $r_{\mathrm{hal}}$ is a PaliGemma-based reward model trained with a Bradley-Terry preference loss plus regression targets on a mix of LLaVA-RLHF, RLHF-V, POVID, and repurposed SugarCrepe data; $r_{\mathrm{rec}}$ counts, via object

What carries the argument

The load-bearing mechanism is the weighted reward score $s = w\,r_{\mathrm{hal}} + (1-w)\,r_{\mathrm{rec}}$, used as the selection criterion in an iterative sentence-level search: every $T$ sentences, $k$ candidate continuations are sampled from the base MLLM and the one with the largest $s$ is appended to the context. The weight $w$ is the precision/recall dial; $k$ and $T$ are the compute/grounding dials. The search turns an off-the-shelf MLLM into an optimizer of a user-specified reward without changing its weights.

Load-bearing premise

The reward model trained on complete preferred/rejected caption pairs can also assign trustworthy scores to partial sentence prefixes, so selecting the best prefix every $T$ sentences actually improves the final caption.

What would settle it

Take a held-out set of images with ground-truth object annotations and a base MLLM not used in the paper; run MRGD with $w=1$ and increasing $k$, and compare the chosen partial responses' $r_{\mathrm{hal}}$ scores with the hallucination rate of the finished captions. If ordering partial candidates by $r_{\mathrm{hal}}$ does not predict which final caption has fewer hallucinated objects, the sentence-level guidance is not the cause of the reported reductions.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A user can set the operating point per query: precision-first ($w$ near 1), recall-first ($w$ near 0), or balanced, with no retraining.
  • The same reward models apply to base MLLMs they were not trained with, making MRGD a plug-in inference-time alignment layer on top of instruction-tuned and RLHF-tuned models.
  • Because evaluating partial responses every $T$ sentences is more sample-efficient than best-of-$k$ over full responses, modest test-time budgets can buy most of the grounding gain.
  • MRGD combines with prompting and fine-tuning rather than replacing them, so existing hallucination mitigations remain usable underneath.
  • The observed precision/recall curve indicates an inherent trade-off in current MLLMs; more compute (larger $k$) shifts the whole curve toward the ideal operating point.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if sentence-level reward scoring is reliable, the same search recipe should extend to other hallucination axes—attributes, counts, spatial relations—by training one reward model per axis and weighting them in the same $w$ dial.
  • Editorial inference: the method's ceiling is set by $r_{\mathrm{hal}}$'s ability to score partial prefixes; a reward model trained specifically on truncated responses, or a value head trained on prefixes, would likely reduce the gap the paper acknowledges in Section 3.2.
  • Editorial inference: since $r_{\mathrm{rec}}$ is assembled from off-the-shelf components, its detector's vocabulary and POS-tagger errors bound recall estimates; a more precise detector or a caption-aware reference set should sharpen the $w=0$ operating point.
  • Editorial inference: a testable extension is to run MRGD on discriminative hallucination tasks such as visual questioning, where the reward would have to judge answers rather than captions; the paper leaves this open in its limitations and future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces MRGD, a reward-guided decoding method for multimodal LLMs. Two reward models are built: rhal, a PaliGemma-based reward model trained with Bradley-Terry and MSE losses on preference data from LLaVA-RLHF, RLHF-V, POVID, and a repurposed SugarCrepe split, and rrec, composed from OWLv2, Sentence-BERT, and NLTK to estimate object recall. At inference, candidate sentence-level completions are sampled and selected by maximizing w·rhal + (1−w)·rrec, giving a precision/recall control knob (w) and a compute/grounding knob (k, T). Experiments on CHAIR and AMBER for LLaVA-1.5, Llama-3.2-Vision, and SmolVLM-2 report substantial hallucination reductions, e.g., CHAIR_i on COCO dropping from 15.05 to 4.53 at w=1, with controllability across w and improved performance over CGD and several fine-tuning baselines.

Significance. If the empirical claims hold, the method is a practically valuable, training-free-at-inference mechanism for adapting existing MLLMs: it provides two user-controllable trade-offs (precision vs. recall, and compute vs. grounding) and transfers reward models across three architecturally different base MLLMs without retraining. The paper is also careful in several respects: it evaluates rhal on held-out VLFeedback data, ablates the preference-data mix and reward-model components, reports latency, and includes qualitative examples. These strengths make the central idea worth pursuing. However, the headline comparative claims are currently supported by an insufficiently specified hyperparameter-selection procedure and by a potential train/evaluation image overlap, so the quantitative results need additional validation before they can be accepted as stated.

major comments (3)
  1. [§4.3, Table 1] The MRGD rows are reported as 'the best performing variant w.r.t. the number of samples k, the reward evaluation period T, and the temperature t,' but no validation procedure is described. The central comparison—e.g., the ~70% CHAIR_i reduction from 15.05 to 4.53—therefore conflates method performance with test-set hyperparameter selection. Since k and T are core controllability parameters, the authors should either report a single fixed configuration used for all rows (e.g., k=30, T=1, t=1.0, as in Table 2 and Figure 2) or provide a validation-based selection protocol and report the chosen values for every w and benchmark.
  2. [§3.1.1, §4.1, Tables 1–2] rhal is trained on 7.5k SugarCrepe examples, and SugarCrepe is constructed from COCO images; the evaluation benchmarks CHAIR and AMBER are also COCO-based. The paper does not report any overlap analysis or disjoint split between rhal's training images and the evaluation images. If overlapping images appear, rhal can memorize object–caption associations and inflate exactly the hallucination reductions used to support the main claim. This is a load-bearing concern, not a minor one: please quantify the overlap and rerun the main comparisons with any overlapping examples removed, or evaluate on a held-out COCO split not used in any rhal training set.
  3. [§3.2, Algorithm 1, Figure 2] rhal is trained on complete chosen/rejected responses, but MRGD with T=1 scores partial sentence-level completions. The paper acknowledges this issue in §3.2 but provides no empirical check that the reward model's ranking quality transfers to incomplete prefixes; the reported VLFeedback accuracy (67.68%) is measured on complete responses. Since the compute-efficiency claim (T=1 being more sample-efficient than rejection sampling) depends directly on this transfer, please add a partial-prefix validation — e.g., rank accuracy on truncated held-out responses as a function of sentence index — or compare T=1 against T=∞ at matched total sample counts.
minor comments (4)
  1. [§4.1, §4.7, Table 3] The naming of the preference datasets is inconsistent: the paper alternately uses 'RLAIF-V' and 'RLHF-V' (Refs. [50] and [49]). Please unify the names and ensure Table 3's 'RLAIF-V' rows are unambiguous.
  2. [§3.1.1, Eq. (2)] The notation in Eq. (2) drops the dependence on x_v and x_q that is explicit in Eq. (1). Also, the parenthetical in the MSE terms is slightly awkward; consider rewriting for clarity.
  3. [Appendix 6.4] The claim that 'CGD can be viewed as a particular instance of MRGD' is instructive, but CGD's SigLIP scores are normalized across candidates while rhal scores are trained to a [0,1] scale; the comparison would benefit from a one-sentence discussion of whether normalization affects the combined score in Eq. (4).
  4. [Figure 2] The left and right panels use different y-axis scales and the marker for T=∞ (rejection sampling) is not shown; adding explicit markers and a legend for T values would improve readability.

Circularity Check

0 steps flagged

No significant circularity: MRGD's reward models are trained/composed from external components and benchmark gains are measured, not constructed.

full rationale

The paper's derivation chain is not circular. The central decoding score is s(x_v,x_q,y) = w·r_hal(x_v,x_q,y) + (1−w)·r_rec(x_v,x_q,y) (Eq. 4), and decoding selects the completion maximizing this score among sampled candidates. r_hal is trained with the Bradley-Terry plus MSE objective (Eqs. 1–2) on external preference data (LLaVA-RLHF, RLHF-V, POVID, repurposed SugarCrepe), not on CHAIR/AMBER or on the benchmark metrics. r_rec (Eq. 3) is composed from off-the-shelf OWLv2, Sentence-BERT, and NLTK with a fixed threshold τ=0.5; no parameter is fitted to the evaluation benchmarks. The headline reductions in CHAIR_i, CHAIR_s, Hal., and gains in Rec./Cov. are empirically measured outcomes of this decoding procedure, not identities that follow from the definition of the rewards. The ablations (Table 3) further show the method is robust to changing the backbone, detector, threshold, and data mix, which would not be expected if the result were forced by construction. There is no load-bearing self-citation: the only self-reference (Bordes et al., ref. [5], co-authored by O. Mañas) is a general survey citation in the introduction and is not used to justify any methodological choice, uniqueness claim, or ansatz. The skeptical concern about possible overlap between SugarCrepe training images and COCO-based CHAIR/AMBER evaluation is a legitimate data-contamination / validity risk, but it is not a circularity in the derivation; no equation reduces to its own inputs, and no fitted parameter is relabeled as a prediction. Accordingly, the circularity score is 0.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The ledger contains no invented physical or architectural entities. What the central claim rests on is the quality and transferability of two reward models (one trained on public preference data with LoRA, one assembled from off-the-shelf modules), plus explicit hyperparameters w, k, T, t, and tau. The most fragile entries are the partial-response assumption and the detector/tagger recall proxy.

free parameters (5)
  • guidance weight w = user-chosen in [0,1]; w* about 0.25 for COCO, 1.0 for AMBER
    Linear combination weight between rhal and rrec in Eq. 4; the paper reports results across w and identifies benchmark-specific optimal w.
  • search breadth k = {3, 5, 10, 30}; k=30 for main tables
    Number of candidate continuations sampled per iteration; trades compute against grounding quality (Algorithm 1).
  • evaluation period T = 1 (main), 2, 3
    Sentence block length before reward evaluation; T=infinity reduces to best-of-k rejection sampling.
  • similarity threshold tau = 0.5; ablated to 0.2 and 0.9
    Threshold in Eq. 3 for declaring a predicted object a true positive in rrec.
  • sampling temperature t = 1.0 for LLaVA-1.5, 0.2 for Llama-3.2-Vision and SmolVLM-2
    Temperature for candidate sampling from the base MLLM; affects diversity and search quality.
axioms (6)
  • domain assumption Preference pairs (y+, y-) in LLaVA-RLHF, RLHF-V, POVID, and repurposed SugarCrepe faithfully identify less hallucinated captions.
    Section 3.1.1 trains rhal with Bradley-Terry loss on this mixture; if chosen/rejected labels are noisy, rhal scores are noisy.
  • standard math Bradley-Terry model of preferences is an appropriate objective for a scalar hallucination reward.
    Eq. 1 uses the standard Bradley-Terry pairwise loss, a standard but substantive assumption about preference structure.
  • domain assumption OWLv2 detections form an adequate reference object set for recall estimation.
    Section 3.1.2 uses OWLv2 to extract Oref; reported detector precision is 63.16% and recall 55.83% on COCO, so missing or spurious detections propagate into rrec.
  • domain assumption POS tagging plus Sentence-BERT similarity with threshold tau=0.5 identifies object mentions in captions.
    Section 3.1.2 Eq. 3; the authors report POS tagger precision 67.04% and recall 54.54% on COCO captions.
  • ad hoc to paper Reward models trained on complete responses provide reliable scores for sentence-level partial responses when evaluated every T sentences.
    Section 3.2 explicitly acknowledges the partial-response scoring risk and introduces T to mitigate it; MRGD's success depends on this transfer.
  • domain assumption Sampling from the base MLLM at temperature t produces candidate completions among which higher-reward completions exist.
    Algorithm 1 selects from k samples; if the base model distribution lacks good completions, search cannot recover them.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Controlling Multimodal LLMs via Reward-guided Decoding." pith.science (2026). https://pith.science/paper/IRVHYJEG

@misc{pith2026250811616,
  author       = {Pith},
  title        = {Pith review of: Controlling Multimodal LLMs via Reward-guided Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IRVHYJEG}},
  note         = {Machine review of arXiv:2508.11616}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

As Multimodal Large Language Models (MLLMs) gain widespread applicability, it is becoming increasingly desirable to adapt them for diverse user needs. In this paper, we study the adaptation of MLLMs through controlled decoding. To achieve this, we introduce the first method for reward-guided decoding of MLLMs and demonstrate its application in improving their visual grounding. Our method involves building reward models for visual grounding and using them to guide the MLLM's decoding process. Concretely, we build two separate reward models to independently control the degree of object precision and recall in the model's output. Our approach enables on-the-fly controllability of an MLLM's inference process in two ways: first, by giving control over the relative importance of each reward function during decoding, allowing a user to dynamically trade off object precision for recall in image captioning tasks; second, by giving control over the breadth of the search during decoding, allowing the user to control the trade-off between the amount of test-time compute and the degree of visual grounding. We evaluate our method on standard object hallucination benchmarks, showing that it provides significant controllability over MLLM inference, while consistently outperforming existing hallucination mitigation methods.

Figures

Figures reproduced from arXiv: 2508.11616 by Adriana Romero-Soriano, Aishwarya Agrawal, Koustuv Sinha, Michal Drozdzal, Oscar Ma\~nas, Pierluca D'Oro.

Figure 1
Figure 1. Figure 1: Illustration of multimodal reward-guided decoding (MRGD) for MLLMs. At each iteration, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: LLaVA-1.5 on COCO. Leveraging the reward model to guide the generation more often (lower [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Object precision and recall for LLaVA-1.5 on COCO, [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Selected qualitative examples for LLaVA-1.5 using the default greedy decoding and our proposed MRGD strategy with different [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 31 canonical work pages · 1 internal anchor

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    Pith/arXiv arXiv

  2. [2]

    Understand- ing alignment in multimodal llms: A comprehensive study

    Elmira Amirloo, Jean-Philippe Fauconnier, Christoph Roes- mann, Christian Kerl, Rinu Boney, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, et al. Understand- ing alignment in multimodal llms: A comprehensive study. arXiv preprint arXiv:2407.02477, 2024. 2

    Pith/arXiv arXiv 2024

  3. [3]

    Hallucination of multimodal large language models: A survey

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930, 2024. 1 9

    Pith/arXiv arXiv 2024

  4. [4]

    Paligemma: A versatile 3b vlm for trans- fer

    Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for trans- fer. arXiv preprint arXiv:2407.07726, 2024. 3

    Pith/arXiv arXiv 2024

  5. [5]

    An introduction to vision-language modeling

    Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C Li, Adrien Bardes, Suzanne Petryk, Oscar Ma˜nas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, et al. An introduction to vision-language modeling. arXiv preprint arXiv:2405.17247, 2024. 1

    Pith/arXiv arXiv 2024

  6. [6]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired compar- isons. Biometrika, 39(3/4):324–345, 1952. 3

    work page 1952

  7. [7]

    Large language monkeys: Scaling inference compute with repeated sampling

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R ´e, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024. 2, 4

    Pith/arXiv arXiv 2024

  8. [8]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 1

    Pith/arXiv arXiv 2005

  9. [9]

    The (r) evolution of multi- modal large language models: A survey

    Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. The (r) evolution of multi- modal large language models: A survey. arXiv preprint arXiv:2402.12451, 2024. 1

    arXiv 2024

  10. [10]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In European confer- ence on computer vision, pages 213–229. Springer, 2020. 8

    work page 2020

  11. [11]

    Plug and play language models: A simple approach to controlled text generation

    Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations, 2020. 2

    work page 2020

  12. [12]

    Seeing is believing: Mitigating hallucination in large vision- language models via clip-guided decoding

    Ailin Deng, Zhirui Chen, and Bryan Hooi. Seeing is believing: Mitigating hallucination in large vision- language models via clip-guided decoding. arXiv preprint arXiv:2402.15300, 2024. 2, 5, 1

    Pith/arXiv arXiv 2024

  13. [13]

    Reward-augmented decod- ing: Efficient controlled text generation with a unidirectional reward model

    Haikang Deng and Colin Raffel. Reward-augmented decod- ing: Efficient controlled text generation with a unidirectional reward model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 11781–11791, 2023. 1, 2

    work page 2023

  14. [14]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    Pith/arXiv arXiv

  15. [15]

    Multi-modal hal- lucination control by visual information grounding

    Alessandro Favero, Luca Zancato, Matthew Trager, Sid- dharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hal- lucination control by visual information grounding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14303–14312, 2024. 2

    work page 2024

  16. [16]

    Value augmented sampling for language model alignment and personalization

    Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, and Pulkit Agrawal. Value augmented sampling for language model alignment and personalization. arXiv preprint arXiv:2405.06639, 2024. 2

    Pith/arXiv arXiv 2024

  17. [17]

    Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kem- bhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. Advances in neural information processing systems, 36, 2024. 4

    work page 2024

  18. [18]

    Lora: Low- rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low- rank adaptation of large language models. In International Conference on Learning Representations, 2022. 4

    work page 2022

  19. [19]

    Args: Alignment as reward-guided search

    Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search. In The Twelfth Inter- national Conference on Learning Representations, 2024. 1, 2

    work page 2024

  20. [20]

    Rewardbench: Evaluating reward models for language modeling

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024. 5

    Pith/arXiv arXiv 2024

  21. [21]

    Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 13872–13882, 2024. 2, 5, 6, 1

    work page 2024

  22. [22]

    Sequential monte carlo steering of large lan- guage models using probabilistic programs

    Alexander K Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash Mansinghka. Sequential monte carlo steering of large lan- guage models using probabilistic programs. In ICML 2023 Workshop: Sampling and Optimization in Discrete Space ,

    work page 2023

  23. [23]

    Cascade reward sampling for efficient decoding-time align- ment

    Bolian Li, Yifan Wang, Ananth Grama, and Ruqi Zhang. Cascade reward sampling for efficient decoding-time align- ment. In ICML 2024 Next Generation of AI Safety Workshop,

    work page 2024

  24. [24]

    Vlfeedback: A large-scale ai feedback dataset for large vision-language models alignment

    Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, Lingpeng Kong, and Qi Liu. Vlfeedback: A large-scale ai feedback dataset for large vision-language models alignment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 6227–6246, 2024. 4

    work page 2024

  25. [25]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pages 292–305, 2023. 9

    work page 2023

  26. [26]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representa- tions, 2024. 1, 2

    work page 2024

  27. [27]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 4, 5

    work page 2024

  28. [28]

    Don’t throw away your value model! generating more preferable text with value-guided monte-carlo tree search decoding

    Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Don’t throw away your value model! generating more preferable text with value-guided monte-carlo tree search decoding. In First Conference on Language Modeling, 2024. 2 10

    work page 2024

  29. [29]

    Investigating and mitigating object hallucinations in pretrained vision-language (clip) models

    Yufang Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, and Aimin Zhou. Investigating and mitigating object hallucinations in pretrained vision-language (clip) models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 18288–18301, 2024. 2, 5

    work page 2024

  30. [30]

    Smolvlm: Redefining small and efficient multimodal models

    Andr ´es Marafioti, Orr Zohar, Miquel Farr ´e, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tun- stall, Leandro von Werra, and Thomas Wolf. Smolvlm: Redefining small and efficient multimodal models. arXiv preprint arXiv:2504.0529...

    Pith/arXiv arXiv 2025

  31. [31]

    Scaling open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. Advances in Neu- ral Information Processing Systems, 36, 2024. 4

    work page 2024

  32. [32]

    Controlled decod- ing from language models

    Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decod- ing from language models. In Forty-first International Con- ference on Machine Learning, 2024. 1, 2

    work page 2024

  33. [33]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Ad- vances in neural information processing systems, 35:27730– 27744, 2022. 3

    work page 2022

  34. [34]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748–8763. PMLR, 2021. 3

    2021

  35. [35]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. 6

    work page 2024

  36. [36]

    A critical look at to- kenwise reward-guided text generation

    Ahmad Rashid, Ruotian Wu, Julia Grosse, Agustinus Kristiadi, and Pascal Poupart. A critical look at to- kenwise reward-guided text generation. arXiv preprint arXiv:2406.07780, 2024. 2

    work page arXiv 2024

  37. [37]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Linguis- tics, 2019. 4

    work page 2019

  38. [38]

    Object hallucination in image cap- tioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035–4045,

    work page 2018

  39. [39]

    Mitigating object hallucination via data augmented contrastive tuning

    Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan ¨O Arık, and Tomas Pfister. Mitigating object hallucination via data augmented contrastive tuning. arXiv preprint arXiv:2405.18654, 2024. 2, 5, 1

    Pith/arXiv arXiv 2024

  40. [40]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Ku- mar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024. 1, 2, 4

    Pith/arXiv arXiv 2024

  41. [41]

    Paligemma 2: A family of versatile vlms for transfer

    Andreas Steiner, Andr ´e Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Grit- senko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, et al. Paligemma 2: A family of versatile vlms for transfer. arXiv preprint arXiv:2412.03555, 2024. 8

    Pith/arXiv arXiv 2024

  42. [42]

    Aligning large multi- modal models with factually augmented rlhf

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multi- modal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023. 1, 2, 4, 5, 7

    Pith/arXiv arXiv 2023

  43. [43]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1

    Pith/arXiv arXiv 2023

  44. [44]

    Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training

    David Wan, Jaemin Cho, Elias Stengel-Eskin, and Mohit Bansal. Contrastive region guidance: Improving grounding in vision-language models without training. arXiv preprint arXiv:2403.02325, 2024. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  45. [45]

    mdpo: Conditional preference optimization for multimodal large language mod- els

    Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mdpo: Conditional preference optimization for multimodal large language mod- els. arXiv preprint arXiv:2406.11839, 2024. 2, 5

    Pith/arXiv arXiv 2024

  46. [46]

    An llm-free multi-dimensional benchmark for mllms hallu- cination evaluation

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An llm-free multi-dimensional benchmark for mllms hallu- cination evaluation. arXiv preprint arXiv:2311.07397, 2023. 4, 1

    Pith/arXiv arXiv 2023

  47. [47]

    Fudge: Controlled text genera- tion with future discriminators

    Kevin Yang and Dan Klein. Fudge: Controlled text genera- tion with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Tech- nologies, pages 3511–3535, 2021. 2

    work page 2021

  48. [48]

    Woodpecker: Hallucination correction for multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310.16045 ,

    Pith/arXiv arXiv

  49. [49]

    Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional hu- man feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional hu- man feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13807– 13816, 2024. 1, 2, 4, 7

    work page 2024

  50. [50]

    Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness

    Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220, 2024. 2, 7

    arXiv 2024

  51. [51]

    Less is more: Mitigat- ing multimodal hallucination from an eos decision perspec- tive

    Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mitigat- ing multimodal hallucination from an eos decision perspec- tive. arXiv preprint arXiv:2402.14545, 2024. 5, 1

    Pith/arXiv arXiv 2024

  52. [52]

    Self- correcting decoding with generative feedback for mitigat- ing hallucinations in large vision-language models

    Ce Zhang, Zifu Wan, Zhehan Kan, Martin Q Ma, Si- mon Stepputtis, Deva Ramanan, Russ Salakhutdinov, Louis- 11 Philippe Morency, Katia Sycara, and Yaqi Xie. Self- correcting decoding with generative feedback for mitigat- ing hallucinations in large vision-language models. arXiv preprint arXiv:2502.06130, 2025. 2

    Pith/arXiv arXiv 2025

  53. [53]

    Multimodal chain-of-thought reasoning in language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, George Karypis, Alex Smola, et al. Multimodal chain-of-thought reasoning in language models. Transactions on Machine Learning Re- search, 2024. 1, 2

    work page 2024

  54. [54]

    Mitigating object hallucination in large vision- language models via classifier-free guidance

    Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu. Mitigating object hallucination in large vision- language models via classifier-free guidance. arXiv preprint arXiv:2402.08680, 2024. 2

    Pith/arXiv arXiv 2024

  55. [55]

    Beyond hallucinations: Enhanc- ing lvlms through hallucination-aware direct preference op- timization

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Ji- aqi Wang, and Conghui He. Beyond hallucinations: Enhanc- ing lvlms through hallucination-aware direct preference op- timization. arXiv preprint arXiv:2311.16839 , 2023. 2, 5, 1

    Pith/arXiv arXiv 2023

  56. [56]

    Aligning modalities in vision large lan- guage models via preference fine-tuning

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large lan- guage models via preference fine-tuning. In ICLR 2024 Workshop on Reliable and Responsible Foundation Models,

    work page 2024

  57. [57]

    Analyzing and mitigating object hallucination in large vision-language models

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In The Twelfth International Con- ference on Learning Representations, 2024. 2

    work page 2024

  58. [58]

    Calibrated self-rewarding vision language models

    Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405.14622 , 2024. 2, 5 12 Controlling Multimodal LLMs via Reward-guided Decoding Supplementary Material

    Pith/arXiv arXiv 2024

  59. [59]

    Describe this image in detail

    Experiments 6.1. Details on evaluation metrics We evaluate object precision and recall with standard metrics from the corresponding benchmarks, defined as follows. CHAIRi (Ci) [38], CHAIR [46]. Measure the frac- tion of hallucinated objects in the generated captions. Ci/CHAIR = |{hallucinated objects}| |{all mentioned objects}| CHAIRs (Cs) [38], Hal. [46]...

    work page

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.