Pith. sign in

REVIEW 4 major objections 5 minor 71 references

VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 7B vision-language verifier, trained with object-detector checks, critique-style reasoning, and iterative margin-based rejection sampling, beats 90B and 72B open rivals on VLRewardBench for hallucination detection and general QA.

desk verdict A plausible and useful recipe for compact VL reward models, but the undefined margin score and inconsistent tables keep the main claim from being fully supported. read the letter →

arxiv 2506.13888 v1 pith:CA7RJKZU submitted 2025-06-16 cs.CL cs.CV

Jipeng Zhang , Kehao Miao , Renjie Pi , Zhaowei Wang , Runtao Liu , Rui Pan , Tong Zhang This is my paper
classification cs.CLcs.CV
keywords vision-languagerewardmodelhallucinationdetectiongenerativechain-of-thoughtrationaleiterativetrainingmargin-basedrejectionsamplingvisionexpertverificationbest-of-Nselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VL-GenRM tackles two problems that block reward-model training for vision-language models: the bootstrapping loop in which high-quality training data already requires a strong model, and the amplification of hallucinated negative examples that carry text-image bias into the verifier. The paper's recipe has three parts: object detectors certify which objects actually appear in an image so hallucinated mentions can be filtered and diverse negative responses fabricated; the verifier is trained to emit critique-style chain-of-thought rationales before its Yes/No verdict; and iterative fine-tuning selects the most informative rationale pairs by the margin between positive and negative reward scores. On VLRewardBench the resulting 7B model scores 72.3 overall and 82.4 on hallucination detection, above GPT-4o (65.8) and Gemini-1.5-Pro (67.2) and far above open models like LLaMA-3.2-90B, though it trails on abstract reasoning. The paper argues this makes compact, verifiable reward signals for multimodal reinforcement learning practical without heavy human annotation.

What carries the argument

The load-bearing object is the generative vision-language reward model (GenRM): a VLM that scores a response by the probability it assigns to the token 'Yes' when asked whether the answer is correct, $r_{\mathrm{GenRM}}(x,y,I)=p_\theta(\text{'Yes'}|x,y,I,p)$. Around it the paper builds three mechanisms. (1) Vision-expert filtering: an open-vocabulary object detector supplies the object list $\mathcal{O}^*(I)$ of the image; a negative response mentioning an object absent from that list is labeled a hallucination, and two sampled objects in a correct response are swapped to fabricate diverse negatives. (2) Critique-style CoT: the model is trained to output a rationale that names only crucial, obvious errors before emitting Yes/No, which the ablation shows beats both plain pairwise data and descriptive CoT. (3) Margin-based rejection sampling: in each iteration the model generates rationales for positive and negative responses, scores them by the Yes-token probability, and keeps only pairs whose margin $m(c_+^i,c_-^i)=\mathrm{score}(c_+^i)-\mathrm{score}(c_-^i)$ falls in a window $[\lambda_l,\lambda_r]$ (e.g. $[0.3,1]$), which are then used for a LoRA (low-rank adaptation) fine-tuning round.

What would settle it

Run the paper's vision-expert filter (Section 4.1.2) on images containing objects that open-vocabulary detectors systematically miss—small, occluded, or rare objects—with responses that correctly name those objects: if a large share is labeled 'hallucination,' the ground-truth object list is incomplete and the preference data and CoT rationales built on it are corrupted. The quantitative version: take a human-annotated sample of the training set, measure the precision of the filter's hallucination label, and check whether precision collapses on images with many undetected objects; the paper's claim predicts precision stays high there, and a drop should track a visible decline in the trained model's hallucination score.

Watch

Extended reading notes

Core claim

The discovery, stated on the paper's own terms, is that a 7B generative vision-language reward model trained with this three-part recipe reaches state-of-the-art performance among open-source VLMs on general QA and hallucination detection, outperforming much larger models like LLaMA-3.2 (90B) and Molmo (72B): on VLRewardBench it records an overall 72.3 (hallucination 82.4, general 54.6, reasoning 58.8) against 65.8 for GPT-4o and 67.2 for Gemini-1.5-Pro. The mechanism the authors identify is that external visual supervision breaks the self-referential bootstrapping loop: an object detector supplies a ground-truth object list, a weak model's responses are filtered for objects absent from that list, and the corrections become contrastive preference pairs with critique-style CoT rationales. Iteration then lets the model refine its own rationales, keeping only the pairs whose reward margin falls inside a chosen window and fine-tuning with LoRA; the ablations show critique CoT beats descriptive CoT and plain pairs, verified pairs beat raw pairs, and iteration adds hallucination detection on top of the reward IFT stage. The paper also states its boundary: the model lags on abstract reasoning, which the authors attribute to the object-detection-centric vision module being less suited to fine-grained abstract inference.

Load-bearing premise

The load-bearing premise is that the object detector's list is complete enough to serve as ground truth, so that any object a response mentions but the detector does not list counts as a hallucination—if the detector misses real objects, correct responses are mislabeled and the preference data and CoT rationales built on that label are corrupted.

Editorial extensions

If this is right

  • A 7B verifier trained this way can replace much larger judges: on VLRewardBench it scores 72.3 overall and 82.4 on hallucination detection, above GPT-4o (65.8 overall) and Gemini-1.5-Pro (67.2), and far above LLaMA-3.2-90B (56.2) and Molmo-72B (44.1).
  • The trained reward model transfers across base generators: used for Best-of-N selection on LLaVA-Wild, it improves Qwen2.5-VL-7B, InternVL2.5-VL-4B, and LLaVA-Next-8B over BT-RM and Direct GenRM baselines, and the gain grows when reward IFT and iterative refinement are added.
  • Iteration is the engine of the gain on hallucination: Reward IFT alone raises reasoning (50.6 to 62.3), Iteration 1 raises hallucination detection (78.5 to 82.1), and a second iteration yields marginal additional gains, suggesting saturation at this model scale.
  • Data construction that injects external visual knowledge matters more than raw pair volume: adding verified pairs improves hallucination detection, while descriptive CoT rationales actually hurt performance (57.0 vs 70.9 for critique CoT).
  • The framework is model-agnostic: results hold on both Qwen-VL-7B and InternVL-4B bases, so the recipe scales to other vision-language backbones without architectural changes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the margin window $[\lambda_l,\lambda_r]$ keeps rationale pairs the model is genuinely uncertain about—neither trivially correct nor wildly wrong—so the iteration looks like a curriculum on the model's own error surface; a testable prediction is that the optimal window drifts toward smaller margins as later iterations become more accurate.
  • Editorial inference: because the detector-based labels target object presence specifically, the hallucination gains should transfer most strongly to object-level errors in generation (for example, when the reward signal is used for RL fine-tuning of the generator), while attribute and counting errors that the filter does not check should benefit less; splitting a hallucination benchmark by error t
  • Editorial inference: the reasoning shortfall (58.8 vs 70.5 for GPT-4o) suggests the recipe biases the verifier toward perceptually checkable content; combining this verifier with a text-only reasoning verifier, or adding process-level rationale checks, is the natural next step the paper does not explore.
  • Editorial inference: the pipeline's ceiling is set by detector recall, so the filter's precision on a human-labeled sample of its own training data is the metric that would expose systematic mislabeling; an ensemble of open-vocabulary detectors, or detector-aware thresholds, is a plausible cheap improvement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes VL-GenRM, a training recipe for vision-language reward models (also called vision-language verifiers). The method has three main components: (1) automated construction of preference data with vision-expert filtering, where an object detector checks whether objects mentioned in a response actually appear in the image; (2) chain-of-thought (CoT) critique rationales generated by a strong VLM and used in an instruction-following fine-tuning (IFT) stage; and (3) an iterative stage that selects self-generated rationale pairs by margin-based rejection sampling and continues LoRA fine-tuning. The model is evaluated on VLRewardBench subtasks (General, Hallucination, Reasoning, Overall) and as a Best-of-N verifier on LLaVA-Wild. The central claim is that a compact 7B VL-GenRM achieves state-of-the-art performance among open-source VLMs on general QA and hallucination robustness, outperforming much larger models such as LLaMA-3.2 (90B) and Molmo (72B).

Significance. If the reported results are reproducible, this is a practically useful contribution: it offers an automated data-pipeline plus a training recipe for a small, open-source vision-language reward model, with releases of code and data-generation details. The paper explicitly targets the real problem that negative examples for VL reward training are often hallucinated or biased, and the idea of injecting vision-expert verification and critique-based CoT rationales is sensible. The Best-of-N evaluation across three base models is a good sanity check of model-agnosticism, and the contamination analysis, while weak, shows awareness of a standard concern. However, the significance of the central claim is currently limited by an undefined score function in the iterative selection mechanism, conflicting tables of the headline result, and an internal inconsistency about whether the object-detector list is assumed complete. These issues must be resolved before the claims can be accepted.

major comments (4)
  1. [Section 5.2, Eq. (11)] The function score(·) in m(c_i^+, c_i^-) = score(c_i^+) - score(c_i^-) is never defined, either in the main text, Appendix B.2, or the prompt templates. The appendix only provides the threshold ranges ([0.3, 1] for iteration 1 and [0.3, 0.99] for iteration 2). Because this margin is the selection criterion that decides which self-generated rationale pairs enter D_iter, the reader cannot tell whether the additive gains in Table 5 come from the advertised margin-based rejection sampling or merely from LoRA fine-tuning on additional samples. Figure 4's caption introduces a different mechanism, saying candidate rationales are 'verified against reference outputs,' but Section 5.2 contains no such reference verification. Please define score(·) precisely (for example, as the model's Yes/No token likelihood or as an external verifier score), and if it is the model's own likelihood, discuss why this is not a self-rewarding loop with no new external supervision.
  2. [Tables 1 and 6; Sections 7.1 and 7.2] The same model, VL-GenRM (7B), is reported with conflicting numbers on VLRewardBench: Table 1 gives General 54.6, Hallucination 82.4, Reasoning 58.8, Overall 72.3, while Appendix D Table 6 gives General 51.9, Hallucination 81.4, Reasoning 54.4, Overall 70.2. These discrepancies change the comparison against LLaMA-3.2 (90B) and Molmo (72B), so the headline claim is not currently supported by a single consistent set of experimental numbers. In addition, Section 7.1 says 'As shown in Table 6' when presenting Table 1, and Section 7.2 repeats nearly the same paragraph with a different table reference; this suggests the manuscript was assembled from overlapping drafts and needs a careful pass.
  3. [Section 4.1.2; Appendix F Tables 10 and 12] The vision-expert filtering rule labels a generated negative response as a hallucination whenever an object mentioned in the response is absent from the detector's output, i.e., O(Y^-) ⊄ O*(I) leads to retention as a hallucinated negative. This treats the object detector's output as a sufficiently complete list of ground-truth objects. That assumption is contradicted within the paper itself: Table 10 tells the false-rejection judge that the detected-object list 'may not include all objects in the image,' and Table 12 instructs the critique generator not to rely solely on the list. If the detector misses an object, a correct response mentioning that object is mislabeled as a hallucination, and the corrupted label propagates into D_pair and into the CoT rationales generated in Section 4.2. Please either estimate detector recall on the data distribution or add a verification step that does not assume completeness.
  4. [Table 5; Section 5.2] The iterative gains in Table 5 are not monotonic and the stated mechanism is not isolated. Reward IFT gives General 54.6, Hallucination 78.5, Reasoning 62.3, Overall 70.9; Iteration 1 increases Hallucination to 82.1 but decreases Reasoning to 58.8; Iteration 2 decreases General from 55.2 to 53.0. The text describes Iteration 2 as showing 'marginal gains,' which is consistent with the table, but because score(·) is undefined and no external reference signal is described in the iterative loop, these fluctuations cannot be attributed to margin-based rejection sampling rather than to the additional 5K training samples and LoRA fine-tuning. A control that trains on 5K additional unfiltered rationale pairs, or an ablation that removes the margin filter, is needed to support the claim that the iterative refinement mechanism itself is responsible for the improvements.
minor comments (5)
  1. [Section 6.3] There is a typo in 'Claudge-3.5-Sonnet'; the correct name is Claude-3.5-Sonnet, and the cited reference [Bai et al., 2022] is not the Claude model paper.
  2. [Figure 4 caption] The caption says 'OpenAI's OI model detects correctness and generates refined annotations,' but Appendix B states that Qwen2.5-32B-Instruct and Qwen2-VL-72B-Instruct are used for these steps; please clarify which model is actually used.
  3. [Section 7.2] Section 7.2 duplicates the first paragraph of Section 7.1 almost verbatim, including the reference to Table 6 when the surrounding text refers to a different table; this duplicate paragraph should be removed or replaced with a genuine comparison of training methods.
  4. [Appendix C] The contamination analysis reports a cosine-similarity distribution with average scores around 0.33–0.37 but provides no statistical test, threshold, or exact-duplicate check against VLRewardBench; the conclusion that improvement is 'not merely due to including data from the same distribution' is not yet established by this analysis.
  5. [Table 3 and Section 7.3] The text says Table 3 evaluates VL-GenRM under Best-of-N accuracy, but the table columns are labeled BT, GenRM, IFT, and Iteration, and the caption refers to 'performance improvement brought by the proposed training pipeline'; the table and its caption should be aligned so the reader can see which numbers are BoN accuracies and which are deltas.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: iterative margin-based rejection sampling is self-referential and underspecified, but the headline SOTA claim rests on external benchmarks.

  1. other [Contributions (Section 1, 3rd bullet), Section 5.2 Eqs. (10)-(11), Appendix B.2]
    "Iterative Bootstrapping with Margin-based Rejection Sampling. We refine VL-GenRM's reasoning through iterative fine-tuning on successful rationales, which are selected through the margin between reward signals of positive and negative examples. The model simultaneously generates the reasoning rationales for both positive and negative response: c+ = f_RM(I, X, Y+), c− = f_RM(I, X, Y−). m(c+i, c−i) = score(c+i) − score(c−i). During iterative training, iteration 1 uses a margin of [0.3, 1], while iteration 2 uses a margin of [0.3, 0.99]."

    Eq. (10) generates the iterative rationales with f_RM, the model being trained, and Eq. (11) filters them by score(·), which the paper never defines. The only reward score defined for the model is r_θ(I,x,y)=f_θ(I,x,y) (Eq. 1), and the contribution text says the margin is between 'reward signals of positive and negative examples,' so the filter is, under the paper's own definitions, the model's current reward estimate. The selected D_iter is then used for LoRA IFT on the same model. Hence 'successful rationales' are those the model already scores as positive/negative, and re-training on them is a self-referential loop that cannot by itself inject the external supervision the paper's bootstrapping discussion says is needed.

full rationale

The paper's main claim—that VL-GenRM (7B) is state-of-the-art among open-source VLMs on general QA and hallucination robustness—is tested on external benchmarks (VLRewardBench and LLaVA-Wild Best-of-N), not on its own training distribution, and the initial data pipeline uses external supervision: GroundingDINO/Detectron2 object detection, Qwen2.5-32B for replacement responses, and Qwen2-VL-72B for CoT critiques (Appendix B). Thus the central empirical result is not circular. The one genuinely self-referential component is the iterative stage: rationales are generated by the model itself (Eq. 10) and filtered by score(·) (Eq. 11), which is undefined but is described as the 'margin between reward signals'—i.e., the model's own reward. Training on that self-filtered set creates a self-consistency loop rather than a new external signal, and the Figure 4 caption's 'verified against reference outputs' is not operationalized in Section 5.2. This weakens the attribution of the Iteration 1/2 gains in Table 5 to the advertised mechanism. Separately, the object-detector-as-ground-truth assumption is acknowledged in the prompt ('may not include all objects') and is a data-quality/validity risk, not a circular reduction. No load-bearing self-citation chain or imported uniqueness theorem appears; citations to RAFT/STaR/GenRM supply background and components. Overall circularity score 4: one mechanism reduces to a self-referential loop, while the headline result retains independent external grounding.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical entities or parameters beyond training hyperparameters. The key assumptions are the reliability of the object detector and teacher VLMs, and the usefulness of the model's own margin scores for selecting rationales.

free parameters (2)
  • margin threshold range (lambda_l, lambda_r) = [0.3, 1] for iteration 1; [0.3, 0.99] for iteration 2
    Chosen by hand to filter self-generated rationale pairs; values change between iterations, with no theoretical or empirical justification beyond performance tuning.
  • IFT mixture weight lambda = not specified
    Appears in Eq. (8) as the weight balancing verification loss and correct-solution loss, but its value is not reported anywhere in the paper.
assumptions (3)
  • domain assumption Object detector outputs (GroundingDINO, Detectron2) are sufficiently complete to treat absence of a mentioned object as a hallucination.
    Used in Section 4.1.2 to label negative responses and in Section 4.2 to guide CoT generation. If detection is incomplete, correct responses can be mislabeled, corrupting the training data.
  • domain assumption Teacher VLMs (Qwen2-VL-72B-Instruct, Qwen2.5-32B-Instruct) provide reliable correctness labels and rationales.
    Used in Sections 4.1.3 and 4.2 to judge 'approximately correct' responses and to generate CoT critiques. Errors in these teachers propagate to the reward model's training signal.
  • ad hoc to paper The margin between the model's own scores for positive and negative rationales correlates with rationale quality and training utility.
    Margin-based rejection sampling in Section 5.2 keeps only pairs whose margin falls in a hand-chosen range. No evidence is provided for this correlation, and the score itself is not defined.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training." pith.science (2026). https://pith.science/paper/CA7RJKZU

@misc{pith2026250613888,
  author       = {Pith},
  title        = {Pith review of: VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CA7RJKZU}},
  note         = {Machine review of arXiv:2506.13888}
}
read the original abstract

Reinforcement Fine-Tuning (RFT) with verifiable rewards has advanced large language models but remains underexplored for Vision-Language (VL) models. The Vision-Language Reward Model (VL-RM) is key to aligning VL models by providing structured feedback, yet training effective VL-RMs faces two major challenges. First, the bootstrapping dilemma arises as high-quality training data depends on already strong VL models, creating a cycle where self-generated supervision reinforces existing biases. Second, modality bias and negative example amplification occur when VL models hallucinate incorrect visual attributes, leading to flawed preference data that further misguides training. To address these issues, we propose an iterative training framework leveraging vision experts, Chain-of-Thought (CoT) rationales, and Margin-based Rejection Sampling. Our approach refines preference datasets, enhances structured critiques, and iteratively improves reasoning. Experiments across VL-RM benchmarks demonstrate superior performance in hallucination detection and multimodal reasoning, advancing VL model alignment with reinforcement learning.

Figures

Figures reproduced from arXiv: 2506.13888 by the authors.

Figure 1
Figure 1. Illustration of the Vision-Language Reward [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An example of a pairwise VLGenRM training data. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The pipeline starts with a hallucinated negative [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Iterative Training Pipeline of VL-GenRM. The training consists of two stages: [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Contamination study of the dataset used in [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

71 extracted references · 23 canonical work pages

  1. [1]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  2. [2]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv:2412.16720, 2024

    arXiv 2024

  3. [3]

    Vlrewardbench: A challenging benchmark for vision-language generative reward models

    Lei Li, Yuancheng Wei, Zhihui Xie, Xuqing Yang, Yifan Song, Peiyi Wang, Chenxin An, Tianyu Liu, Sujian Li, Bill Yuchen Lin, et al. Vlrewardbench: A challenging benchmark for vision-language generative reward models. arXiv preprint arXiv:2411.17451, 2024 a

    arXiv 2024

  4. [4]

    Aligning large multimodal models with factually augmented rlhf

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. In ACL, 2024

    work page 2024

  5. [5]

    Silkie: Preference distillation for large visual language models, 2023

    Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models, 2023

    work page 2023

  6. [6]

    Visual-rft: Visual reinforcement fine-tuning

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025

    arXiv 2025

  7. [7]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  8. [8]

    Strengthening multimodal large language model with bootstrapped preference optimization, 2024 a

    Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. Strengthening multimodal large language model with bootstrapped preference optimization, 2024 a

    work page 2024

Show all 71 references
  1. [9]

    RLAIF-V : Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness

    Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. RLAIF-V : Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220, 2024 a

    2024

  2. [10]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2023

    2023

  3. [11]

    Self-rewarding language models

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. In ICML, 2024

    2024

  4. [12]

    Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In ICML, 2024 a

    2024

  5. [13]

    Rlaif vs

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023

    2023 arXiv

  6. [14]

    Star: Bootstrapping reasoning with reasoning

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. In NeurIPS, 2022

    2022

  7. [15]

    RAFT : Reward ranked finetuning for generative foundation model alignment

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT : Reward ranked finetuning for generative foundation model alignment. TMLR, 2023

    2023

  8. [16]

    Rlhf workflow: From reward modeling to online rlhf

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. TMLR, 2024

    2024

  9. [17]

    Self-play fine-tuning converts weak language models to strong language models

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024 a

    2024 arXiv

  10. [18]

    Image textualization: An automatic framework for creating accurate and detailed image descriptions

    Renjie Pi, Jianshu Zhang, Jipeng Zhang, Rui Pan, Zhekai Chen, and Tong Zhang. Image textualization: An automatic framework for creating accurate and detailed image descriptions. arXiv preprint arXiv:2406.07502, 2024 b

    2024 arXiv

  11. [19]

    Aide: Agentically improve visual language model with domain experts

    Ming-Chang Chiu, Fuxiao Liu, Karan Sapra, Andrew Tao, Yaser Jacoob, Xuezhe Ma, Zhiding Yu, and Guilin Liu. Aide: Agentically improve visual language model with domain experts. arXiv preprint arXiv:2502.09051, 2025

    2025 arXiv

  12. [20]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240, 2024 a

    2024 arXiv

  13. [21]

    Dpo meets ppo: Reinforced token optimization for rlhf

    Han Zhong, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404.18922, 2024

    2024 arXiv

  14. [22]

    Regularizing hidden states enables learning generalizable reward model for llms

    Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for llms. arXiv preprint arXiv:2406.10216, 2024 a

    2024 arXiv

  15. [23]

    Secrets of rlhf in large language models part ii: Reward modeling

    Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.06080, 2024 a

    2024 arXiv

  16. [24]

    Aligning modalities in vision large language models via preference fine-tuning

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411, 2024

    2024 arXiv

  17. [25]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022

    2022

  18. [26]

    Iterative reasoning preference optimization

    Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems, 37: 0 116617--116637, 2024

    2024

  19. [27]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2023 a

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2023 a

    2023

  20. [28]

    Detectron2

    Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https://github.com/facebookresearch/detectron2, 2019

    2019

  21. [29]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347, 2017

    2017 arXiv

  22. [30]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 0 2...

    2022

  23. [31]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  24. [32]

    Hello gpt-4o, 2024

    OpenAI. Hello gpt-4o, 2024. URL https://openai.com/index/hello-gpt-4o/

    2024

  25. [33]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    2022 arXiv

  26. [34]

    Gemini: A family of highly capable multimodal models, 2023

    Google. Gemini: A family of highly capable multimodal models, 2023. URL https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf

    2023

  27. [35]

    Visual instruction tuning, 2023 b

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023 b

    2023

  28. [36]

    Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    2023

  29. [37]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding, 2024 b

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding, 2024 b

    2024

  30. [38]

    Sharegpt4v: Improving large multi-modal models with better captions, 2023

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions, 2023

    2023

  31. [39]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024 b

    2024 arXiv

  32. [40]

    Fast r-cnn

    Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440--1448, 2015

    2015

  33. [41]

    G-detkd: towards general distillation framework for object detectors via contrastive and semantic-guided feature imitation

    Lewei Yao, Renjie Pi, Hang Xu, Wei Zhang, Zhenguo Li, and Tong Zhang. G-detkd: towards general distillation framework for object detectors via contrastive and semantic-guided feature imitation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3...

    2021

  34. [42]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part I 16, pages...

    2020

  35. [43]

    Global-local path networks for monocular depth estimation with vertical cutdepth, 2022

    Doyeon Kim, Woonghyun Ka, Pyungwhan Ahn, Donggyu Joo, Sehwan Chun, and Junmo Kim. Global-local path networks for monocular depth estimation with vertical cutdepth, 2022

    2022

  36. [44]

    Depth anything: Unleashing the power of large-scale unlabeled data, 2024 b

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data, 2024 b

    2024

  37. [45]

    Open-vocabulary object detection via vision and language knowledge distillation

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921, 2021

    2021 arXiv

  38. [46]

    Detclipv3: Towards versatile generative open-vocabulary object detection, 2024

    Lewei Yao, Renjie Pi, Jianhua Han, Xiaodan Liang, Hang Xu, Wei Zhang, Zhenguo Li, and Dan Xu. Detclipv3: Towards versatile generative open-vocabulary object detection, 2024

    2024

  39. [47]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023

    2023

  40. [48]

    Entropy-regularized process reward model

    Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, and Tong Zhang. Entropy-regularized process reward model. arXiv preprint arXiv:2412.11006, 2024 c

    2024

  41. [49]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. arXiv preprint arXiv:2312.08935, 2023

    2023 arXiv

  42. [50]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 0 46595--46623, 2023

    2023

  43. [51]

    Llava-critic: Learning to evaluate multimodal models

    Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava-critic: Learning to evaluate multimodal models. arXiv preprint arXiv:2410.02712, 2024 b

    2024 arXiv

  44. [52]

    Some things are more cringe than others: Iterative preference optimization with the pairwise cringe loss

    Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Iterative preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682, 2023

    2023 arXiv

  45. [53]

    Reinforced self-training (rest) for language modeling

    Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. In EMNLP, 2024

    2024

  46. [54]

    Self-play fine-tuning converts weak language models to strong language models, 2024 b

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models, 2024 b

    2024

  47. [55]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1 0 (2): 0 3, 2022

    2022

  48. [56]

    Wildvision: Evaluating vision-language models in the wild with human preferences

    Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. Wildvision: Evaluating vision-language models in the wild with human preferences. arXiv preprint arXiv:2406.11069, 2024 a

    2024 arXiv

  49. [57]

    RlHF-V : Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. RlHF-V : Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In CVPR, 2024 b

    2024

  50. [58]

    Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024

    2024 arXiv

  51. [59]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169--186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169--186. Spri...

    2024

  52. [60]

    Improving llm reasoning through scaling inference computation with collaborative verification

    Zhenwen Liang, Ye Liu, Tong Niu, Xiangliang Zhang, Yingbo Zhou, and Semih Yavuz. Improving llm reasoning through scaling inference computation with collaborative verification. arXiv preprint arXiv:2410.05318, 2024

    2024 arXiv

  53. [61]

    V-star: Training verifiers for self-taught reasoners

    Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06457, 2024

    2024 arXiv

  54. [62]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024 b

    2024 arXiv

  55. [63]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024 c

    2024 arXiv

  56. [64]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  57. [65]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    2023

  58. [66]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  59. [67]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146, 2024

    2024 arXiv

  60. [68]

    Deepseek-vl: towards real-world vision-language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525, 2024 b

    2024 arXiv

  61. [69]

    Aria: An open multimodal native mixture-of-experts model

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993, 2024 c

    2024 arXiv

  62. [70]

    Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale

    Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint arXiv:2412.05237, 2024

    2024 arXiv

  63. [71]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024 c

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.