Pith. sign in

REVIEW 3 major objections 6 minor 47 references

Continual SFT Matches Multimodal RLHF with Negative Supervision

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that the success of multimodal RLHF comes mostly from negative supervision, and that a simple SFT loss on LLM-constructed corrective conversations matches DPO and PPO while using fewer models.

desk verdict Broad empirical comparison, but the theory is wrong and the causal claim about negative supervision is not actually supported by an ablation. read the letter →

arxiv 2411.14797 v1 pith:6LJETYVC submitted 2024-11-22 cs.LG cs.AIcs.CLcs.CV

Ke Zhu , Yu Wang , Yanpeng Sun , Qiang Chen , Jiangjiang Liu , Gang Zhang , Jingdong Wang This is my paper · ORCID
classification cs.LGcs.AIcs.CLcs.CV
keywords negativesupervisedfinetuningmultimodalRLHFdirectpreferenceoptimizationvision-languagemodelshallucinationcorrectionalignmentLLMdataconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that multimodal RLHF is not superior to continual SFT because of reinforcement learning machinery; the decisive ingredient is the negative supervision carried by rejected responses. It proposes nSFT, which runs the base vision-language model to produce responses, asks an LLM with a vision error codebook to identify the errors in those rejected responses, and constructs corrective question-answer conversations. The model is then trained with an ordinary SFT loss on the chosen captions plus the constructed corrections. The authors report that nSFT matches or beats DPO, PPO, and iterative DPO variants across three data sources, three model scales, and nine benchmarks, while requiring only one VLM in training instead of two or four.

What carries the argument

The load-bearing object is the negative supervision disentangled from the DPO logit: the term $-L_{sft}(y_r)$ that appears when the DPO loss is rewritten as a difference of two next-token-prediction losses. Since that term is a sequence-level SFT loss on the rejected response, nSFT replaces it by $L_{sft}(G(y_r; y_c, Q))$, where $G(\cdot)$ is an LLM prompted with the rejected response, the chosen response, and a vision error codebook $Q$ listing instance-level and image-level error types (object identity, attribute, action, location, relative position, background, events, reasoning). The construction function converts the pairwise preference signal into plain corrective conversation data, so the alignment stage can run with one model and a standard SFT loss.

What would settle it

Take a held-out set of preference pairs; compute the gradient of the DPO loss with respect to the policy and the gradient of the nSFT loss restricted to the constructed negative term, and measure their cosine similarity across examples. If many examples show near-zero or negative similarity, the claimed equivalence fails; alternatively, replace the GPT-4 construction function with a random or non-error-aware captioner and check whether nSFT's gains over continual SFT disappear.

Watch

Extended reading notes

Core claim

The central claim is that the key success factor of multimodal RLHF (e.g., DPO) mostly attributes to the negative supervision in the rejected responses. The paper derives that DPO's gradient is a linear combination of two SFT gradients, one for the chosen response and one for the rejected response, so continual SFT falls behind mainly because it lacks the rejected-response term. nSFT recovers that missing term by disentangling it from the pairwise DPO logit and expressing it as an SFT target: it uses a construction function $G(\cdot)$ that identifies the wrong statements in a rejected response, guided by a vision error codebook and the chosen response, and writes them into a corrective conversation. The final loss is $L_{nSFT} = L_{sft}(y_c) + L_{sft}(G(y_r; y_c, Q))$, and the paper claims this strictly matches multimodal preference alignment methods (both DPO and PPO) under different datasets, base VLMs, and evaluation metrics.

Load-bearing premise

The load-bearing premise is that GPT-4's error identification, guided by the vision error codebook, extracts the same corrective signal that DPO's rejected-response logit would provide, and the paper's derivation of this equivalence (Eq. 20-23) does not fully prove it because it drops the sigmoid derivative.

Editorial extensions

If this is right

  • Preference alignment for VLMs can be done with an SFT loss once the rejected-response errors are turned into corrective text, so no reward model, reference model, or policy-reference pair is required.
  • Training memory drops from two models (DPO) or four (PPO) to one, and the measured training time is substantially lower.
  • nSFT improves over pure continual SFT on hallucination benchmarks, with the largest gains on POPE, CHAIR, and MMHal across OCRVQA, TextCaps, and LLaVA-150k data.
  • The approach transfers to stronger and larger VLMs (LLaVA-1.5-13B and LLaVA-NeXT-13B) and outperforms iterative DPO (CSR) and PPO on the tested benchmarks.
  • Adding a per-token KL constraint to nSFT further improves results, suggesting the method composes with RLHF-style regularization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the equivalence holds, the practical distinction between SFT and RLHF becomes a difference in data selection rather than optimizer: any preference signal that can be written as corrective text is trainable with cross-entropy.
  • The construction step shifts cost from GPU memory to an LLM API; a natural extension is to distill the construction function into a smaller local model, or to generate corrective conversations on the fly during training.
  • A direct way to test the paper's mechanism is to measure the gradient cosine similarity between DPO's negative logit term and the nSFT negative term on the same preference pairs; the paper's formal derivation (Eq. 20-23) skips the sigmoid derivative, so that similarity is not yet established.
  • The same recipe may transfer to NLP alignment, where the error types are toxicity, style, or factual mistakes rather than visual hallucinations; the paper lists this as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that the success of multimodal RLHF (DPO/PPO) in continual VLM alignment is largely due to the negative supervision provided by rejected responses. It proposes nSFT, which uses an LLM (GPT-4) guided by a vision error codebook to construct corrective conversations from the rejected responses, and trains the VLM with a standard SFT loss on the chosen caption plus the constructed conversation. The paper reports experiments across three alignment data sources, multiple base VLMs (LLaVA-1.5-7B/13B, LLaVA-NeXT-13B), and various RLHF baselines (GT-DPO, SeVa, SIMA, PPO, iterative DPO), finding that nSFT matches or exceeds the RLHF baselines while requiring only one model in training. The central claim is that nSFT 'strictly matches' multimodal preference alignment by excavating the same negative supervision.

Significance. If established, the claim that a simple continual SFT loss on LLM-reconstructed corrective conversations matches DPO/PPO would have substantial practical value, since nSFT avoids the memory overhead of maintaining reference, policy, and reward models. The empirical coverage is a strength: the paper evaluates on nine benchmarks, three data sources, two model scales, and four RLHF variants, and the VEC ablation indicates that the codebook contributes to the gains. However, the theoretical derivation that forms the paper's stated basis for the negative-supervision equivalence contains a calculus error, and the empirical design does not directly isolate the contribution of the rejected response, so the causal claim as stated is not yet supported.

major comments (3)
  1. [Appendix A.1, Eqs. (20)-(23)] The derivative of -log sigma(beta p_dpo) with respect to theta is computed incorrectly. Since d/dz[-log sigma(z)] = -sigma(-z), the prefactor in Eq. (20) is -beta sigma(-beta p_dpo), not -1/p_dpo. Consequently, Eq. (9) and Eq. (23) are wrong as written. A corrected derivation would still express the DPO gradient as a linear combination of the chosen and rejected SFT gradients, but with a sigmoid coefficient rather than 1/p_dpo scaling. This does not invalidate the overall idea, but the specific mathematical claim used to justify the equivalence is incorrect and must be fixed.
  2. [Section 3.3, Eq. (15); Table 6] The nSFT loss L_nSFT = L_sft(yc) + L_sft(G(yr; yc, Q)) is evaluated on the constructed correction G(yr), not on the rejected response yr itself. DPO's negative supervision is realized by decreasing the likelihood of yr. The manuscript does not provide a gradient-level argument showing that increasing the likelihood of G(yr) has an equivalent effect on the model's tendency to produce the original error, and Table 6 does not include an ablation that removes the rejected response from the construction; it ablates the vision error codebook and the chosen response. To support the central causal claim, the authors should add a 'w/o rejected response' condition or an explicit derivation linking the gradient of L_sft(G(yr)) on the error-related tokens to the DPO negative gradient.
  3. [Section 3.2, Eqs. (11)-(13)] The conclusion that DPO is 'biased towards how to reject samples' is based on the ratio of partial derivatives with respect to the abstract variables t1 and t2. This ratio alone does not determine the relative contributions to the parameter gradient, because the Jacobians dt1/dtheta and dt2/dtheta are not equal. Please either provide a derivation in terms of the parameter gradient or explicitly defer to the argument in [7] rather than presenting Eqs. (11)-(13) as a self-contained proof.
minor comments (6)
  1. [Section 4.1] The phrase 'a huge discrepancy of mutlimodal RLHF literature' contains a typo: 'mutlimodal' should be 'multimodal'.
  2. [Section 3.2] The phrase 'with faster gradient updating rate of t1' appears to contradict the ratio in Eq. (13), which indicates |∂L/∂t2| > |∂L/∂t1| when t2 < t1; please correct the direction or rephrase.
  3. [Table 1 and Table 9] The calculation of the 'total' score is not defined; please specify whether it is the sum of per-column improvements over the baseline.
  4. [Section 4.4, Table 2] The 15k nSFT data are merged with the LLaVA-665k SFT data during training, which differs from the continual-learning setup in Table 1; this should be explicitly acknowledged in the main text rather than only in the caption.
  5. [Appendix B.1] The number of constructed conversations per image (5) is a free parameter; no sensitivity analysis is provided for this choice.
  6. [Abstract and Conclusion] The phrase 'fruitful of ablations' should be rephrased, e.g., 'fruitful ablations' or 'a wealth of ablations'.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: nSFT is validated on held-out benchmarks, the minor SeVa self-citation is not load-bearing, and the invalid DPO-to-SFT derivation is a correctness gap rather than a circular reduction.

full rationale

I walked the paper's derivation chain and found no step where a prediction reduces to its inputs by construction. The central empirical claim—nSFT matches multimodal RLHF/DPO/PPO—is tested on held-out benchmarks (Tables 1-5) and is not obtained by fitting a parameter to those benchmarks. The construction function G(yr; yc, Q) is an external GPT-4-based process, and the nSFT loss Lsft(yc)+Lsft(G(yr; yc, Q)) is a standard SFT objective on LLM-written corrective conversations; it does not directly contain the DPO logit or the rejected-response likelihood term. The theoretical bridge in Sec. 3.2 and Appendix A.1 is the load-bearing weakness: Eq. 20-23 misdifferentiate -log sigma(beta p_dpo), since the correct coefficient is beta sigma(-beta p_dpo) rather than 1/p_dpo, and Table 6 never ablates the rejected response itself, so the causal attribution to 'negative supervision' is under-supported. However, an invalid or incomplete argument is a correctness risk, not circularity, because the result does not reduce to its inputs by definition. The only self-citation is to SeVa [46] by the same first author, used as a baseline and for experimental settings; it is not invoked to forbid alternatives or to supply a uniqueness theorem, and the nSFT results are independently validated on external benchmarks. Score 2 reflects this minor, non-load-bearing self-citation rather than any circular derivation.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim relies on the GPT-4-based construction and the equivalence between the nSFT loss and the DPO negative gradient. The paper provides no machine-checked proof, no code, and the derivation supporting this equivalence contains an error. The vision error codebook is a method component rather than an invented physical entity.

free parameters (1)
  • Number of constructed conversations per image (5) = 5
    Chosen by hand in Appendix B.1; more conversations increase overlap with previous constructed data.
assumptions (5)
  • standard math Bradley-Terry preference model and DPO closed-form reward (Eq. 3-6)
    Standard RLHF and DPO theory adopted as background for the analysis.
  • domain assumption Ground-truth annotations are accurate references for image content
    nSFT construction uses GT as chosen response yc and as reference for error identification; if GT is noisy, the supervision is corrupted.
  • domain assumption GPT-4 can reliably identify hallucinations using the vision error codebook
    The construction function G(·) depends on LLM judgment; its accuracy is not measured independently and could vary across images and prompt versions.
  • ad hoc to paper The reconstructed conversation G(yr; yc, Q) captures the same training signal as the DPO negative gradient
    This substitution is the core of the method and is not proven; the paper's derivation supporting it contains a calculus error.
  • ad hoc to paper Reference model term can be omitted in DPO gradient analysis
    Eq. 8 and Appendix A.1 drop pi_ref from the logit; the authors claim this is valid because the reference model receives no gradient, but the omitted term changes the coefficient in the gradient.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Continual SFT Matches Multimodal RLHF with Negative Supervision." pith.science (2026). https://pith.science/paper/6LJETYVC

@misc{pith2026241114797,
  author       = {Pith},
  title        = {Pith review of: Continual SFT Matches Multimodal RLHF with Negative Supervision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6LJETYVC}},
  note         = {Machine review of arXiv:2411.14797}
}
read the original abstract

Multimodal RLHF usually happens after supervised finetuning (SFT) stage to continually improve vision-language models' (VLMs) comprehension. Conventional wisdom holds its superiority over continual SFT during this preference alignment stage. In this paper, we observe that the inherent value of multimodal RLHF lies in its negative supervision, the logit of the rejected responses. We thus propose a novel negative supervised finetuning (nSFT) approach that fully excavates these information resided. Our nSFT disentangles this negative supervision in RLHF paradigm, and continually aligns VLMs with a simple SFT loss. This is more memory efficient than multimodal RLHF where 2 (e.g., DPO) or 4 (e.g., PPO) large VLMs are strictly required. The effectiveness of nSFT is rigorously proved by comparing it with various multimodal RLHF approaches, across different dataset sources, base VLMs and evaluation metrics. Besides, fruitful of ablations are provided to support our hypothesis. We hope this paper will stimulate further research to properly align large vision language models.

Figures

Figures reproduced from arXiv: 2411.14797 by the authors.

Figure 1
Figure 1. Standard DPO training pipeline (in the first row) and our [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. 2a-2b shows training time, GPU memory of LLaVa-1.5 (‘Base’), 3 DPO techniques: SeVa [46], SIMA [31] and CSR [42], 1 PPO technique [29]. 2c shows multimodal results of averaged DPO methods, pure continual (Cont.) SFT and our nSFT. ing about the image, helping the model to realize these self mistakes from these rejected responses (cf [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Visualization of a standard DPO pipeline and our nSFT method. In DPO, GT annotations usually directly serves as chosen [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Visualization of benchmarks results (relative improvement over baseline LLaVA-1.5-7B) using continual SFT, DPO and our [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Visualizations of response generated from LLaVA-v1.5-7B, DPO and nSFT models. In the left part, the correct content are [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Visualizations of the instruct following score and de [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Visualizations of prompt to evaluate in-domain results [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Visualization of the constructed conversation w/o and w/ our vision error codebook, derived from a 5k subset of LLaVA-150k. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: The prompt template used for error identification and conversation reconstruction. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Visualization of groundtruth (GT) captions in OCRVQA, TextCaps and LLaVA-150k dataset. The GT length from left to right [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Random sampled cases of our negative constructed supervision. The middle part are the model’s original responses to the [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 17 canonical work pages

  1. [7]

    Towards analyzing and understanding the limitations of dpo: A theoretical perspective

    Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and understanding the limitations of dpo: A theoretical perspective. arXiv preprint arXiv:2404.04626, 2024. 3, 4, 7, 9

    arXiv 2024

  2. [1]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. 2023. 7

    work page 2023

  3. [2]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 2

    arXiv 2023

  4. [3]

    Self-play fine-tuning converts weak language models to strong language models

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. In Forty-first Interna- tional Conference on Machine Learning. 1, 2, 3, 4

    work page

  5. [4]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 1

    arXiv 2024

  6. [5]

    Scaling instruction- finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024. 2

    2024

  7. [6]

    DreamLLM: Synergistic multimodal com- prehension and creation

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. DreamLLM: Synergistic multimodal com- prehension and creation. In The Twelfth International Con- ference on Learning Representations, 2024. 2

    work page 2024

  8. [8]

    GQA: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 6

    work page 2019

Show all 47 references
  1. [9]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 1

    2023

  2. [11]

    Multi- modal preference alignment remedies regression of visual instruction tuning on language model

    Shengzhi Li, Rongyu Lin, and Shichao Pei. Multi- modal preference alignment remedies regression of visual instruction tuning on language model. arXiv preprint arXiv:2402.10884, 2024. 2

    2024 arXiv

  3. [12]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pages 292–305, 2023. 6

    2023

  4. [13]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  5. [14]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 1, 2, 5, 6

    2023 arXiv

  6. [15]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 2, 5, 6

    2024

  7. [16]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 10

    2024

  8. [17]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. 6, 10

    2023 arXiv

  9. [18]

    Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer

    Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405.16436, 2024. 2

    2024 arXiv

  10. [19]

    Online merging optimizers for boosting rewards and mitigating tax in alignment

    Keming Lu, Bowen Yu, Fei Huang, Yang Fan, Runji Lin, and Chang Zhou. Online merging optimizers for boosting rewards and mitigating tax in alignment. arXiv preprint arXiv:2405.17931, 2024. 2

    2024 arXiv

  11. [20]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems ...

    2022

  12. [21]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–

    2019

  13. [22]

    Strengthening multi- modal large language model with bootstrapped preference optimization

    Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Run- tao Liu, Rui Pan, and Tong Zhang. Strengthening multi- modal large language model with bootstrapped preference optimization. arXiv preprint arXiv:2403.08730, 2024. 1, 2, 3, 4, 5

    2024 arXiv

  14. [23]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. 2024. 1, 2, 3

    2024

  15. [24]

    Object hallucination in image cap- tioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035–4045,

    2018

  16. [25]

    Multitask prompted training enables zero-shot task generalization

    Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud 15 Stiegler, Arun Raja, Manan Dey, et al. Multitask prompted training enables zero-shot task generalization. In Interna- tional Conference on Learning Representations. 1, 2

  17. [26]

    Proximal policy optimization algo- rithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization algo- rithms. arXiv preprint arXiv:1707.06347, 2017. 1, 2

    2017 arXiv

  18. [27]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 5, 6, 8

    2019

  19. [28]

    Improving multi-modal large language model through boost- ing vision capabilities

    Yanpeng Sun, Huaxin Zhang, Qiang Chen, Xinyu Zhang, Nong Sang, Gang Zhang, Jingdong Wang, and Zechao Li. Improving multi-modal large language model through boost- ing vision capabilities. arXiv preprint arXiv:2410.13733 ,

    arXiv

  20. [29]

    Aligning large multi- modal models with factually augmented rlhf

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multi- modal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023. 1, 2, 7, 8, 10

    2023 arXiv

  21. [30]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023. 2

    2023 arXiv

  22. [31]

    Enhancing visual- language modality alignment in large vision language mod- els via self-improvement

    Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, et al. Enhancing visual- language modality alignment in large vision language mod- els via self-improvement. arXiv preprint arXiv:2405.15973,

    arXiv

  23. [32]

    Vary: Scaling up the vision vocabulary for large vision-language models

    Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language models. arXiv preprint arXiv:2312.06109,

    arXiv

  24. [33]

    Finetuned language models are zero-shot learn- ers

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learn- ers. In International Conference on Learning Representa- tions. 2

  25. [34]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau- mond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R ´emi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, ...

    2020

  26. [35]

    Visual chatgpt: Talking, drawing and editing with visual foundation models

    Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023. 2

    2023 arXiv

  27. [36]

    Is dpo superior to ppo for llm alignment? a comprehensive study

    Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study. In Forty-first International Conference on Machine Learning. 2

  28. [37]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 6, 7, 10

    2023 arXiv

  29. [38]

    Token-level direct prefer- ence optimization

    Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct prefer- ence optimization. In Forty-first International Conference on Machine Learning. 7

  30. [39]

    Chatspot: Bootstrapping multimodal llms via precise referring instruction tuning

    Liang Zhao, En Yu, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunrui Han, et al. Chatspot: Bootstrapping multimodal llms via precise referring instruction tuning. arXiv preprint arXiv:2307.09474, 2023. 2

    2023 arXiv

  31. [40]

    Beyond hallucinations: Enhanc- ing lvlms through hallucination-aware direct preference op- timization

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Ji- aqi Wang, and Conghui He. Beyond hallucinations: Enhanc- ing lvlms through hallucination-aware direct preference op- timization. arXiv preprint arXiv:2311.16839, 2023. 2

    2023 arXiv

  32. [41]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024. 1, 2

    2024

  33. [42]

    Calibrated self-rewarding vision language models

    Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405.14622 , 2024. 2, 7, 8

    2024 arXiv

  34. [43]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 2

    2023 arXiv

  35. [44]

    Multi-label self- supervised learning with scene images

    Ke Zhu, Minghao Fu, and Jianxin Wu. Multi-label self- supervised learning with scene images. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 6694–6703, 2023. 3

    2023

  36. [45]

    Quantized feature distillation for network quantization

    Ke Zhu, Yin-Yin He, and Jianxin Wu. Quantized feature distillation for network quantization. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 11452– 11460, 2023. 10

    2023

  37. [46]

    Self- supervised visual preference alignment

    Ke Zhu, Liang Zhao, Zheng Ge, and Xiangyu Zhang. Self- supervised visual preference alignment. arXiv preprint arXiv:2404.10501, 2024. 1, 2, 3, 4, 5, 6, 7, 11

    2024 arXiv

  38. [47]

    Llava-phi: Efficient multi-modal assistant with small language model

    Yichen Zhu, Minjie Zhu, Ning Liu, Zhiyuan Xu, and Yaxin Peng. Llava-phi: Efficient multi-modal assistant with small language model. In Proceedings of the 1st International Workshop on Efficient Multimedia Computing under Limited, pages 18–22, 2024. 10

    2024

  39. [48]

    Multi: Multimodal understanding leaderboard with text and images

    Zichen Zhu, Yang Xu, Lu Chen, Jingkai Yang, Yichuan Ma, Yiming Sun, Hailin Wen, Jiaqi Liu, Jinyu Cai, Yingzi Ma, et al. Multi: Multimodal understanding leaderboard with text and images. arXiv preprint arXiv:2402.03173, 2024. 10 16

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.