Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

DPO often matches DRL for aligning vision-language models

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 23:08 UTC pith:RAEDOOVG

load-bearing objection Useful survey map with a misprinted DPO equation and loose system-card citations; fine as a first overview, not a reliable reference for implementation. the 3 major comments →

arxiv 2509.06759 v1 pith:RAEDOOVG submitted 2025-09-08 cs.LG cs.AI

Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimization

Thanh Thi Nguyen , Campbell Wilson , Janis Dalins This is my paper
classification cs.LG cs.AI
keywords preference learninghuman feedbackvision-language modelsdeep reinforcement learningdirect preference optimizationmultimodal alignmentreward modelinghallucination reduction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper maps the post-training alignment of large vision-language models (LVLMs) into two families: deep reinforcement learning, where a reward model guides policy optimization, and direct preference optimization, where paired preferences update the policy directly without a separate reward model. Its aim is to give a newcomer a usable map: which method to choose, where reward signals come from, how preference data are ranked, and what the main open problems are. The survey claims that DPO often matches or exceeds DRL on alignment benchmarks while being simpler and more stable, and that the real trade-off is between flexibility and efficiency. If the map is faithful, it lets practitioners select both algorithm and dataset by task rather than by trial and error.

Core claim

The paper's organizing claim is that every current preference-learning method for fine-tuning LVLMs belongs to one of two camps. DRL formulates generation as a Markov decision process and optimizes a policy against rewards that come from human-trained reward models, from AI feedback, or from handcrafted rules; representative studies use PPO or GRPO. DPO instead turns pairwise preference data into a classification loss, making the policy its own implicit reward model, and the survey groups its applications by variants such as dynamic reward scaling, iterative DPO, retrieval-augmented DPO, and mixed preference optimization. It further claims that human ranking gives high fidelity at high cost

What carries the argument

The load-bearing formalisms are the DRL Markov decision process (states are image-plus-prefix contexts, actions are next tokens, rewards come from human, AI, or rule-based sources, optimized by PPO or GRPO) and the DPO objective, a classification loss over chosen-versus-rejected responses that uses the policy and reference model as an implicit reward model. The DRL side carries the paper's claims about flexibility and instability; the DPO side carries its claims about efficiency and benchmark parity.

Load-bearing premise

The survey's entire value is as a faithful map, so every method it classifies must actually use DRL or DPO as attributed, and the formulas it prints must match the cited originals.

What would settle it

Check the printed DPO equation in Section II.B against the original DPO paper: the rejected-response ratio appears without the logarithm, and inspect the cited system cards for GPT-4V, Gemini 1.5, and Claude 3 for documented DRL fine-tuning; either discrepancy would falsify the survey's claim to be a faithful map.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A practitioner can pick DRL when the goal needs complex, multi-signal rewards, and DPO when the goal is large-scale preference tuning with limited compute.
  • Reward source choice matters more than algorithm choice: human feedback buys fidelity, AI feedback buys scale, and rule-based rewards buy transparency in well-defined tasks.
  • Machine-ranked preference data are sufficient for many alignment targets, including hallucination reduction, but inherit the ranking model's biases.
  • Hybrid schemes that combine DRL and DPO, as with the latest Llama release, are a natural next step beyond choosing one family.
  • Open problems named by the survey—sample efficiency, reward modeling, generalization, and safety—define the near-term research agenda for LVLM alignment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If DPO truly matches DRL on typical benchmarks, the field's investment may shift from reward-model engineering to preference-data collection, since the extra machinery of DRL buys little in common cases.
  • The same DRL/DPO dichotomy plausibly extends to video, audio, and embodied inputs, because neither formalism is tied to images or text.
  • A direct, controlled comparison across DRL and DPO on identical preference data and base models would quantify when the claimed parity holds and when it breaks.
  • The survey's table of datasets suggests a practical recipe: start with machine-ranked data for hallucination and safety tasks, and reserve human ranking for subjective or high-stakes alignment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript is a survey of preference-learning methods for fine-tuning large vision-language models (LVLMs), organized around two families: Deep Reinforcement Learning (DRL) and Direct Preference Optimization (DPO). It provides formal descriptions of both approaches (Sections II.A and II.B), a qualitative comparison (Table I), a tabular review of DRL and DPO studies (Tables II and III), a list of preference datasets (Table IV), and a discussion of future research directions (Section VI). The abstract claims the survey provides 'a clear understanding' of how DRL and DPO can align LVLMs with human preferences, and the paper's main value is as a map for practitioners selecting alignment methods and datasets.

Significance. If the survey were technically accurate and faithfully representative, it would fill a useful niche: the LVLM alignment literature is growing quickly, and a structured comparison of DRL and DPO with reward-source and preference-source taxonomies would help newcomers navigate it. The organizational framework (Fig. 1) and the tabulated summaries of studies and datasets are a reasonable starting point. The paper does not present new derivations, experiments, or machine-checked artifacts, so its contribution is exclusively didactic and organizational. That makes technical fidelity and attribution accuracy load-bearing: a reader who implements the DPO objective as printed, or who trusts the system-card attributions, would be misled. The strengths are the clear high-level taxonomic idea, the inclusion of recent GRPO-based work, and the collection of preference datasets with links. These strengths do not compensate for the concrete correctness problems detailed below, but they do mean the manuscript is salvageable with targeted revisions.

major comments (3)
  1. [II.B, DPO objective] The printed DPO loss omits the logarithm on the rejected-response ratio. The equation is L(θ) = −E[log σ( β log(πθ(yw|x,I)/πref(yw|x,I)) − β πθ(yl|x,I)/πref(yl|x,I) )]. In Rafailov et al. (2023), the second term is β log(πθ(yl|x,I)/πref(yl|x,I)). This is not a typesetting typo: at initialization πθ=πref, the printed loss equals −log σ(−β) = log(1+e^β), which depends on β, whereas the correct loss equals log 2 for any β. The gradient with respect to the rejected response is also qualitatively different. A practitioner who implements the loss exactly as printed would not be running DPO. Since the survey's central claim is to provide a faithful technical map of DPO, this error must be corrected before publication.
  2. [II.C, DRL attribution] The sentence 'DRL ... has been successfully applied in systems like GPT-4V [9], Gemini 1.5 [10], Claude 3 [11], and DeepSeek-R1 [12]' is not supported by the cited sources. The GPT-4V system card, Gemini 1.5 technical report, and Claude 3 model card do not document DRL-based fine-tuning of those models; they primarily describe capabilities, safety evaluations, and (in some cases) post-training procedures that are not DRL. DeepSeek-R1 does use reinforcement learning, but it is a text-only LLM, not an LVLM, and the cited paper is about LLM reasoning, not vision-language alignment. This attribution error directly undermines the survey's reliability as a map. The authors should either remove these examples or replace them with documented DRL-based LVLM systems.
  3. [Table I, Performance row] The row 'Performance: Strong ... Strong, often matches or exceeds DRL in benchmarks' is a substantive comparative claim, but the manuscript provides no citation or supporting evidence for it. Table III lists DPO studies, yet none of them is shown to compare against a DRL baseline, and Table II does not include DPO baselines either. Given the survey's didactic purpose, this unqualified claim could mislead a reader into selecting DPO over DRL without seeing the underlying evidence. The authors should either add supporting citations, qualify the claim to specific benchmarks and studies, or remove it.
minor comments (5)
  1. [Table IV, LVLM NLF row] The domain entry says 'Reduce harmlessness'. This is likely a typo for 'Reduce harmfulness'. As written, the text suggests the dataset aims to make models less harmless, which is the opposite of the intended meaning.
  2. [Section III, Table II] The label 'PPO-clip' in Table II for [17] is listed as the DRL algorithm, but the method in the cited paper is a PPO variant that uses CLIP-based rewards. The naming 'PPO-clip' is ambiguous because PPO itself already has a clipped objective; consider renaming to 'PPO + CLIP reward' or similar.
  3. [Section IV, Table III] Some entries have typographical artifacts, e.g., 'LLaV A-OV-7B' and 'LLaV A 1.5' should be 'LLaVA-OV-7B' and 'LLaVA-1.5'. These are minor but should be cleaned up in a camera-ready version.
  4. [Section V, Table IV] The dataset table would benefit from a column specifying the LVLM base model used for preference generation, since several datasets (e.g., VLFeedback, SPA-VL) were constructed using different generators and labellers. This would help practitioners judge portability.
  5. [Section II.B] The text says 'DPO maximizes the log-likelihood ratio between the chosen and rejected outputs', but the DPO loss actually minimizes the negative log-sigmoid of a difference of log-ratios. The wording is imprecise and could be tightened to match the equation.

Circularity Check

0 steps flagged

No significant circularity: survey is a literature map with no derivation chain, prediction, or load-bearing self-citation.

full rationale

This paper is a survey/taxonomy of DRL and DPO methods for LVLM fine-tuning. It makes no quantitative predictions and fits no parameters, so the principal circularity failure modes (self-definitional equations, fitted inputs renamed as predictions, derived result equal to input) do not arise. The only self-citations are refs [6] and [43], used respectively for general PPO/policy-gradient background and for human-in-the-loop DRL as a future direction; neither citation is load-bearing for the survey's organizing claim that DRL and DPO are the two main preference-learning families. The DPO objective in Section II.B is misprinted (the logarithm is missing on the rejected-response ratio relative to Rafailov et al.), but a typographical/correctness error is not circularity: the survey does not claim to derive DPO, and it explicitly attributes the objective to external reference [4]. Similarly, the statement that DRL was used in GPT-4V, Gemini 1.5, and Claude 3 is an attribution claim, not a derivation; even if the cited system cards do not support it, that is a fidelity/factual issue, not a circular reduction. The survey is self-contained as an organization of existing published work, and its central categorization is not equivalent by construction to any input it cites. Hence a circularity score of 0 is appropriate.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The survey rests on the correctness of its restatements of prior methods and on the representativeness of its paper selection. Three domain assumptions carry most of the epistemic weight: faithful restatement of the DPO/GRPO objectives, a representative selection of studies in Tables II-IV, and the claim that specific production systems used DRL. None of these is demonstrated inside the paper, and the first is violated by the misprinted DPO loss. No free parameters or invented entities are introduced.

axioms (3)
  • domain assumption The DPO and GRPO objectives restate the originals correctly (Rafailov et al. 2023; DeepSeekMath).
    Section II.A-II.B present both losses as the working definitions. The DPO equation drops the logarithm in the second ratio, so the restatement is not faithful; a survey's usefulness depends on this restatement.
  • domain assumption Tables II-IV are a representative selection of the DRL/DPO-for-LVLM literature.
    Sections III-V list roughly 30 studies and datasets without stating search or inclusion criteria; the conclusion that these are 'the key approaches' depends on the undocumented selection.
  • domain assumption The production systems GPT-4V, Gemini 1.5, Claude 3, and DeepSeek-R1 were aligned with DRL.
    Section II.C asserts this and cites the GPT-4V system card, Gemini 1.5 report, Claude 3 model card, and DeepSeek-R1 paper; only DeepSeek-R1 documents RL, so this premise is questionable for the other three.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimization." pith.science (2026). https://pith.science/paper/RAEDOOVG

@misc{pith2026250906759,
  author       = {Pith},
  title        = {Pith review of: Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RAEDOOVG}},
  note         = {Machine review of arXiv:2509.06759}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Vision-Language Models (LVLMs) or multimodal large language models represent a significant advancement in artificial intelligence, enabling systems to understand and generate content across both visual and textual modalities. While large-scale pretraining has driven substantial progress, fine-tuning these models for aligning with human values or engaging in specific tasks or behaviors remains a critical challenge. Deep Reinforcement Learning (DRL) and Direct Preference Optimization (DPO) offer promising frameworks for this aligning process. While DRL enables models to optimize actions using reward signals instead of relying solely on supervised preference data, DPO directly aligns the policy with preferences, eliminating the need for an explicit reward model. This overview explores paradigms for fine-tuning LVLMs, highlighting how DRL and DPO techniques can be used to align models with human preferences and values, improve task performance, and enable adaptive multimodal interaction. We categorize key approaches, examine sources of preference data, reward signals, and discuss open challenges such as scalability, sample efficiency, continual learning, generalization, and safety. The goal is to provide a clear understanding of how DRL and DPO contribute to the evolution of robust and human-aligned LVLMs.

Figures

Figures reproduced from arXiv: 2509.06759 by Campbell Wilson, Janis Dalins, Thanh Thi Nguyen.

Figure 1
Figure 1. Figure 1: The structure of this survey paper where we categorize preference learning methods for fine-tuning large vision-language models (LVLMs) into Deep [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The differences between DRL (left) and DPO (right) approaches for fine-tuning LVLMs (adapted from [ [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Beyond the Literal: Decomposing Pragmatic Intent in Multimodal Meme Understanding

    cs.CL 2026-06 unverdicted novelty 6.0

    Intent Projection decomposes literal and pragmatic signals in multimodal memes via orthogonal projection and contrastive objectives in LVLMs, outperforming baselines on six benchmarks especially for high-divergence posts.

Reference graph

Works this paper leans on

44 extracted references · 27 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS), vol. 36, pp. 34 892–34 916, 2023

    work page 2023

  2. [2]

    MiniGPT-4: Enhancing vision-language understanding with advanced large language models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “MiniGPT-4: Enhancing vision-language understanding with advanced large language models,” inInternational Conference on Learning Representations (ICLR), 2024, https://openreview.net/forum?id=1tZbq88f27

    work page 2024

  3. [3]

    FuRL: visual-language models as fuzzy rewards for reinforcement learning,

    Y . Fu, H. Zhang, D. Wu, W. Xu, and B. Boulet, “FuRL: visual-language models as fuzzy rewards for reinforcement learning,” inProceedings of the 41st International Conference on Machine Learning, 2024, pp. 14 256–14 274

    work page 2024

  4. [4]

    Direct preference optimization: your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, “Direct preference optimization: your language model is secretly a reward model,” inAnnual Conference on Neural Information Processing Systems (NeurIPS), 2023, pp. 53 728–53 741

    work page 2023

  5. [5]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  6. [6]

    Deep reinforcement learning for cyber security,

    T. T. Nguyen and V . J. Reddi, “Deep reinforcement learning for cyber security,”IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 8, pp. 3779–3795, 2023

    2023

  7. [7]

    DeepSeekMath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wuet al., “DeepSeekMath: Pushing the limits of mathematical reasoning in open language models,”arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Vision-R1: Evolving human-free alignment in large vision- language models via vision-guided reinforcement learning,

    Y . Zhan, Y . Zhu, S. Zheng, H. Zhao, F. Yang, M. Tang, and J. Wang, “Vision-R1: Evolving human-free alignment in large vision- language models via vision-guided reinforcement learning,”arXiv preprint arXiv:2503.18013, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    GPT-4V(ision) System Card,

    OpenAI, “GPT-4V(ision) System Card,” 2023, https://cdn.openai.com/ papers/GPTV System Card.pdf, Last accessed: 2025-06-15

    work page 2023

  10. [10]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    G. Google, P. Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wanget al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,”arXiv preprint arXiv:2403.05530, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    The Claude 3 model family: Opus, Sonnet, Haiku,

    Anthropic, “The Claude 3 model family: Opus, Sonnet, Haiku,” 2024, https://assets.anthropic.com/m/61e7d27f8c8f5919/original/ Claude-3-Model-Card.pdf, Last accessed: 2025-06-15

    work page 2024

  12. [12]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Biet al., “DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning,”arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    The Llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The Llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Qwen2.5-VL technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tanget al., “Qwen2.5-VL technical report,”arXiv preprint arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation,

    Meta-AI, “The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation,” 2025, https://ai.meta.com/blog/ llama-4-multimodal-intelligence/, Last accessed: 2025-06-15

    work page 2025

  16. [16]

    Aligning large multimodal models with factually augmented RLHF,

    Z. Sun, S. Shen, S. Cao, H. Liu, C. Li, Y . Shen, C. Gan, L. Gui, Y .-X. Wang, Y . Yanget al., “Aligning large multimodal models with factually augmented RLHF,” inFindings of the Association for Computational Linguistics ACL 2024, 2024, pp. 13 088–13 110

    work page 2024

  17. [17]

    Fusing pre-trained language models with multimodal prompts through reinforcement learning,

    Y . Yu, J. Chung, H. Yun, J. Hessel, J. S. Park, X. Lu, R. Zellers, P. Am- manabrolu, R. Le Bras, G. Kimet al., “Fusing pre-trained language models with multimodal prompts through reinforcement learning,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 10 845–10 856

    work page 2023

  18. [18]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational Conference on Machine Learning, 2021, pp. 8748–8763

    2021

  19. [19]

    Improving vision-language-action model with online reinforcement learning,

    Y . Guo, J. Zhang, X. Chen, X. Ji, Y .-J. Wang, Y . Hu, and J. Chen, “Improving vision-language-action model with online reinforcement learning,”arXiv preprint arXiv:2501.16664, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Fine-tuning large vision-language models as decision-making agents via reinforcement learning,

    Y . Zhai, H. Bai, Z. Lin, J. Pan, S. Tong, Y . Zhou, A. Suhr, S. Xie, Y . LeCun, Y . Ma, and S. Levine, “Fine-tuning large vision-language models as decision-making agents via reinforcement learning,” inAnnual Conference on Neural Information Processing Systems (NeurIPS), 2024, https://openreview.net/forum?id=nBjmMF2IZU

    work page 2024

  21. [21]

    VLFeedback: A large-scale AI feedback dataset for large vision-language models alignment,

    L. Li, Z. Xie, M. Li, S. Chen, P. Wang, L. Chen, Y . Yang, B. Wang, L. Kong, and Q. Liu, “VLFeedback: A large-scale AI feedback dataset for large vision-language models alignment,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 6227–6246

    work page 2024

  22. [22]

    MM-RLHF: The next step forward in multi- modal LLM alignment,

    Y .-F. Zhang, T. Yu, H. Tian, C. Fu, P. Li, J. Zeng, W. Xie, Y . Shi, H. Zhang, J. Wuet al., “MM-RLHF: The next step forward in multi- modal LLM alignment,”arXiv preprint arXiv:2502.10391, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Insight-V: Exploring long-chain visual reasoning with multimodal large language models,

    Y . Dong, Z. Liu, H.-L. Sun, J. Yang, W. Hu, Y . Rao, and Z. Liu, “Insight-V: Exploring long-chain visual reasoning with multimodal large language models,”arXiv preprint arXiv:2411.14432, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training language models to follow instructions with human feedback,” inProceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS), 2022, pp. 27 730–27 744

    work page 2022

  25. [25]

    RLHF-V: Towards trustworthy MLLMs via behavior alignment from fine-grained correctional human feedback,

    T. Yu, Y . Yao, H. Zhang, T. He, Y . Han, G. Cui, J. Hu, Z. Liu, H.- T. Zheng, M. Sunet al., “RLHF-V: Towards trustworthy MLLMs via behavior alignment from fine-grained correctional human feedback,” in IEEE/CVF Conference on CVPR, 2024, pp. 13 807–13 816

    work page 2024

  26. [26]

    RLAIF-V: Aligning MLLMs through open- source AI feedback for super GPT-4V trustworthiness,

    T. Yu, H. Zhang, Y . Yao, Y . Dang, D. Chen, X. Lu, G. Cui, T. He, Z. Liu, T.-S. Chuaet al., “RLAIF-V: Aligning MLLMs through open- source AI feedback for super GPT-4V trustworthiness,”arXiv preprint arXiv:2405.17220, 2024

    arXiv 2024

  27. [27]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization,

    W. Wang, Z. Chen, W. Wang, Y . Cao, Y . Liu, Z. Gao, J. Zhu, X. Zhu, L. Lu, Y . Qiaoet al., “Enhancing the reasoning ability of multimodal large language models via mixed preference optimization,” arXiv preprint arXiv:2411.10442, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Aligning modalities in vision large language models via preference fine-tuning,

    Y . Zhou, C. Cui, R. Rafailov, C. Finn, and H. Yao, “Aligning modalities in vision large language models via preference fine-tuning,” inICLR 2024 Workshop on Reliable and Responsible Foundation Models, 2024, https://openreview.net/forum?id=GRGvC0rpA8

    work page 2024

  29. [29]

    Re-Align: Aligning vision language models via retrieval-augmented direct preference optimization,

    S. Xing, Y . Wang, P. Li, R. Bai, Y . Wang, C.-w. Hu, C. Qian, H. Yao, and Z. Tu, “Re-Align: Aligning vision language models via retrieval-augmented direct preference optimization,”arXiv preprint arXiv:2502.13146, 2025

    work page arXiv 2025

  30. [30]

    Enhancing visual-language modality alignment in large vision language models via self-improvement,

    X. Wang, J. Chen, Z. Wang, Y . Zhou, Y . Zhou, H. Yao, T. Zhou, T. Gold- stein, P. Bhatia, F. Huanget al., “Enhancing visual-language modality alignment in large vision language models via self-improvement,” in Findings of the Association for Computational Linguistics: NAACL, 2025, pp. 268–282

    work page 2025

  31. [31]

    VILA: On pre-training for visual language models,

    J. Lin, H. Yin, W. Ping, P. Molchanov, M. Shoeybi, and S. Han, “VILA: On pre-training for visual language models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 689–26 699

    work page 2024

  32. [32]

    SPA-VL: A comprehensive safety preference alignment dataset for vision language model,

    Y . Zhang, L. Chen, G. Zheng, Y . Gao, R. Zheng, J. Fu, Z. Yin, S. Jin, Y . Qiao, X. Huanget al., “SPA-VL: A comprehensive safety preference alignment dataset for vision language model,”arXiv preprint arXiv:2406.12030, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    DRESS: Instructing large vision-language models to align and interact with humans via natural language feedback,

    Y . Chen, K. Sikka, M. Cogswell, H. Ji, and A. Divakaran, “DRESS: Instructing large vision-language models to align and interact with humans via natural language feedback,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 239–14 250

    work page 2024

  34. [34]

    Active learning for vision-language models,

    B. Safaei and V . M. Patel, “Active learning for vision-language models,” inIEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2025, pp. 4902–4912

    work page 2025

  35. [35]

    S-CLIP: Semi-supervised vision- language learning using few specialist captions,

    S. Mo, M. Kim, K. Lee, and J. Shin, “S-CLIP: Semi-supervised vision- language learning using few specialist captions,”Advances in Neural Information Processing Systems, vol. 36, pp. 61 187–61 212, 2023

    work page 2023

  36. [36]

    RL-VLM-F: reinforcement learning from vision language foundation model feedback,

    Y . Wang, Z. Sun, J. Zhang, Z. Xian, E. Biyik, D. Held, and Z. Erickson, “RL-VLM-F: reinforcement learning from vision language foundation model feedback,” inProceedings of the 41st International Conference on Machine Learning, 2024, pp. 51 484–51 501

    work page 2024

  37. [37]

    VL- Rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning,

    H. Wang, C. Qu, Z. Huang, W. Chu, F. Lin, and W. Chen, “VL- Rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning,”arXiv preprint arXiv:2504.08837, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    Text-to-decision agent: Offline meta-reinforcement learning from natural language supervision,

    S. Zhang, Z. Hu, W. Wu, X. Xie, J. Tang, C. Chen, D. Dong, Y . Cheng, Z. Sun, and Z. Wang, “Text-to-decision agent: Offline meta-reinforcement learning from natural language supervision,”arXiv preprint arXiv:2504.15046, 2025

    work page arXiv 2025

  39. [39]

    Zero-shot model-based reinforcement learning using large language models,

    A. Benechehab, Y . A. E. Hili, A. Odonnat, O. Zekri, A. Thomas, G. Paolo, M. Filippone, I. Redko, and B. K ´egl, “Zero-shot model-based reinforcement learning using large language models,” inThe Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=uZFXpPrwSh

    work page 2025

  40. [40]

    Boosting the generalization and reasoning of vision language models with curriculum reinforcement learning,

    H. Deng, D. Zou, R. Ma, H. Luo, Y . Cao, and Y . Kang, “Boosting the generalization and reasoning of vision language models with curriculum reinforcement learning,”arXiv preprint arXiv:2503.07065, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    Open-ended VQA benchmarking of vision-language models by exploiting classification datasets and their semantic hierarchy,

    S. Ging, M. A. Bravo, and T. Brox, “Open-ended VQA benchmarking of vision-language models by exploiting classification datasets and their semantic hierarchy,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https: //openreview.net/forum?id=EXitynZhYn

    work page 2024

  42. [42]

    VLP: Vision-Language Preference Learning for Embodied Manipulation

    R. Liu, C. Bai, J. Lyu, S. Sun, Y . Du, and X. Li, “VLP: Vision- language preference learning for embodied manipulation,”arXiv preprint arXiv:2502.11918, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  43. [43]

    Multi-agent deep reinforcement learning with human strategies,

    T. Nguyen, N. D. Nguyen, and S. Nahavandi, “Multi-agent deep reinforcement learning with human strategies,” inIEEE International Conference on Industrial Technology (ICIT), 2019, pp. 1357–1362

    work page 2019

  44. [44]

    ETA: Evaluating then align- ing safety of vision language models at inference time,

    Y . Ding, B. Li, and R. Zhang, “ETA: Evaluating then align- ing safety of vision language models at inference time,” inIn- ternational Conference on Learning Representations (ICLR), 2025, https://openreview.net/forum?id=QoDDNkx4fP

    work page 2025

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.