Pith. sign in

REVIEW 3 major objections 5 minor 61 references

ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ASPO claims sentence-level adaptive reward weights make DPO-based multimodal alignment more precise and hallucination-resistant without adding parameters.

desk verdict The paper's proposed loss cancels to standard DPO by its own definitions, so the reported gains cannot be attributed to the described mechanism. read the letter →

arxiv 2505.19100 v1 pith:BS2IDNFC submitted 2025-05-25 cs.CL cs.CV

Yeyuan Wang , Dehong Gao , Rujiao Long , Lei Yi , Linbo Jin , Libin Yang , Xiaoyan Cai This is my paper
classification cs.CLcs.CV
keywords directpreferenceoptimizationmultimodallargelanguagemodelssentence-levelrewardsrewardmodelinghallucinationreductionvision-languagealignmentperplexityimage-textsimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ASPO, a variant of Direct Preference Optimization that assigns each sentence in a response its own reward weight instead of treating the whole response as one unit. The weights are computed from two self-supervised signals: how similar each sentence is to the input image (via CLIP cosine similarity) and how confident the model is in that sentence (via negated, normalized perplexity). The authors claim that this sentence-level adaptive reweighting enables more precise preference optimization and substantially improves multimodal reasoning, including on hallucination benchmarks. If the mechanism works as described, it would offer a parameter-free way to reduce hallucination and strengthen reasoning in vision-language models with only a small change to the DPO objective.

What carries the argument

The central object is the adaptive implicit reward margin $M^*$ in Equation (12), a sentence-level reweighted version of the DPO margin. Each chosen-response sentence $s_i$ is weighted by $\beta(1+w_i)$, where $w_i = \alpha S_i' + (1-\alpha) PPL_i'$ combines min-max normalized image-text similarity (CLIP cosine) and negated, normalized per-sentence perplexity; the paper introduces the ratio $R_c/R_c^*$ as a normalizer intended to prevent longer responses from disproportionately benefiting from a total weight increase. The claimed mechanism is that the weights $w_i$ modulate the reward strength of each sentence, directing optimization toward sentences that are both visually grounded and textually confident.

What would settle it

Expand Equation (12) using the definitions of $R_c$ and $R_c^*$ given in Section 3.2.1 and compute the gradient of the ASPO loss in Equation (13) with respect to the policy parameters. If $\frac{R_c}{R_c^*} \sum_i \beta(1+w_i) \log \frac{\pi_\theta(s_i^c \mid x)}{\pi_{\text{ref}}(s_i^c \mid x)}$ reduces to $\sum_i \beta \log \frac{\pi_\theta(s_i^c \mid x)}{\pi_{\text{ref}}(s_i^c \mid x)}$ for every $\theta$, then the weights $w_i$ do not appear in the objective and the claimed sentence-level modulation is not actually trained; the reported benchmark gains would then require another explanation.

Watch

Extended reading notes

Core claim

ASPO's central claim is that the binary, response-level implicit reward margin of DPO can be replaced by a sentence-level adaptive margin $M^* = \frac{R_c}{R_c^*} \sum_{i=1}^K \beta (1+w_i) \log \frac{\pi_\theta(s_i^c \mid x)}{\pi_{\text{ref}}(s_i^c \mid x)} - \beta \log \frac{\pi_\theta(y_r \mid x)}{\pi_{\text{ref}}(y_r \mid x)}$, where $w_i$ is the adaptive weight of the $i$-th sentence of the chosen response, and $R_c/R_c^*$ is a length-normalization factor described in the paper. The paper argues that this reformulation lets the model reward correct or important sentences more strongly while downweighting noisy ones, yielding more fine-grained preference optimization than standard DPO. It further claims that this translates into consistent benchmark gains across three multimodal base models, with the largest improvements on hallucination-sensitive tasks.

Load-bearing premise

The load-bearing premise is that the normalization factor $R_c/R_c^*$, defined as the ratio of the original to the reweighted sum of sentence-level implicit rewards of the chosen response, rescales the reweighted sum without eliminating the adaptive weights $w_i$; if the weights cancel out of Equation (12), the ASPO objective is algebraically identical to the standard DPO margin and the claimed adaptive mechanism has no effect on the training loss.

Editorial extensions

If this is right

  • Multimodal models trained with ASPO should show reduced hallucination and improved reasoning scores relative to DPO-trained baselines, as measured on POPE, SHR, and MMVet.
  • Because ASPO adds no trainable parameters and requires no external reward models or paid APIs, the same pipeline can be applied to any DPO-based post-training setup at negligible extra cost.
  • The sentence-level weighting is claimed to make training robust to noisy preference pairs, since partially correct or incorrect sentences no longer receive equal reward, addressing a root cause of suboptimal convergence in DPO.
  • The method scales with base-model quality: the paper observes larger gains on stronger base models, suggesting the adaptive weights become more accurate as the model's own predictions improve.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would be to annotate sentence-level ground truth on a small evaluation set and check whether the learned weights $w_i$ actually track sentence correctness, which would directly test the paper's interpretation that the weights 'prioritize accurate reasoning.'
  • The same self-supervised weighting scheme could be applied to other fine-grained units such as clauses or content-word spans, and to modalities beyond vision-language, since neither CLIP similarity nor perplexity is inherently specific to this setting.
  • If the normalization factor $R_c/R_c^*$ cancels the adaptive weights as written, the reported gains might instead come from the data collection pipeline (contrastive augmented-image pairs) rather than from the sentence-level loss; comparing ASPO against DPO on the exact same preference pairs would disentangle these factors.
  • The paper leaves implicit that weights are computed only for the chosen response; extending the scheme to weight rejected-response sentences could further suppress partially correct but overall rejected answers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ASPO, an adaptive sentence-level preference optimization method for multimodal large language models. The method computes per-sentence adaptive weights from CLIP image-text similarity and textual perplexity, then claims to reformulate the DPO implicit reward margin by reweighting each sentence in the chosen response and normalizing the result. The authors report consistent improvements over DPO and other preference optimization baselines across LLaVA and InstructBLIP models on multiple multimodal benchmarks. However, the central derivation in Section 3.2.1, Equation (12), is algebraically identical to the standard DPO margin: the adaptive weights cancel exactly under the definitions given in the same section. Consequently, the proposed objective reduces to the ordinary DPO loss, and the reported experimental gains cannot be produced by the described mechanism.

Significance. If the sentence-level adaptive mechanism were real, the paper would offer a parameter-free fine-grained alternative to DPO and its broad experimental evaluation would be valuable. The manuscript is transparent about experimental settings, includes comparisons with many existing methods, and reports ablations over reward components and granularity levels. These are genuine strengths. However, the load-bearing mathematical claim is false: Equation (12) is a definitional no-op relative to DPO, so the paper's central contribution is void. The experimental results, which show large differences between ASPO and DPO under identical configurations, are inconsistent with the stated loss and suggest that the actual implementation differs from the written equations. The central claim cannot be repaired within the manuscript's scope because the proposed objective is equivalent to the baseline by construction.

major comments (3)
  1. [§3.2.1, Eq. (12)] Equation (12) is algebraically identical to the standard DPO margin of Equation (11). The text states that Rc and R*_c are 'the original and the reweighted sum of the sentence-level implicit rewards of chosen response, respectively'; that is, Rc = sum_i beta log(pi_theta(s_i^c|x)/pi_ref(s_i^c|x)) and R*_c = sum_i beta(1+w_i) log(pi_theta(s_i^c|x)/pi_ref(s_i^c|x)). Substituting into Equation (12) gives M* = (Rc/R*_c) * R*_c - beta log(pi_theta(y_r|x)/pi_ref(y_r|x)) = Rc - beta log(pi_theta(y_r|x)/pi_ref(y_r|x)), which is exactly the chosen-response term of Equation (11). The adaptive weights w_i cancel identically for every theta, including when w_i depends on the model's own predictions. Therefore LASPO in Equation (13) is the ordinary DPO loss, and the claimed sentence-level adaptive weighting has no effect on the objective.
  2. [§4.1 and §4.2, Tables 1 and 5] The experimental results contradict the equations. The paper states that 'ASPO and standard DPO share the same configuration above' (Section 4.1), and if Equation (12) equals Equation (11), the two methods must produce identical training losses and therefore identical models. Yet Table 1 reports, for example, LLaVA-1.5-7B average scores of 63.12 for +DPO and 65.16 for +ASPO, and Table 5 reports +DPO at 63.12 versus +ASPO-S+P at 65.16. These differences cannot arise from the stated objective. The paper must provide the actual objective used in training or explain the discrepancy; as written, the reported gains indicate that the implementation differs from the described method in some unstated way.
  3. [§3.2.1, sentence-level granularity claim] The claim that ASPO 'uses sentences as the fundamental units' for fine-grained optimization is unsupported because the weights wi vanish in the final margin. The special-case remark that 'when a response contains only one sentence, the adaptive weight wi is normalized to 0 and our method degenerates into the standard DPO' is misleading: the cancellation in Equation (12) holds for every response, not only single-sentence responses. The paper's central narrative—that fine-grained sentence-level rewards drive the improvements—therefore has no basis in the derived objective.
minor comments (5)
  1. [§3.2.1] The quantities Rc and R*_c are described in words but never written as explicit equations; this ambiguity obscures the cancellation and should be fixed by defining them as formal sums in the text.
  2. [§4.1] There are typographical errors such as 'SeVa pipline' (should be 'pipeline') and inconsistent model naming ('LLaV A-1.5' versus 'LLaVA-1.5'); these should be standardized.
  3. [§1 and §3.2.1] The paper claims the method works 'without additional models or parameters,' but Sentence-level CLIP similarity requires a CLIP model during training; the authors should clarify whether this external model is frozen and whether it counts as an additional model.
  4. [Table 1] The column header 'LLaV AW' is unclear; it should be written as 'LLaVA-Bench (in the wild)' to match the benchmark name. Also, the SHR column is marked as a lower-is-better metric, but baseline values are missing for many models in the upper part of the table.
  5. [§4.3] The ablation in Table 6 reports 'ASPO Response' and 'ASPO Token' results that are far below the base model; given the mathematical equivalence in Equation (12), the paper should explain what these row labels mean operationally, because they cannot correspond to different settings of the written objective.

Circularity Check

1 steps flagged · score 10.0 of 10

Equation (12)'s normalization ratio cancels the adaptive sentence weights, making the ASPO objective algebraically identical to standard DPO.

  1. self definitional [Section 3.2.1, Equation (12), with Rc and R*c definitions in the same paragraph]
    "M∗ = Rc/R∗c Σ_{i=1}^K β(1 + wi) log(πθ(s_i^c | x)/πref(s_i^c | x)) − β log(πθ(yr | x)/πref(yr | x)) (12) where Rc and R∗c represent the original and the reweighted sum of the sentence-level implicit rewards of chosen response, respectively."

    By the definitions in §3.2.1, Rc = Σ_i β log(πθ(s_i^c|x)/πref(s_i^c|x)) and R∗c = Σ_i β(1+wi) log(πθ(s_i^c|x)/πref(s_i^c|x)). Substituting into Eq. (12) gives M∗ = (Rc/R∗c)·R∗c − β log(πθ(yr|x)/πref(yr|x)) = Rc − β log(πθ(yr|x)/πref(yr|x)), which is exactly the standard DPO margin M in Eq. (11). The adaptive weights wi cancel identically for every θ, including when wi depends on model predictions. Consequently Eq. (13), LASPO = −E_D log σ(M∗), is the ordinary DPO loss. The claimed adaptive sentence-level mechanism is therefore a definitional no-op: the proposed objective reduces by construction to the baseline it claims to improve.

full rationale

The paper's central claim is that ASPO introduces an adaptive sentence-level reward margin M∗ that modulates per-sentence reward strength and differs from standard DPO. However, the paper's own Equation (12) defines Rc and R∗c as the original and reweighted sums of the sentence-level implicit rewards. With those definitions, the prefactor Rc/R∗c precisely cancels the reweighted sum, leaving the original DPO margin M from Equation (11). Thus LASPO = LDPO exactly, and the adaptive weights wi have no effect on the optimization objective. This is not a case of a minor self-citation or a missing external benchmark; it is the core contribution reducing to the baseline by algebraic identity. The extensive experiments cannot evidence a mechanism that the equations show is absent from the training loss. No alternative definition of R∗c is supplied, and no external machine-checked or code-verified derivation is cited. The circularity is therefore maximal: the 'adaptive reward' is equivalent to its input (the standard DPO margin) by definition.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central method depends on standard DPO machinery plus two heuristic reward signals (CLIP similarity, perplexity) combined with a hand-tuned alpha. The invented normalization ratio Rc/R*_c is presented as a length control but is the exact source of the cancellation that makes the method identical to DPO. No fundamentally new entities are introduced.

free parameters (2)
  • alpha (metric-weighting factor) = 0.5
    Weights the combination of image-text similarity and textual perplexity in Equation (10); chosen via ablation in Figure 3 on the evaluation benchmarks, i.e., tuned to maximize reported scores.
  • beta (DPO temperature) = 0.1
    Scale of the implicit reward in Equation (5) and Equation (12); set by hand following prior DPO practice, not fitted to data but a free hyperparameter.
assumptions (5)
  • standard math Bradley-Terry preference model describes human preferences (Equation 1)
    Standard assumption inherited from RLHF and DPO literature.
  • standard math DPO closed-form reward expression r*(x,y) = beta log(pi*/pi_ref) + Delta (Equation 4)
    Imported from Rafailov et al. 2024.
  • domain assumption CLIP cosine similarity between a sentence and the whole image is a valid proxy for sentence correctness (Equation 6)
    No evidence provided that global image-level similarity aligns with fine-grained factual correctness of a sentence.
  • domain assumption Lower model perplexity implies higher probability of correctness (Section 3.2.1)
    Cited to prior uncertainty work (Kuhn et al. 2023), but this is a heuristic, not a guarantee, especially for hallucinated fluent text.
  • ad hoc to paper Min-max normalization of similarity and perplexity across sentences yields comparable adaptive weights (Equation 7)
    Introduced specifically for this method; the normalization is not derived from any principle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning." pith.science (2026). https://pith.science/paper/BS2IDNFC

@misc{pith2026250519100,
  author       = {Pith},
  title        = {Pith review of: ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BS2IDNFC}},
  note         = {Machine review of arXiv:2505.19100}
}
read the original abstract

Direct Preference Optimization (DPO) has gained significant attention for its simplicity and computational efficiency in aligning large language models (LLMs). Recent advancements have extended DPO to multimodal scenarios, achieving strong performance. However, traditional DPO relies on binary preference optimization, rewarding or penalizing entire responses without considering fine-grained segment correctness, leading to suboptimal solutions. The root of this issue lies in the absence of fine-grained supervision during the optimization process. To address this, we propose Adaptive Sentence-level Preference Optimization (ASPO), which evaluates individual sentences for more precise preference optimization. By dynamically calculating adaptive rewards at the sentence level based on model predictions, ASPO enhances response content assessment without additional models or parameters. This significantly improves the alignment of multimodal features. Extensive experiments show that ASPO substantially enhances the overall performance of multimodal models.

Figures

Figures reproduced from arXiv: 2505.19100 by the authors.

Figure 1
Figure 1. Comparison between DPO and our ASPO. Unlike the traditional DPO algorithm that calculates implicit [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Data collection pipeline. Preference data. We sample about 20K instruc￾tions from LLaVA-Instruct-150K (Liu et al., 2024b) and go through SeVa pipline (Zhu et al., 2024) to produce the preference dataset. The data collection pipeline is shown in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Ablation study of the metric-weighting α. their credibility becomes higher and higher, and the adaptive rewards calculated based on this be￾come much more accurate, so the improvement in model performance is much larger. We show the comparison results of our method with other preference optimization-based methods in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 25 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 1

    arXiv 2023

  5. [5]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

    arXiv 2023

  6. [6]

    Jiuhai Chen, Lichang Chen, Chen Zhu, and Tianyi Zhou. 2023 a . How many demonstrations do you need for in-context learning? arXiv preprint arXiv:2303.08119

    arXiv 2023

  7. [7]

    Jiuhai Chen and Jonas Mueller. 2024. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. In Annual Meeting of the Association for Computational Linguistics , pages 5186--5200

    work page 2024

  8. [8]

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023 b . Shikra: Unleashing multimodal llm's referential dialogue magic. arXiv preprint arXiv:2306.15195

    arXiv 2023

Show all 61 references
  1. [9]

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. 2024. Halc: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425

    2024 arXiv

  2. [10]

    Chenhang Cui, An Zhang, Yiyang Zhou, Zhaorun Chen, Gelei Deng, Huaxiu Yao, and Tat-Seng Chua. 2024. Fine-grained verifiers: Preference modeling as next-token prediction in vision-language alignment. arXiv preprint arXiv:2410.14148

    2024 arXiv

  3. [11]

    Wenliang Dai, Junnan Li, D Li, AMH Tiong, J Zhao, W Wang, B Li, P Fung, and S Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500

    2023 arXiv

  4. [12]

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306

    2024 arXiv

  5. [13]

    Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. 2024. Multi-modal hallucination control by visual information grounding. In Conference on Computer Vision and Pattern Recognitio...

    2024

  6. [14]

    Deqing Fu, Tong Xiao, Rui Wang, Wang Zhu, Pengchuan Zhang, Guan Pang, Robin Jia, and Lawrence Chen. 2024. Tldr: Token-level detective reward model for large vision language models. arXiv preprint arXiv:2410.04734

    2024 arXiv

  7. [15]

    Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifei Liu. 2024. Regiongpt: Towards region understanding vision language model. In Conference on Computer Vision and Pattern Recognition , pages 13796--13806

    2024

  8. [16]

    Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. In Conference on Empirical Methods in Natural Language Processing, pages 11170--11189

    2024

  9. [17]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  10. [18]

    Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. 2023. Language is not all you need: Aligning perception with language models. Neural Information Processing Systems , 36:72096--72109

    2023

  11. [19]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Conference on Computer Vision and Pattern Recognition , pages 6700--6709

    2019

  12. [20]

    Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. 2024. Hallucination augmented contrastive learning for multimodal large language model. In Conference on Computer Vision and Pattern Recognition , pages 2...

    2024

  13. [21]

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. Segment anything. In International Conference on Computer Vision , pages 4015--4026

    2023

  14. [22]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664

    2023 arXiv

  15. [23]

    Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629

    2024 arXiv

  16. [24]

    Hugo Lauren c on, Lucile Saulnier, L \'e o Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. 2024. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Neural Information Pr...

    2024

  17. [25]

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. 2024. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. In International Conference ...

    2024

  18. [26]

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023 a . Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125

    2023 arXiv

  19. [27]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2024. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Neural Information Processing Systems , 36

    2024

  20. [28]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023 b . Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730--19742. PMLR

    2023

  21. [29]

    Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. 2023 c . Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312.10665

    2023 arXiv

  22. [30]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023 d . Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355

    2023 arXiv

  23. [31]

    Weibin Liao, Xu Chu, and Yasha Wang. 2024. Tpo: Aligning large language models with multi-branch & multi-step preference trees. arXiv preprint arXiv:2410.12854

    2024

  24. [32]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 a . Improved baselines with visual instruction tuning. In Conference on Computer Vision and Pattern Recognition , pages 26296--26306

    2024

  25. [33]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024 b . Visual instruction tuning. Neural Information Processing Systems , 36

    2024

  26. [34]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2025. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision , pages 216--233. Springer

    2025

  27. [35]

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Neural Information Processing Systems , 35:2507--2521

    2022

  28. [36]

    Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. 2025. Clip-dpo: Vision-language models as a source of preference for fixing hallucinations in lvlms. In European Conference on Computer Vision , pages 395--413. Springer

    2025

  29. [37]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learni...

    2021

  30. [38]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Neural Information Processing Systems , 36

    2024

  31. [39]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    2017 arXiv

  32. [40]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2023. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525

    2023 arXiv

  33. [41]

    Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Daniel Cox, Yiming Yang, and Chuang Gan. 2024. Salmon: Self-alignment with instructable reward models. In International Conference on Learning Representations

    2024

  34. [42]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  35. [43]

    Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2024 a . mdpo: Conditional preference optimization for multimodal large language models. arXiv preprint arXiv:2406.11839

    2024 arXiv

  36. [44]

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. 2024 b . Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442

    2024 arXiv

  37. [45]

    Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, et al. 2024 c . Enhancing visual-language modality alignment in large vision language models via self-improvement. arXiv preprint arXiv:2405.15973

    2024 arXiv

  38. [46]

    Yuanhao Wang, Qinghua Liu, and Chi Jin. 2023. Is rlhf more difficult than standard rl? a theoretical perspective. Neural Information Processing Systems , 36:76006--76032

    2023

  39. [47]

    Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. 2024. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.00451

    2024 arXiv

  40. [48]

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. 2024. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Conference on Computer Vision and Pattern Recognition , pages 13040--13051

    2024

  41. [49]

    Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Wontae Nam, Daejin Jo, Kyoung-Woon On, Mark A Hasegawa-Johnson, Sungwoong Kim, and Chang D Yoo. 2024. Tlcr: Token-level continuous reward for fine-grained reinforcement learning from human feedback. arXiv preprint arX...

    2024 arXiv

  42. [50]

    Tianyu Yu, Jinyi Hu, Yuan Yao, Haoye Zhang, Yue Zhao, Chongyi Wang, Shan Wang, Yinxv Pan, Jiao Xue, Dahai Li, et al. 2023 a . Reformulating vision-language foundation models and datasets towards universal multimodal assistants. arXiv preprint arXiv:2310.00653

    2023 arXiv

  43. [51]

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. 2024. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Conference on Computer Vision and Patt...

    2024

  44. [52]

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023 b . Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490

    2023 arXiv

  45. [53]

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024 a . Self-rewarding language models. arXiv preprint arXiv:2401.10020

    2024 arXiv

  46. [54]

    Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. 2024 b . Osprey: Pixel understanding with visual instruction tuning. In Conference on Computer Vision and Pattern Recognition , pages 28202--28211

    2024

  47. [55]

    Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, and Yiming Yang. 2024. Improve vision language model chain-of-thought reasoning. arXiv preprint arXiv:2410.16198

    2024 arXiv

  48. [56]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  49. [57]

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. 2023. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839

    2023 arXiv

  50. [58]

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. 2024 a . Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411

    2024 arXiv

  51. [59]

    Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. 2024 b . Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405.14622

    2024 arXiv

  52. [60]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

    2023 arXiv

  53. [61]

    Ke Zhu, Liang Zhao, Zheng Ge, and Xiangyu Zhang. 2024. Self-supervised visual preference alignment. In International conference on Multimedia , pages 291--300

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.