Pith. sign in

REVIEW 4 major objections 7 minor 42 references

Groc-PO applies preference optimization at three grounded stages — object grounding, contextual grounding, grounded reasoning — and outperforms final-answer-only DPO on hallucination and complex-reasoning benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 03:56 UTC pith:NBJN2MEO

load-bearing objection The paper's central claim is plausible and well-supported on hallucination benchmarks, but the complex-reasoning headline rests on an underspecified DPO baseline; worth a serious referee. the 4 major comments →

arxiv 2607.13712 v1 pith:NBJN2MEO submitted 2026-07-15 cs.CV cs.AIcs.CLcs.MM

Groc-PO: Grounded Context Preference Optimization for Truthful Multimodal LLMs

Zhixiao Zheng , Zheren Fu , Zhiyuan Yao , Chunxiao Liu , Dongming Zhang , Zhendong Mao This is my paper
classification cs.CV cs.AIcs.CLcs.MM
keywords preferencegroundedreasoningcontextgroundingoptimizationstagesgroc-po
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large multimodal models — AI systems that answer questions about images — often invent content that is not in the picture. A standard fix, direct preference optimization (DPO), trains the model to prefer a human-approved answer over a rejected one, but only grades the final answer. The authors' observation is that mistakes often start earlier: a model that misidentifies an object in step one tends to reason wrongly in step three, so feedback at the final answer is indirect.

Groc-PO splits reasoning into three supervised stages. Stage 1 lists entities and attributes in the image. Stage 2, built by reusing the public RLHF-V dataset, describes relationships and context. Stage 3 asks a complex question that requires the earlier context. Each stage has preference pairs: chosen and rejected responses. The chosen answers for stages 1 and 3 are drafted by GPT-4o, revised through self-correction, and spot-checked by human reviewers; rejected answers come from the target model itself or from rule-based corruption.

Training uses the standard DPO loss plus two multiplicative weights: a stage weight that grows with later, harder stages, and a hardness weight that focuses on pairs the model still confuses. On LLaVA-1.5 7B and 13B, and Qwen2.5-VL, the authors report lower hallucination rates on AMBER and MM-Hal and large gains on complex reasoning subsets — for example 82.5 versus 56.9 for DPO on LLaVA-complex reasoning. A 'flattened' version without stage history scores much worse, supporting the structured-context design. Caveats: numbers are single runs with no error bars, the two loss hyperparameters were tuned on the headline benchmark, and neither dataset nor code is released.

Core claim

Groc-PO, built on the new three-stage GCPD dataset, introduces explicit preference supervision over grounded pre-final stages (object grounding, contextual grounding, grounded reasoning) and, with a stage-aware and hardness-aware DPO loss, improves MLLM faithfulness and complex reasoning, mitigating cross-stage error propagation. Headline numbers: on LLaVA-v1.5-7B, MM-Hal hallucination rate drops to 47.0 (from 61.4 base / 58.3 DPO); LLaVA-complex-reasoning rises to 82.5 vs 56.9 for DPO (Table 3); the same pattern holds at 13B. The 'Flattened DPO' control (same 17k pairs without stage context) scores 2.17 vs 2.76 MM-Hal, attributing the gain to structured grounded context.

Load-bearing premise

The framework's rationale rests on the causal premise — established only by the Fig. 1b 'controlled explorations' whose methodology is never described (how grounding-stage errors were injected, which model generations were measured, over how many samples) — that final-answer failures originate in early grounding stages and that correcting those stages transfers to final answers. A distinct and concrete fragility: Stage-2 preference pairs are transplanted from RLHF-V (§3.3.1), where each pair was collected as an independent single-turn human correction, into the cumulative multi-turn prompt {I, Q1, A+_s1, Q2} (§3.2). If a human-validated S2 chosen response contradicts the injected S1 grounding (e.g., S1 lists different entities than S2 describes), the 'grounded context' supervision trains the model on internally inconsistent context. The paper reports no consistency check between injected S1 answers and reused S2 pairs. If either premise fails, the reported gains could stem from multi-t

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Groc-PO (Grounded Context Preference Optimization), a preference-optimization framework for multimodal LLMs that adds explicit supervision over three grounded pre-final stages: object grounding, contextual grounding, and grounded reasoning. The authors construct the GCPD dataset (17,199 preference pairs from 5,733 images) by combining RLHF-V data with teacher-generated and model-centric samples, then train LLaVA-v1.5-7B/13B and Qwen2.5-VL-7B with a DPO loss augmented by a stage-aware weight and a hardness-aware focusing weight. Experiments report improvements on hallucination benchmarks (MM-Hal, AMBER) and complex-reasoning benchmarks (LLaVA-Bench-complex, MME-reason, MM-MT), with ablations of the loss components, stage contributions, context depth, and a flattened-context control.

Significance. If the results hold, the paper makes a useful contribution to MLLM alignment: it provides a concrete way to move from final-answer-level preference supervision to stage-wise grounded supervision, along with a reusable dataset and a plausible mechanism (reducing cross-stage error propagation). The paper includes several strengths: evaluation on external benchmarks not used in training, a same-data 'Flattened DPO' control that directly tests the role of structured context, and component ablations of the adaptive loss. The central claims are therefore not circular. However, the missing specification of the Table 3 DPO baseline and the unexamined consistency of transplanted S2 pairs are substantive gaps that need to be addressed before the mechanism can be taken as established.

major comments (4)
  1. [Table 3 / §4.2] The paper's headline complex-reasoning gains compare 'DPO' against 'Groc-PO' (e.g., LLaVA-complex-reasoning 56.9 vs 82.5), but the manuscript never states what preference data or loss the 'DPO' baseline in Table 3 was trained on. In Table 1, '+ DPO' appears among baselines typically trained on RLHF-V data, while in Table 4, 'DPO Loss' is standard DPO on the GCPD data. If Table 3 uses a different preference set, the reported gains may reflect dataset content or size rather than stage-structured supervision. Because the central claim is that grounded context supervision improves complex reasoning, please specify the DPO baseline precisely and, ideally, report the same-data 'Flattened DPO' control (Table 5) on the complex-reasoning benchmarks (LLaVA-complex, MME-reason, MM-MT).
  2. [§1, Fig. 1b] The motivation for the paper rests on the controlled explorations in Fig. 1b, which claim that introducing errors into 0, 1, or 2 grounding stages progressively lowers final reasoning accuracy. The methodology behind this figure is never described: how were grounding-stage errors injected, which model and decoding setting were used, how many samples were evaluated, and what metric was measured? Without this detail, the causal premise that final-answer errors can be traced to early grounding stages is not established. Please provide a full description (in §4 or an appendix), including the error-injection protocol and sample size.
  3. [§3.3.1 / §3.2] Stage-2 preference pairs are transplanted from RLHF-V, where each pair was collected as an independent single-turn human correction, into the cumulative multi-turn prompt {I, Q1, A+_s1, Q2} in GCPD. If a human-validated S2 chosen response contradicts the injected S1 chosen grounding (e.g., S1 lists different entities than S2 describes), then the 'grounded context' is internally inconsistent and the model is trained on contradictory context. The paper reports no consistency check between A+_s1 and the reused S2 pairs. This is particularly relevant because S1 chosen responses are teacher-generated and only 10% were human-audited. Please report a consistency/filtering check at the S1–S2 interface and state whether any pairs were removed or revised.
  4. [§4, Tables 1–3] All reported numbers appear to come from a single training run with no error bars, multiple seeds, or significance tests. Given the large claimed improvements and the sensitivity of the loss to the hyperparameters α and η (Table 4), single-run numbers are insufficient to rule out training variance, especially for hallucination metrics. Please report at least 2–3 seeds (or, if computationally prohibitive, a bootstrap over evaluation items) for the main tables.
minor comments (7)
  1. [Title/header] The header repeats 'GROunded Context Preference Optimization' but the title reads 'Grounded Context Preference Optimization'; please harmonize.
  2. [§3.4, Eq. (4)] The notation λ_r uses r both for the stage index and for the implicit reward r_i(θ). Later text refers to λ_t. Please use a distinct symbol (e.g., λ_s) for the stage weight.
  3. [Fig. 2] The 'Implicit Reward' formula in the figure is garbled by LaTeX artifacts (e.g., '�'). Replace with the clean equation from §3.1.
  4. [Fig. 1b] The y-axis label has an unexplained '(↑)' and the figure title is incomplete. Clarify the axis and the meaning of the upward arrow.
  5. [Table 3] The table does not state the base model (7B or 13B) or whether the DPO baseline is the same as in Table 1. Also, the column header 'LLaVA-complex reason' is inconsistent with the text's 'LLaVA-Bench-complex reasoning'. Specify the backbone and evaluation protocol.
  6. [§3.3.4] The text mentions revisions of 'severely problematic samples (primarily in R3)' but the stages are named S1/S2/S3. Use consistent notation.
  7. [Fig. 3a] The label 'Not-Sampling' should be 'No Model-Centric Sampling' or 'Teacher-only' to match the description.

Circularity Check

0 steps flagged

No circularity: Groc-PO is assessed on external benchmarks and its structured-context claim is tested with a same-data flattened control.

full rationale

The paper's derivation chain is empirical rather than formal: GCPD is constructed from RLHF-V images plus teacher-generated and model-centric preference pairs (§3.2–3.3), and Groc-PO's claimed gains are evaluated on AMBER, MM-Hal, LLaVA-Bench, SEED, MME, and MM-MT (§4.1), benchmarks not used to build the preference pairs. The same-data control 'Flattened DPO' (§4.4.1, Table 5) removes only the cumulative stage context while keeping the 17k pairs, so the structured-context comparison is not self-referential. The loss reweighting in §3.4 (λ_r = 1+α(r−1), γ_i = (1−σ(r_i))^η) is defined on the model's own implicit reward and is ablated against an unweighted DPO loss in Table 4; the gain is an empirical outcome, not an algebraic identity. The only author-overlapping citation is the Qwen2.5-VL technical report [4], used merely to name the base model in scalability experiments; it is not load-bearing for the central claim. One caveat: α and η are selected using MM-Hal in Table 4, which weakens the independence of the headline MM-Hal number, but this is a tuning/selection concern rather than a definitional reduction. Reporting gaps exist—the Fig. 1b 'controlled explorations' protocol is never specified, and Table 3 does not state what preference data its DPO baseline was trained on—but these are evidence/reporting weaknesses, not cases in which a predicted result equals its input by construction. No equation or fitted variable is renamed as a prediction.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 1 invented entities

The paper contributes a training method and dataset, not a derivation, so its ledger is dominated by design assumptions rather than free parameters. Two hyperparameters (α, η) are fitted on MM-Hal, which is also the headline evaluation metric; the sensitivity to α is non-monotonic. The three-stage decomposition, the monotonic stage weighting, the focal-style hardness weighting, the reuse of RLHF-V pairs in a new cumulative format, and reliance on GPT-4o self-correction with partial human audit are domain assumptions with limited independent verification. No new physical entities are postulated; GCPD is a constructed dataset whose quality rests on the paper's own pipeline.

free parameters (4)
  • α (stage-aware weight growth rate) = 0.25
    Growth rate in λ_r = 1 + α(r−1) (Eq. 4); selected from {0, 0.25, 0.5} by MM-Hal score (Table 4B), where α=0.5 degrades the score from 2.76 to 2.40. Tuned on the headline evaluation benchmark.
  • η (hardness-aware focusing exponent) = 2
    Exponent in γ_i = (1 − σ(r_i))^η (Eq. 5); selected from {0, 1, 2} by MM-Hal score (Table 4C). Tuned on the headline evaluation benchmark.
  • β (DPO temperature) = not reported
    Inherited from DPO (Eq. 1). Its value sets the implicit reward scale but is never stated, so the loss magnitude is not reproducible as specified.
  • Self-correction rejection thresholds = unspecified
    Section 3.3.2: the teacher rewrites chosen responses when hallucinations are found or the 'average score is too low'. The scoring rubric and thresholds are not given, so the chosen-response quality bar is not reproducible.
axioms (6)
  • standard math Bradley-Terry latent reward model with the DPO implicit reward identity r*(x,y) = β(log π_θ(y|x) − log π_ref(y|x))
    Assumed from Rafailov et al. [25] in §3.1 (Eq. 1 and surrounding text); the Groc-PO loss is only a valid preference objective under this model.
  • domain assumption The three-stage decomposition (object grounding → contextual grounding → grounded reasoning) captures the actual error-propagation structure of MLLM reasoning
    Core design of GCPD (§3.2, Fig. 2); supported only by the under-described Fig. 1b experiment. If the stage structure does not match how MLLMs actually reason, the stage supervision is mis-targeted.
  • ad hoc to paper Later stages carry more learning value, so stage weights should increase monotonically
    Eq. 4: λ_r = 1 + α(r−1). A chosen schedule, not derived; the sensitivity analysis (Table 4B) shows the optimum is narrow.
  • ad hoc to paper Samples with (1 − σ(r_i)) near 1 are 'hard' and deserve more training weight
    Eq. 5, a focal-loss-style weighting. Plausible but only evaluated through the MM-Hal ablation (Table 4C), not independently validated.
  • domain assumption RLHF-V single-turn preference pairs remain valid when inserted as Stage-2 inside a cumulative multi-turn prompt
    §3.3.1 adopts RLHF-V pairs wholesale for S2. No consistency check is reported between the injected S1 chosen context and the reused S2 chosen/rejected pairs; if they conflict, the 'grounded' signal is inconsistent.
  • domain assumption GPT-4o outputs, after self-correction and a ~12% human audit, are factually reliable enough to serve as chosen responses for S1 and S3
    §3.3.2 and §3.3.4. Only ~12% of chosen samples were audited and ~2% rewritten, meaning the remaining ~88% rest on teacher self-critique with no independent factual check.
invented entities (1)
  • GCPD (Grounded Context Preference Dataset) no independent evidence
    purpose: Provides 17,199 three-stage preference pairs over 5,733 images to enable stage-specific DPO supervision
    Not released (no repository link); its quality rests on the authors' own verification pipeline (teacher self-correction + 12% human audit). It has no external falsifiable handle in this preprint; a public release would change this.

pith-pipeline@v1.3.0-alltime-deepseek · 14094 in / 19517 out tokens · 184652 ms · 2026-08-02T03:56:57.679242+00:00 · methodology

0 comments
read the original abstract

Despite the rapid progress of Multimodal Large Language Models (MLLMs), they still suffer from untruthfulness issues, such as visual hallucinations, content fabrication, and unfaithful reasoning, which substantially undermine their faithfulness and practical utility. Alignment methods based on human preference, such as Direct Preference Optimization (DPO), have been widely adopted to address these issues. However, multimodal reasoning errors often propagate across stages, and final-answer errors can often be traced to mistakes in early grounding stages, yet standard DPO typically applies preference optimization at the final-answer level. This credit-assignment challenge means that supervision for early grounding stages is indirect rather than stage-specific, making it difficult to suppress error propagation arising from grounding drift and context inconsistency. To address this, we propose Grounded Context Preference Optimization (Groc-PO), a grounded preference optimization framework for MLLMs. We further construct the Grounded Context Preference Dataset (GCPD), organizing multi-stage preference samples around three stages of Object Grounding, Contextual Grounding, and Grounded Reasoning, to capture the formation, integration, and utilization of grounded context. By introducing more explicit preference supervision over multiple grounded stages, Groc-PO strengthens context-dependent reasoning and mitigates cross-stage error propagation. Extensive experiments show that, compared with standard DPO and other strong baselines, Groc-PO achieves improved performance in hallucination mitigation, faithful reasoning, and overall reliability, supporting the value of more explicit grounded supervision for trustworthy multimodal reasoning.

Figures

Figures reproduced from arXiv: 2607.13712 by Chunxiao Liu, Dongming Zhang, Zhendong Mao, Zheren Fu, Zhixiao Zheng, Zhiyuan Yao.

Figure 1
Figure 1. Figure 1: Motivating example of error propagation across [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our Grounded Context Preference Optimization (Groc-PO) framework, including GCPD dataset construc [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ablations and Analysis. (a) Effectiveness of Model-Centric Sampling or not. (b) Contribution of individual stages: [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Token-level log-probability (log-prob) comparison [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 21 linked inside Pith

  1. [1]

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. 2024. Pixtral 12B.arXiv preprint arXiv:2410.07073 (2024)

    Pith/arXiv arXiv 2024

  2. [2]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

  3. [3]

    Elmira Amirloo, Jean-Philippe Fauconnier, Christoph Roesmann, Christian Kerl, Rinu Boney, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, et al. 2024. Understanding alignment in multimodal llms: A comprehensive study. arXiv preprint arXiv:2407.02477(2024)

    Pith/arXiv arXiv 2024

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Rep...

    Pith/arXiv arXiv 2025

  5. [5]

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of multimodal large language models: A survey.arXiv preprint arXiv:2404.18930(2024)

    Pith/arXiv arXiv 2024

  6. [6]

    Assaf Ben-Kish, Moran Yanuka, Morris Alper, Raja Giryes, and Hadar Averbuch- Elor. 2023. Mocha: Multi-objective reinforcement mitigating caption hallucina- tions.arXiv preprint arXiv:2312.036312 (2023)

    Pith/arXiv arXiv 2023

  7. [7]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences.Advances in neural information processing systems30 (2017)

    2017

  8. [8]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. 2023. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models.arXiv preprint arXiv:2306.13394(2023)

    Pith/arXiv arXiv 2023

  9. [9]

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. 2024. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2024

  10. [10]

    Anisha Gunjal, Jihan Yin, and Erhan Bas. 2024. Detecting and preventing halluci- nations in large vision language models. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 18135–18143

    2024

  11. [11]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models.ICLR1, 2 (2022), 3

    2022

  12. [12]

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hal- lucination in multi-modal large language models via over-trust penalty and retrospection-allocation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13418–13427

    2024

  13. [13]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024)

    Pith/arXiv arXiv 2024

  14. [14]

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13872–13882

    2024

  15. [15]

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125(2023)

    Pith/arXiv arXiv 2023

  16. [16]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen

  17. [17]

    Xiwen Liang, Min Lin, Weiqi Ruan, Rongtao Xu, Yuecheng Liu, Jiaqi Chen, Bingqian Lin, Yuzheng Zhuang, and Xiaodan Liang. 2025. Structured preference optimization for vision-language long-horizon task planning.arXiv preprint arXiv:2502.20742(2025)

    arXiv 2025

  18. [18]

    Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. 2024. A Survey of Multimodel Large Language Models. InProceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering. 405–409

    2024

  19. [19]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 26296–26306. Groc-PO: GROunded Context Preference Optimization for Truthful Multimodal LLMs Conference’17, July 2017, Washington, DC, USA

    2024

  20. [20]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruc- tion Tuning

    2023

  21. [21]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning.Advances in neural information processing systems36 (2023), 34892–34916

    2023

  22. [22]

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024. A survey on hallucination in large vision-language models.arXiv preprint arXiv:2402.00253(2024)

    Pith/arXiv arXiv 2024

  23. [23]

    Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101(2017)

    Pith/arXiv arXiv 2017

  24. [24]

    Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. 2024. Strengthening multimodal large language model with bootstrapped preference optimization. InEuropean Conference on Computer Vision. Springer, 382–398

    2024

  25. [25]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems36 (2023), 53728–53741

    2023

  26. [26]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  27. [27]

    Guohao Sun, Can Qin, Yihao Feng, Zeyuan Chen, Ran Xu, Sohail Dianat, Majid Rabbani, Raghuveer Rao, and Zhiqiang Tao. 2025. Structured Policy Optimiza- tion: Enhance Large Vision-Language Model via Self-referenced Dialogue. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 741–751

    2025

  28. [28]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2023. Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525(2023)

    Pith/arXiv arXiv 2023

  29. [29]

    Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zujun Ma, and Chao Zhang. 2024. Enhancing multimodal LLM for detailed and accurate video captioning using multi-round preference optimization.arXiv preprint arXiv:2410.06682(2024)

    Pith/arXiv arXiv 2024

  30. [30]

    Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2024. mdpo: Conditional preference optimization for multimodal large language models.arXiv preprint arXiv:2406.11839(2024)

    Pith/arXiv arXiv 2024

  31. [31]

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. 2023. An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation.arXiv preprint arXiv:2311.07397(2023)

    Pith/arXiv arXiv 2023

  32. [32]

    Wenyi Xiao, Ziwei Huang, Leilei Gan, Wanggui He, Haoyuan Li, Zhelun Yu, Fangxun Shu, Hao Jiang, and Linchao Zhu. 2024. Detecting and mitigating hallucination in large vision language models via fine-grained ai feedback.arXiv preprint arXiv:2404.14233(2024)

    Pith/arXiv arXiv 2024

  33. [33]

    Wenyi Xiao, Ziwei Huang, Leilei Gan, Wanggui He, Haoyuan Li, Zhelun Yu, Fangxun Shu, Hao Jiang, and Linchao Zhu. 2025. Detecting and mitigating hallucination in large vision language models via fine-grained ai feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 25543–25551

    2025

  34. [34]

    Yuxi Xie, Guanzhen Li, Xiao Xu, and Min-Yen Kan. 2024. V-dpo: Mitigating hallucination in large vision language models via vision-guided direct preference optimization.arXiv preprint arXiv:2411.02712(2024)

    Pith/arXiv arXiv 2024

  35. [35]

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al . 2024. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13807–13816

    2024

  36. [36]

    Mengxi Zhang, Wenhao Wu, Yu Lu, Yuxin Song, Kang Rong, Huanjin Yao, Jianbo Zhao, Fanglong Liu, Haocheng Feng, Jingdong Wang, et al . 2024. Automated multi-level preference for mllms.Advances in Neural Information Processing Systems37 (2024), 26171–26194

    2024

  37. [37]

    Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, et al . 2025. Mm-rlhf: The next step forward in multimodal llm alignment.arXiv preprint arXiv:2502.10391 (2025)

    Pith/arXiv arXiv 2025

  38. [38]

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. 2024. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411(2024)

    Pith/arXiv arXiv 2024

  39. [39]

    Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chen- hang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. 2024. Calibrated self-rewarding vision language models.arXiv preprint arXiv:2405.14622(2024)

    Pith/arXiv arXiv 2024

  40. [2017]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347 (2017)

    Pith/arXiv arXiv 2017

  41. [2022]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems35 (2022), 23716–23736

    2022

  42. [2023]

    Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355(2023)

    Pith/arXiv arXiv 2023