Pith. sign in

REVIEW 4 major objections 7 minor 4 cited by

Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For autoregressive image generation, DPO beats GRPO in-domain but GRPO generalizes better out-of-domain; reward models with stronger intrinsic generalization yield policies that generalize better.

desk verdict Useful DPO vs GRPO comparison for autoregressive image generation; main findings plausible, but the reward-model generalization claim is undercut by a same-benchmark correlation and unexplained omissions. read the letter →

arxiv 2505.17017 v2 pith:CG76JTO7 submitted 2025-05-22 cs.CV cs.AIcs.CLcs.LG

Chengzhuo Tong , Ziyu Guo , Renrui Zhang , Wenyu Shan , Xinyu Wei , Zhenghao Xing , Hongsheng Li , Pheng-Ann Heng This is my paper
classification cs.CVcs.AIcs.CLcs.LG
keywords autoregressiveimagegenerationchain-of-thoughtreasoningdirectpreferenceoptimizationgrouprelativepolicyrewardmodelgeneralizationtext-to-imagealignmentout-of-domainscalingstrategies
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to settle a practical question for fine-tuning autoregressive image generators: should you use DPO, which trains on a fixed set of preferred and rejected images, or GRPO, which samples candidate images on the fly and compares them in groups? Using Janus-Pro as the base model, the authors find that DPO scores higher on the in-domain benchmark T2I-CompBench, on average 11.53% above GRPO across reward models, while GRPO outscores DPO on the out-of-domain benchmark GenEval by an average of 1.14%. They also claim that the intrinsic generalization of the reward model, measured by how often it picks the best of four candidate images on GenEval, predicts how well the resulting RL policy generalizes. The paper then works through three scaling levers—more sampled images per prompt, more training data, and iterative retraining—and reports that each algorithm responds to them differently.

What carries the argument

The load-bearing design is a controlled comparison with matched compute: one autoregressive generator (Janus-Pro, which outputs images as discrete visual tokens), two benchmarks that define the domain boundary (T2I-CompBench for long compositional prompts, GenEval for short templated prompts), and a set of reward models (Human Preference Score, ImageReward, UnifiedReward, a fine-tuned outcome reward model, and task-specific metric scores) that supply the preference signals. The central device is best-of-$N$ evaluation: each reward model is used as an outcome reward model to select among four candidate images on GenEval, producing an intrinsic-generalization ranking that the paper compares against the out-of-domain performance of the RL-trained policies. The scaling analysis varies the GRPO group size and the DPO sampling count, doubles and triples the in-domain prompt set with a structured language-model prompt-generation pipeline, and runs iterative training rounds with refreshed preference pairs and updated reference models.

What would settle it

Train a reward model that ranks T2I-CompBench images accurately but is deliberately handicapped on GenEval, for example by removing GenEval-style short prompts from its training data, then run both DPO and GRPO with it. If the resulting policies still generalize as well as those trained with UnifiedReward, the paper's claim that reward-model intrinsic generalization transfers to RL generalization would be refuted.

Watch

Extended reading notes

Core claim

The central discovery is that DPO and GRPO are not interchangeable in autoregressive text-to-image generation: DPO is the stronger in-domain optimizer, lifting T2I-CompBench scores consistently across every reward model tested, while GRPO is the stronger out-of-domain generalizer, edging out DPO on GenEval. The mechanism the paper points to for generalization is the reward model itself: ranking the reward models by best-of-4 selection accuracy on GenEval gives UnifiedReward, then ImageReward, then HPS, and the GenEval performance of both DPO- and GRPO-trained policies follows the same ordering. On scaling, sampling more images per prompt is the most computationally efficient route to GRPO's in-domain gains, whereas iterative training maximizes DPO's in-domain performance but degrades its out-of-domain generalization after two rounds; moderate sampling sharpens DPO's preference contrast, and excessive sampling introduces bias.

Load-bearing premise

The load-bearing premise is that best-of-4 selection accuracy on GenEval is a valid measure of a reward model's intrinsic generalization, and that this ranking determines how much generalization the RL-trained policy inherits.

Editorial extensions

If this is right

  • If these results hold, teams optimizing for in-domain text-to-image benchmark scores should prefer DPO, while teams expecting deployment on simpler, unseen prompt styles should prefer GRPO.
  • Improving the reward model's own generalization may be a more direct lever on policy generalization than tuning the RL algorithm, since out-of-domain policy performance tracked the reward-model best-of-4 ranking.
  • For GRPO, increasing the number of sampled images per prompt gives the best in-domain returns per unit of compute; doubling that sampling also helps out-of-domain performance, but tripling it starts to overfit.
  • For DPO, one or two iterative rounds are the strongest in-domain lever but erode out-of-domain performance, and moderate sampling of about eight candidates per prompt beats both smaller and much larger sampling for generalization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment, not run in the paper, is to measure best-of-$N$ reward-model generalization for the two omitted reward models (the fine-tuned outcome reward model and the metric reward) on GenEval; their inclusion would confirm or qualify the claimed alignment between reward-model and policy generalization.
  • The paper uses GenEval as the single out-of-domain probe, so the reward-model-generalization claim is only as strong as that benchmark's coverage; testing on additional out-of-domain prompt distributions would show whether the transfer effect is general or GenEval-specific.
  • The best-of-4 proxy measures a reward model's ranking ability at inference, not its training signal quality; it remains possible that a reward model ranks well on GenEval yet provides a weak or biased learning signal during RL, so the causal link from reward-model generalization to policy generalization is not yet demonstrated.
  • Because all experiments use one base generator, the DPO-versus-GRPO ordering could shift with architecture, tokenizer, or base-data distribution; replicating the comparison on other autoregressive visual generators would test whether the conclusion is algorithm-level.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This paper presents an empirical comparison of DPO and GRPO for autoregressive image generation using Janus-Pro, evaluating in-domain performance on T2I-CompBench and out-of-domain generalization on GenEval, across five reward models and three scaling strategies. It reports that DPO outperforms GRPO in-domain, GRPO generalizes better out-of-domain, DPO is more sensitive to reward-model choice, and reward models with higher best-of-4 GenEval selection accuracy tend to produce RL policies with better GenEval generalization. Scaling experiments examine sampling size, training-data diversity, and iterative training, with computational-cost tables in the appendix.

Significance. If the findings hold, this is a useful systematic reference for practitioners choosing between on-policy and off-policy RL for autoregressive image generation. The paper's strengths include the first systematic comparison in this setting, released code, transparent compute-time records, a structured GPT-4o prompt-generation pipeline, and cautious 'potentially' wording on the reward-model generalization claim. The study is, however, entirely empirical and single-run, and the reward-model generalization result is currently a same-benchmark three-point correlation, so its scientific significance is provisional until the confounds identified below are addressed.

major comments (4)
  1. [Section 2.2, Tables 1 and 2] All reported comparisons are single-run point estimates without error bars, seeds, or significance tests. The headline out-of-domain advantage of GRPO over DPO is small (0.58 to 1.87 percentage points across reward models), and the sensitivity claim in Section 2.3 (variances 0.5486 vs 0.9547) is computed from single runs. Please add repeated runs and report variance or confidence intervals, or explicitly downgrade these claims to observations.
  2. [Section 2.3, Tables 2 and 3] The claim that reward models with stronger intrinsic generalization improve RL generalization is not yet supported. Table 3 measures 'intrinsic generalization' by best-of-4 selection accuracy on GenEval, and Table 2 measures RL generalization on the same GenEval benchmark; a reward model that scores higher on GenEval is, by construction, more aligned with GenEval's evaluator, so the correlation may reflect train/eval reward alignment rather than a general property. Moreover, Table 3 includes only three of the five reward models studied in Table 1, and Table 2 omits Fine-tuned ORM despite its caption stating consistency with Table 1. Please validate the proxy on an independent out-of-domain benchmark or held-out prompt distribution, include all reward models, and specify the generative policy and prompt set used for the best-of-4 candidates.
  3. [Section 2.4, Table 5] The 'Sampling 8' row reports DPO Overall = 55.17, which is inconsistent with the DPO baseline of 77.31 and with the subcategory values in the same row (e.g., Color Attr. 73.25, Counting 70.00); this appears to be a copy error from Table 4, where DPO Overall = 55.17 under Sampling 8. Because the text cites a +1.52% out-of-domain gain at sampling 8, this table error is load-bearing for the scaling conclusion and must be corrected. There is also a smaller inconsistency between Table 2 HPS Colors (DPO 89.36) and Table 5 Base Size Colors (DPO 89.63).
  4. [Section 2.3, Table 3] The intrinsic-generalization ranking rests on three point estimates (82.14, 82.41, 84.49) with no error bars or significance testing, and the differences are small relative to the noise typical of GenEval subset scores. The claim that the rankings 'align perfectly' is therefore fragile. Please provide per-category error estimates, repeated evaluations, and a statistical test of the monotonic relationship, or weaken the conclusion accordingly.
minor comments (7)
  1. [Section 2.2] DPO is introduced as 'Deep Policy Optimization'; the correct expansion is Direct Preference Optimization.
  2. [Section 2.2] The stated average in-domain advantage of 11.53% is not reproducible from Table 1; for example, the per-reward-model relative average I compute from the Average column is about 10.0%. Please clarify the calculation.
  3. [Tables 4 and 5] The column header appears garbled ('Average Attribute Binding Object Relationship Complex Color Shape Texture Spatial Non-Spatial') and should list the actual T2I-CompBench categories used.
  4. [Figure 4] The label 'Hard Reward' is not defined in the text; the metric-based reward is called 'Metric Reward' in Section 2.3.
  5. [Table 3] Please state which generative policy produced the candidate images, how many prompts were used, and whether the same random samples were scored by all reward models.
  6. [Section 2.4, Table 7] Table 7 shows that sampling scaling is not exactly comparable in cost to data or iterative scaling (e.g., DPO ratio 2: 5.33h vs 5.99h); the statement that each factor incurs approximately comparable computational cost should be qualified.
  7. [Conclusion] Please add a limitations paragraph noting that all results are single-run and that the reward-model generalization measure is a proxy requiring independent validation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical comparisons and reward-model correlations are self-contained, with a same-benchmark confound that is a validity concern, not a definitional reduction.

full rationale

This paper is an empirical study, not a derivation. The central comparisons (DPO vs. GRPO in-domain on T2I-CompBench and out-of-domain on GenEval) are measured outcomes of training runs, and no equation or fitted parameter is renamed as a prediction. The reward-model generalization claim is supported by Table 3, where each reward model's intrinsic generalization is independently scored via best-of-4 selection on GenEval, and then compared with RL generalization in Table 2. Although both the reward-model ranking and the RL out-of-domain score use GenEval, the reward-model measurement is obtained without any RL training and is not fitted to the RL outcomes; the correlation is therefore an empirical claim, not a tautology. The same-benchmark design is a validity threat (a reward model that selects GenEval images well may be better aligned with that evaluator), but circularity requires a definitional reduction or a fitted parameter renamed as a prediction, which is not present. The paper's self-citations, including the ORM best-of-N evaluation framework from Guo et al. [18], are methodological and do not carry the burden of the result; the result is the observed ranking. The exclusion of Fine-tuned ORM from Tables 2 and 3 weakens the scope of the alignment claim but does not make it circular. No step in the paper's chain reduces to its own input.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

This is an empirical study with no theoretical derivation, so the ledger captures the key experimental assumptions and hand-chosen hyperparameters that the conclusions rest on. No new theoretical entities are introduced.

free parameters (4)
  • GRPO group size = 4
    Number of completions sampled per prompt for group-relative advantage estimation; set in Section 2.2 and used in all baseline GRPO runs. Affects the comparison and the scaling analysis.
  • DPO samples per prompt = 4
    Number of generated images per prompt used to select the highest and lowest scoring images as the chosen and rejected pair; matched to the GRPO group size for fair comparison (Section 2.2).
  • Baseline training iterations = 1
    The baseline GRPO and DPO runs are trained for one iteration before the iterative scaling experiments add more cycles (Section 2.4).
  • KL penalty coefficient
    The KL regularization coefficient for DPO and GRPO is not reported, yet it strongly affects training dynamics and the balance between reward optimization and distribution shift.
assumptions (4)
  • standard math DPO loss and GRPO group-relative advantage estimator are valid objectives for autoregressive image generation.
    The paper directly adapts DPO (Rafailov et al.) and GRPO (Shao et al.) to image generation without re-deriving or questioning their theoretical validity (Section 2.1).
  • domain assumption T2I-CompBench and GenEval respectively represent in-domain and out-of-domain distributions for Janus-Pro.
    All comparisons and conclusions about in-domain vs out-of-domain performance depend on this distribution split (Section 2.1).
  • domain assumption A reward model's best-of-4 selection accuracy on GenEval measures its intrinsic generalization.
    Used to rank reward models in Section 2.3 and Table 3; the paper does not validate this proxy against any independent ground truth.
  • domain assumption Selecting the highest and lowest scoring images as DPO chosen/rejected pairs is a valid construction of preference data.
    Extreme-pair selection may violate DPO's implicit Bradley-Terry preference assumptions and could bias training, but the paper adopts it without analysis (Section 2.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO." pith.science (2026). https://pith.science/paper/CG76JTO7

@misc{pith2026250517017,
  author       = {Pith},
  title        = {Pith review of: Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CG76JTO7}},
  note         = {Machine review of arXiv:2505.17017}
}
read the original abstract

Recent advancements underscore the significant role of Reinforcement Learning (RL) in enhancing the Chain-of-Thought (CoT) reasoning capabilities of large language models (LLMs). Two prominent RL algorithms, Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO), are central to these developments, showcasing different pros and cons. Autoregressive image generation, also interpretable as a sequential CoT reasoning process, presents unique challenges distinct from LLM-based CoT reasoning. These encompass ensuring text-image consistency, improving image aesthetic quality, and designing sophisticated reward models, rather than relying on simpler rule-based rewards. While recent efforts have extended RL to this domain, these explorations typically lack an in-depth analysis of the domain-specific challenges and the characteristics of different RL strategies. To bridge this gap, we provide the first comprehensive investigation of the GRPO and DPO algorithms in autoregressive image generation, evaluating their in-domain performance and out-of-domain generalization, while scrutinizing the impact of different reward models on their respective capabilities. Our findings reveal that GRPO and DPO exhibit distinct advantages, and crucially, that reward models possessing stronger intrinsic generalization capabilities potentially enhance the generalization potential of the applied RL algorithms. Furthermore, we systematically explore three prevalent scaling strategies to enhance both their in-domain and out-of-domain proficiency, deriving unique insights into efficiently scaling performance for each paradigm. We hope our study paves a new path for inspiring future work on developing more effective RL algorithms to achieve robust CoT reasoning in the realm of autoregressive image generation. Code is released at https://github.com/ZiyuGuo99/Image-Generation-CoT

Figures

Figures reproduced from arXiv: 2505.17017 by the authors.

Figure 1
Figure 1. Investigation for GRPO and DPO in Autoregressive Image Generation. We analyze the advantages of GRPO and DPO in both in-domain and out-of-domain scenarios (Top-left), the effect of different reward models (Top-right), and the influence of scaling strategies (Bottom), providing unique insights to this field. (GRPO) [43]. DPO offers compelling advantages in reduced training time and computational cost, while achieving… view at source ↗
Figure 2
Figure 2. Visualization Results of In-Domain vs. Out-of-Domain Performance Comparison. multiple responses per query. Our empirical investigation initiates with a GRPO training phase, employing a group size of four completions per query with the hyperparameter for iteration times set to 1, following the standard GRPO workflow. Building on the design of reward models, as elaborated in 2.3, we focus here on the curation of train… view at source ↗
Figure 3
Figure 3. (a) The Impact of Different Reward Models’ Intrinsic Generalization Capability. We evaluate the generalization performance of GRPO, DPO, and the intrinsic generalization performance (represented by best-of-4 strategy) of three reward models. (b-e) Effects of Three Scaling Strategies. We examine the effects of various scaling strategies, including sampling size, in-domain data diversity, and iterative training, on bo… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visualization Results of the Impact of Different Reward Models [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Visualization Results of Insights from Investigating Scaling Strategies. 2. Semantic Novelty Enforcement: We implemented specific generation constraints to prevent superficial variations, requiring GPT-4o to produce genuinely novel compo￾sitions rather than simple lexi…

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HumorGen: Cognitive Synergy for Humor Generation in Large Language Models via Persona-Based Distillation

    cs.CL 2026-03 unverdicted novelty 6.0 of 10

    Persona-based Mixture-of-Thought data curation lets a 7B student outperform larger models on humor generation, while DPO and O-GRPO add no gain over SFT.

  2. Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generation

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A curated GPT-4o synthetic image dataset improves open-source generation models on instruction-following, surreal scenes, and multi-reference synthesis, plus two new benchmarks to measure those skills.

  3. EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models

    cs.AI 2026-02 conditional novelty 5.0 of 10

    EMO-R3, which combines a three-step emotional reasoning prompt with a reward for the model agreeing with its own image–emotion judgments, raises visual emotion-recognition accuracy by about one point over plain GRPO.

  4. MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning

    cs.CV 2025-06 conditional novelty 5.0 of 10

    MINT-CoT-7B interleaves fine-grained visual tokens into each math reasoning step and reports 73.70 on MathVista-Math, 64.72 on GeoQA, and 69.6 on MMStar-Math.

Reference graph

Works this paper leans on

67 extracted references · 18 canonical work pages · cited by 4 Pith papers

  1. [1]

    https://www.anthropic.com/claude/sonnet/, 2025

    Claude 3.7 sonnet. https://www.anthropic.com/claude/sonnet/, 2025

    work page 2025

  2. [2]

    https://deepmind.google/technologies/gemini/pro/, 2025

    Gemini 2.5 pro. https://deepmind.google/technologies/gemini/pro/, 2025

    work page 2025

  3. [3]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024

    arXiv 2024

  4. [4]

    MathQA: Towards interpretable math word problem solving with operation-based formalisms

    Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319, 2019

    arXiv 1905

  5. [5]

    Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J

    Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V . Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732, 2021

    arXiv 2021

  6. [6]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  7. [7]

    MaskGIT: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGIT: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11315–11325, 2022

    work page 2022

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar...

    arXiv 2021

Show all 67 references
  1. [9]

    Janus-Pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-Pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811, 2025

    2025 arXiv

  2. [10]

    Self-play fine-tuning converts weak language models to strong language models

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024. 13

    2024 arXiv

  3. [11]

    DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, et al. DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  4. [12]

    Scaling rectified flow trans- formers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first international conference on machin...

    2024

  5. [13]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021

    2021

  6. [14]

    Video-R1: Reinforcing video reasoning in mllms, 2025

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-R1: Reinforcing video reasoning in mllms, 2025

    2025

  7. [15]

    Geneval: An object-focused framework for evaluating text-to-image alignment

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132–52152, 2023

    2023

  8. [16]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  9. [17]

    Sciverse: Unveiling the knowledge comprehension and visual reasoning of lmms on multi-modal scientific problems

    Ziyu Guo, Ray Zhang, Hao Chen, Jialin Gao, Dongzhi Jiang, Jiaze Wang, and Pheng-Ann Heng. Sciverse: Unveiling the knowledge comprehension and visual reasoning of lmms on multi-modal scientific problems. arXiv preprint arXiv:2503.10627, 2025

    2025 arXiv

  10. [18]

    Can we generate images with cot? let’s verify and reinforce image generation step by step

    Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng-Ann Heng. Can we generate images with cot? let’s verify and reinforce image generation step by step. arXiv preprint arXiv:2501.13926, 2025

    2025 arXiv

  11. [19]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021

    2021

  12. [20]

    Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans

    Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models, 2022

    2022

  13. [21]

    T2I-CompBench: A com- prehensive benchmark for open-world compositional text-to-image generation

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2I-CompBench: A com- prehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723–78747, 2023

    2023

  14. [22]

    LiveCodeBench: Holistic and contami- nation free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contami- nation free evaluation of large language models for code. CoRR, abs/2403.07974, 2024

    2024 arXiv

  15. [23]

    T2I-R1: Reinforcing image generation with collaborative semantic-level and token-level cot

    Dongzhi Jiang, Ziyu Guo, Renrui Zhang, Zhuofan Zong, Hao Li, Le Zhuo, Shilin Yan, Pheng- Ann Heng, and Hongsheng Li. T2I-R1: Reinforcing image generation with collaborative semantic-level and token-level cot. arXiv preprint arXiv:2505.00703, 2025

    2025 arXiv

  16. [24]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022

    2022

  17. [25]

    VideoPoet: A large language model for zero-shot video generation

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. VideoPoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023

    2023 arXiv

  18. [26]

    Black Forest Labs. Flux. https://github.com/black-forest-labs/flux , 2024

    2024

  19. [27]

    LLaV A-OneVision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. LLaV A-OneVision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024. 14

    2024 arXiv

  20. [28]

    VideoChat-R1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning

    Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. VideoChat-R1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning. arXiv preprint arXiv:2504.06958, 2025

    2025 arXiv

  21. [29]

    VideoChat-R1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning, 2025

    Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. VideoChat-R1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning, 2025

    2025

  22. [30]

    Cppo: Accelerating the training of group relative policy optimization-based reasoning models

    Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the training of group relative policy optimization-based reasoning models. arXiv preprint arXiv:2503.22342, 2025

    2025

  23. [31]

    Visual-RFT: Visual reinforcement fine-tuning

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-RFT: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  24. [32]

    American invitational mathematics examination - aime

    MAA. American invitational mathematics examination - aime. In American Invitational Mathematics Examination - AIME 2024, February 2024

    2024

  25. [33]

    OpenAI. Chatgpt. https://chat.openai.com, 2023

    2023

  26. [34]

    Hello gpt-4o

    OpenAI. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/ , 2024

    2024

  27. [35]

    OpenAI o1 system card, 2024

    OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, et al. OpenAI o1 system card, 2024

    2024

  28. [36]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  29. [37]

    Iterative reasoning preference optimization

    Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems, 37:116617–116637, 2024

    2024

  30. [38]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  31. [39]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  32. [40]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  33. [41]

    High- dimensional continuous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015

    2015 arXiv

  34. [42]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  35. [43]

    DeepSeekMath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  36. [44]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024

    2024 arXiv

  37. [45]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025. 15

    2025 arXiv

  38. [46]

    LaMDA: Language models for dialog applications, 2022

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, et al. LaMDA: Language models for dialog applications, 2022

    2022

  39. [47]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  40. [48]

    SimpleAR: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl

    Junke Wang, Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, and Yu-Gang Jiang. SimpleAR: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl. arXiv preprint arXiv:2504.11455, 2025

    2025 arXiv

  41. [49]

    Reasoning in conversation: Solving subjective tasks through dialogue simulation for large language models

    Xiaolong Wang, Yile Wang, Yuanchi Zhang, Fuwen Luo, Peng Li, Maosong Sun, and Yang Liu. Reasoning in conversation: Solving subjective tasks through dialogue simulation for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin...

    2024

  42. [50]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V . Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  43. [51]

    Unified reward model for multimodal understanding and generation

    Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified reward model for multimodal understanding and generation. arXiv preprint arXiv:2503.05236, 2025

    2025 arXiv

  44. [52]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  45. [53]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410.13848, 2024

    arXiv

  46. [54]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341, 2023

    2023 arXiv

  47. [55]

    Show-o: One single trans- former to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single trans- former to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  48. [56]

    ImageReward: Learning and evaluating human preferences for text-to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. ImageReward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903–15935, 2023

    2023

  49. [57]

    Is dpo superior to ppo for llm alignment? a comprehensive study.arXiv preprint arXiv:2404.10719, 2024

    Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study.arXiv preprint arXiv:2404.10719, 2024

    2024 arXiv

  50. [58]

    Dancegrpo: Unleashing grpo on visual generation

    Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.07818, 2025

    2025 arXiv

  51. [59]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  52. [60]

    Vector-quantized image modeling with improved vqgan

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627, 2021

    2021 arXiv

  53. [61]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  54. [62]

    ReST-MCTS*: Llm self-training via process reward guided tree search

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. ReST-MCTS*: Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735–64772, 2024

    2024

  55. [63]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3836–3847, 2023

    2023

  56. [64]

    Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention

    Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In ICLR 2024, 2024

    2024

  57. [65]

    MathVerse: Does your multi-modal llm truly see the diagrams in visual math problems? ECCV 2024, 2024

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. MathVerse: Does your multi-modal llm truly see the diagrams in visual math problems? ECCV 2024, 2024

    2024

  58. [66]

    Mavis: Mathematical visual instruction tuning with an automatic data engine

    Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Ziyu Guo, Shicheng Li, Yichi Zhang, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, et al. Mavis: Mathematical visual instruction tuning with an automatic data engine. arXiv preprint arXiv:2407.08739, 2024

    2024 arXiv

  59. [67]

    SafetyBench: Evaluating the safety of large language models

    Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. SafetyBench: Evaluating the safety of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.