Pith. sign in

REVIEW 3 major objections 6 minor 5 cited by

RewardDance: Reward Scaling in Visual Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RewardDance claims that recasting reward as a yes-token probability in a VLM unlocks scaling to 26B parameters and yields resistance to reward hacking.

desk verdict Useful scaling study; the 'proving' reward-hacking claim rests on an unvalidated variance proxy and should be softened. read the letter →

arxiv 2509.08826 v1 pith:VSKFQMK2 submitted 2025-09-10 cs.CV

Jie Wu , Yu Gao , Zilyu Ye , Ming Li , Liang Li , Hanzhong Guo , Jie Liu , Zeyue Xue
show 4 more authors
Xiaoxia Hou Wei Liu Yan Zeng Weilin Huang
This is my paper
classification cs.CV
keywords rewardmodelsvisualgenerationreinforcementlearningfromhumanfeedbackhackingscalinglawsvision-languagetext-to-imagetext-to-video
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RewardDance argues that the usual way to build visual reward models—a scalar regression head trained with Bradley-Terry loss—is what keeps them from scaling well. Instead, it reformulates reward as the probability that a vision-language model outputs the token 'yes' when asked whether one image beats another, which natively fits how VLMs generate text. This reformulation unlocks scaling along two axes: model parameters (from 1B to 26B) and input context (task instructions, reference images, and chain-of-thought reasoning). Across text-to-image, text-to-video, and image-to-video, the paper reports consistent quality gains from both kinds of scaling. It also reports that large reward models maintain high reward variance during RL fine-tuning, which it reads as evidence that the policy avoids reward hacking and mode collapse.

What carries the argument

The central object is the 'yes'-token probability: r(x1,x2,y,i) = P('yes' | x1,x2,y,i), where x1,x2 are images being compared, y is the prompt, and i is a task instruction. This turns reward prediction into the VLM's native autoregressive task and removes the regression head. Two scaling axes then become effective: model scaling (InternVL variants from 1B to 26B) and context scaling (task instructions, reference images, and chain-of-thought reasoning). The reward curves during RL fine-tuning, with their variance bands, serve as the diagnostic that larger models keep exploring rather than collapsing.

What would settle it

Run an RL fine-tuning loop with a large generative reward model and measure actual output diversity (e.g., perceptual feature coverage or pairwise image distance) alongside the reward variance. If reward variance stays high while output diversity drops sharply, or if held-out human preference scores flatten or worsen, the anti-hacking claim would be refuted. Comparing a 26B generative RM against a matching 26B regression RM on identical reference images would isolate whether the benefit comes from the generative formulation or simply from parameter count.

Watch

Extended reading notes

Core claim

RewardDance replaces the regression head of a VLM reward model with a generative question: 'Is image 2 better than image 1?' The reward is simply the predicted probability of the token 'yes'. This aligns the reward objective with the VLM's next-token prediction, and with that alignment the paper demonstrates that scaling model parameters from 1B to 26B and scaling context (instructions, reference images, chain-of-thought) yields consistent quality improvements in text-to-image, text-to-video, and image-to-video generation. The paper also observes that large reward models keep high reward variance during RL fine-tuning, interpreting this as evidence that the policy avoids reward hacking and m

Load-bearing premise

The paper treats high reward variance during RL fine-tuning as proof that the policy is not reward-hacking, but variance alone could also reflect a noisy or miscalibrated reward signal rather than genuinely broad exploration.

Editorial extensions

If this is right

  • If scaling RMs is the right principle, then visual generation systems should invest in larger VLM-based reward models rather than only larger generators.
  • Context scaling with reference examples and chain-of-thought provides a practical path to better reward signals without changing the base generator.
  • The reported variance signature gives a cheap, training-time early warning for reward hacking: a shrinking variance band during RL fine-tuning indicates the policy is collapsing.
  • The same generative reward formulation should carry over to other preference-based multimodal tasks, such as editing or audio-to-video, with minimal changes.
  • OOD accuracy of the reward model, not in-domain accuracy, is the metric that predicts downstream RL gains, pointing toward new benchmark design for RMs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's anti-hacking evidence rests on reward variance as a proxy; a stronger test would measure output diversity directly (e.g., feature coverage or perceptual diversity) independently of the reward model used.
  • Scaling laws for reward models may interact with generator scale: the paper already notes that larger diffusion models benefit more from larger RMs, implying a joint-scaling recipe rather than an isolated RM-scaling law.
  • If high variance is indeed the key signal, then reward-model designers might deliberately tune calibration and output entropy, not just accuracy, when training RMs for RL.
  • The pairwise reference-image formulation introduces an N-way search cost at inference; a testable extension would be to amortize or distill the pairwise comparisons into a pointwise model that retains the scaling benefits.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces RewardDance, a generative reward modeling framework for visual generation. Instead of a regression head, the reward score is defined as a VLM's probability of predicting a 'yes' token for 'image 2 is better than image 1' under task-specific instructions, reference examples, and chain-of-thought reasoning. The framework is scaled from 1B to 26B parameters and evaluated on text-to-image, text-to-video, and image-to-video tasks under RL fine-tuning and test-time scaling. The main reported results are consistent scaling gains in alignment score, GenEval, Bench-240, and SeedVideoBench-1.0. The paper further claims that large reward models exhibit high reward variance during RL fine-tuning, which it interprets as proof of resistance to reward hacking and sustained output diversity.

Significance. If the scaling results hold, the paper makes a practically important contribution: it is the first systematic study of scaling a generative VLM-based reward model for visual generation, and it demonstrates gains across multiple base generators (FLUX.1-dev, Seedream-3.0, Seedance-1.0) and two optimization regimes (RL and test-time scaling). The OOD-accuracy discussion is a useful suggestion for reward-model evaluation. However, the headline 'Crucially, we resolve reward hacking' claim is not currently established: it rests entirely on an unvalidated proxy, reward variance, and the body text itself only says the evidence 'suggests' resistance. Because this is the central novelty emphasized in the abstract and introduction, the paper needs either direct validation of the anti-hacking mechanism or a substantial tempering of the claim.

major comments (3)
  1. [Abstract; §4.4; Figs. 2, 5, 6; Eq. (2)] The anti-hacking conclusion is based solely on reward-variance dynamics. The abstract says high reward variance 'proves' resistance to hacking, but §4.4 only says the relationship 'strongly suggests' it, and Fig. 1 calls variance an 'indicator.' No independent validation is provided: no output-diversity metric (e.g., LPIPS, Vendi), no reward-overoptimization curve on held-out human preference, and no comparison with a policy known to have been hacked. A noisy or miscalibrated reward model can produce high temporal variance even for a degenerate policy, while a well-calibrated model can assign stable rewards to diverse high-quality outputs. As the central 'Crucially' contribution, this needs direct evidence, not a proxy.
  2. [Table 2; §4.2] The in-domain RM accuracy is non-monotonic with scale (64.70, 69.36, 65.37, 74.92, 78.44), and the paper introduces OOD accuracy post hoc as 'more critical' after observing that it correlates with the desired scaling trend. With only five RM sizes and no independent, pre-registered metric selection, this is an ad-hoc reinterpretation. OOD accuracy could indeed be the better predictor, but that claim needs support beyond a single correlation on the authors' own data, e.g., cross-validation across RM families or benchmarks, or an a priori argument for why OOD accuracy should be decisive.
  3. [Tables 5 and 6; §4.1.2] Two of the three headline benchmarks, Bench-240 and SeedVideoBench-1.0, are developed by the same organization that produces the evaluated models (Seedream and Seedance). The paper reports no independent human evaluation, no confidence intervals, and no third-party replication for these benchmarks. The SOTA claims in Tables 5 and 6 therefore rest largely on internally constructed evaluation sets. Independent evaluation or release of the full prompt/rating protocol is needed before the SOTA claims can be accepted at face value.
minor comments (6)
  1. [Abstract] Typo: 'It primarily due' should be 'It is primarily due.'
  2. [Figs. 2, 5, 6] The numbers next to the curves (e.g., '=7.2e-3') are not labeled. Please state explicitly that these are standard deviations of raw/smoothed reward scores, and report the sliding-window length (stated in §4.4 as 1,000) in the captions.
  3. [Table 3] The cell entries such as '+28%+32% +4%' are visually confusing. Use separate columns for GSB improvement and its uncertainty, or explain the notation in the caption.
  4. [§3.3.2] Typo: 'point-wisee' should be 'point-wise.'
  5. [§3.2.3] The weighted CE loss coefficient for the pointwise generative variant is described only as 'small.' Please give the exact value for reproducibility.
  6. [§4.4 and Fig. 7] The claim that larger DiT architectures benefit more from reward scaling rests on a single comparison (Seedream-Lite vs. Seedream). Add at least one more model pair or error bars before drawing a scaling-law conclusion.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; central scaling results are empirically grounded, though the anti-hacking claim relies on an unvalidated proxy rather than a circular derivation.

full rationale

RewardDance's core reward score is defined as P(yes) (Eq. 2), and the scaling experiments compare RM sizes on a mix of external (GenEval, ImageReward, HPS) and internal (Bench-240, SeedVideoBench) benchmarks. The GenEval results and OOD preference sets provide independent grounding for the scaling claims, so the central scaling conclusion is not circular. Self-citations to Seedream/Seedance technical reports and benchmarks are present, but they are not used to derive the scaling law; they are evaluation instruments, and external GenEval mitigates the risk. The abstract's statement that high reward variance 'proves' resistance to reward hacking is stronger than Section 4.4's 'strongly suggests'; however, this is a validity threat—variance is an unvalidated proxy for diversity and anti-hacking—rather than a circularity, because the paper does not define resistance to hacking as reward variance by equation. No fitted parameter is renamed as a prediction, and no load-bearing argument reduces to a self-citation. The anti-hacking claim deserves scrutiny on evidential grounds, but it does not meet the standard for derivation-level circularity.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its conceptual invention is a metric, reward variance as an anti-hacking indicator, which is an unvalidated proxy rather than an entity. The free parameters and assumptions above are the load-bearing choices the central claims depend on.

free parameters (3)
  • CE loss coefficient for pointwise variant = not reported (described as 'a small coefficient')
    Introduced in Section 3.2.3 to stabilize pointwise generative RM training; the value is unspecified, so it is a hand-chosen hyperparameter the reader cannot audit.
  • Best-of-N reference count and selection = N=2, 6, or 16, with Top-2 selection in main experiments
    Section 3.3.1 and Table 8; reference image quality is load-bearing for pairwise reward scoring, and the choice is experimental, not derived.
  • Reward variance window length = 1,000 RL iterations
    Used in Figures 5 and 6 to quantify exploration; the window length is chosen by hand and determines the reported variance values that support the anti-hacking claim.
assumptions (6)
  • ad hoc to paper High reward variance during RL training indicates resistance to reward hacking and sustained exploration.
    Figures 2, 5, 6 and Section 4.4 use variance as the evidence for the paper's central anti-hacking claim; no external measure of hacking or mode collapse is provided.
  • domain assumption In-house Bench-240 and SeedVideoBench-1.0 human scores are reliable measures of generation quality.
    Section 4.1.2; benchmarks and scoring rubrics are defined by the authors' organization, and inter-annotator agreement and error bars are not reported.
  • ad hoc to paper OOD RM accuracy, rather than ID accuracy, is the predictive metric for downstream RL quality.
    Section 4.2 adopts this after observing non-monotonic ID accuracies, making it a post hoc modeling choice rather than a pre-specified hypothesis.
  • domain assumption Uniform scaling of the InternVL backbone from 1B to 26B preserves and improves preference-judgment capability.
    Section 3.2.2; no analysis of data scaling, compute, or possible negative scaling effects is given beyond the reported trend.
  • domain assumption CoT reasoning distilled from SEED-VL 1.5 is accurate and improves reward judgments.
    Section 3.2.1 and Figure 4; teacher quality is assumed, not independently evaluated.
  • standard math ReFL with a frozen generative reward model provides a valid RLHF signal for diffusion and flow generators.
    Section 3.3.1; this is a standard assumption from prior work such as ReFL and DDPO.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RewardDance: Reward Scaling in Visual Generation." pith.science (2026). https://pith.science/paper/VSKFQMK2

@misc{pith2026250908826,
  author       = {Pith},
  title        = {Pith review of: RewardDance: Reward Scaling in Visual Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VSKFQMK2}},
  note         = {Machine review of arXiv:2509.08826}
}
read the original abstract

Reward Models (RMs) are critical for improving generation models via Reinforcement Learning (RL), yet the RM scaling paradigm in visual generation remains largely unexplored. It primarily due to fundamental limitations in existing approaches: CLIP-based RMs suffer from architectural and input modality constraints, while prevalent Bradley-Terry losses are fundamentally misaligned with the next-token prediction mechanism of Vision-Language Models (VLMs), hindering effective scaling. More critically, the RLHF optimization process is plagued by Reward Hacking issue, where models exploit flaws in the reward signal without improving true quality. To address these challenges, we introduce RewardDance, a scalable reward modeling framework that overcomes these barriers through a novel generative reward paradigm. By reformulating the reward score as the model's probability of predicting a "yes" token, indicating that the generated image outperforms a reference image according to specific criteria, RewardDance intrinsically aligns reward objectives with VLM architectures. This alignment unlocks scaling across two dimensions: (1) Model Scaling: Systematic scaling of RMs up to 26 billion parameters; (2) Context Scaling: Integration of task-specific instructions, reference examples, and chain-of-thought (CoT) reasoning. Extensive experiments demonstrate that RewardDance significantly surpasses state-of-the-art methods in text-to-image, text-to-video, and image-to-video generation. Crucially, we resolve the persistent challenge of "reward hacking": Our large-scale RMs exhibit and maintain high reward variance during RL fine-tuning, proving their resistance to hacking and ability to produce diverse, high-quality outputs. It greatly relieves the mode collapse problem that plagues smaller models.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SciForma: Structure-Faithful Generation of Scientific Diagrams

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A 9B open-weights model trained with axis-decomposed conjunctive preference optimization (M-DPO) and a structural inventory beats GPT-Image-1.5 on scientific-diagram structural-fidelity benchmarks.

  2. NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learning

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    NormGuard, a hinge penalty on excess velocity norm during RL post-training of flow models, improves perceptual quality and realism without sacrificing reward.

  3. Z-Reward: Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Z-Reward trains a 27B reasoning teacher VLM on score distributions via GDSO and distills it via RISD into a 9B student, reaching 89.6% and 88.6% human preference accuracy with 41.3% optimization gain over SFT baseline.

  4. Distribution Matching Distillation Meets Reinforcement Learning

    cs.CV 2025-11 conditional novelty 6.0 of 10

    Combining DMD distillation with RL during training produces few-step text-to-image models that outperform their multi-step teacher on several benchmarks.

  5. Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generation

    cs.LG 2026-07 conditional novelty 5.0 of 10

    A per-category best-of-K RL reward, multi-axis max@K, shifts SD3.5-M perceived-appearance distributions toward uniform coverage (Fairness Score +0.23 to +0.36) without quality loss.

Reference graph

Works this paper leans on

72 extracted references · 37 linked inside Pith · cited by 5 Pith papers

  1. [1]

    Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models

    Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233, 2024

    arXiv 2024

  2. [2]

    Improving image generation with better captions.Computer Science

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions.Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023

    2023

  3. [3]

    Training diffusion models with reinforcement learning

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301, 2023

    arXiv 2023

  4. [4]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  5. [5]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952

    1952

  6. [6]

    Video generation models as world simulators.OpenAI Blog, 1(8):1, 2024

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators.OpenAI Blog, 1(8):1, 2024

    2024

  7. [7]

    Pixart-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023

    arXiv 2023

  8. [8]

    Control-a-video: Controllable text-to-video generation with diffusion models.CoRR, 2023

    Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models.CoRR, 2023

    2023

Show all 72 references
  1. [9]

    The accuracy paradox in rlhf: When better reward models don’t yield better language models.arXiv preprint arXiv:2410.06554, 2024

    Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, and Xiaoyu Shen. The accuracy paradox in rlhf: When better reward models don’t yield better language models.arXiv preprint arXiv:2410.06554, 2024

    2024 arXiv

  2. [10]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  3. [11]

    Raft: Reward ranked finetuning for generative foundation model alignment

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023

    2023 arXiv

  4. [12]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-firstinternational conference on machine l...

    2024

  5. [13]

    Reinforcement learning for fine-tuning text-to-image diffusion mod- els

    Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuning text-to-image diffusion mod- els. InThirty-seventhConference on Neural Information Processing ...

    2023

  6. [14]

    Seedream 3.0 technical report.arXiv preprint arXiv:2504.11346, 2025

    Yu Gao, Lixue Gong, Qiushan Guo, Xiaoxia Hou, Zhichao Lai, Fanshi Li, Liang Li, Xiaochen Lian, Chao Liao, Liyang Liu, et al. Seedream 3.0 technical report.arXiv preprint arXiv:2504.11346, 2025

    2025 arXiv

  7. [15]

    Seedance 1.0: Exploring the boundaries of video generation models

    Yu Gao, Haoyuan Guo, Tuyen Hoang, Weilin Huang, Lu Jiang, Fangyuan Kong, Huixia Li, Jiashi Li, Liang Li, Xiaojie Li, et al. Seedance 1.0: Exploring the boundaries of video generation models. arXiv preprint arXiv:2506.09113, 2025

    2025 arXiv

  8. [16]

    Seedream 2.0: A native chinese-english bilingual image generation foundation model.arXiv preprint arXiv:2503.07703, 2025

    Lixue Gong, Xiaoxia Hou, Fanshi Li, Liang Li, Xiaochen Lian, Fei Liu, Liyang Liu, Wei Liu, Wei Lu, Yichun Shi, et al. Seedream 2.0: A native chinese-english bilingual image generation foundation model.arXiv preprint arXiv:2503.07703, 2025

    2025 arXiv

  9. [17]

    Veo.https://deepmind.google/models/veo/, 2025

    Google. Veo.https://deepmind.google/models/veo/, 2025

    2025

  10. [18]

    Multi-reward as condition for instruction-based image editing.arXiv preprint arXiv:2411.04713, 2024

    Xin Gu, Ming Li, Libo Zhang, Fan Chen, Longyin Wen, Tiejian Luo, and Sijie Zhu. Multi-reward as condition for instruction-based image editing.arXiv preprint arXiv:2411.04713, 2024. 16

    2024 arXiv

  11. [19]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023

    2023 arXiv

  12. [20]

    A simple and effective reinforcement learning method for text-to-image diffusion fine-tuning

    Shashank Gupta, Chaitanya Ahuja, Tsung-Yu Lin, Sreya Dutta Roy, Harrie Oosterhuis, Maarten de Rijke, and Satya Narayan Shukla. A simple and effective reinforcement learning method for text-to-image diffusion fine-tuning. arXiv preprint arXiv:2503.00897, 2025

    2025

  13. [21]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  14. [22]

    Ideogram.https://about.ideogram.ai/1.0., 2024

    Ideogram. Ideogram.https://about.ideogram.ai/1.0., 2024

    2024

  15. [23]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation.Advancesin neural information processing systems, 36:36652–36663, 2023

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation.Advancesin neural information processing systems, 36:36652–36663, 2023

    2023

  16. [24]

    klingai.https://app.klingai.com/cn/, 2025

    klingai. klingai.https://app.klingai.com/cn/, 2025

    2025

  17. [25]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  18. [26]

    Flux: Official inference repository for flux.1 models, 2024

    Black Forest Labs. Flux: Official inference repository for flux.1 models, 2024. URL https://github.com/ black-forest-labs/flux. Accessed: 2024-11-12

    2024

  19. [27]

    Flux.https://github.com/black-forest-labs/flux, 2024

    Black Forest Labs. Flux.https://github.com/black-forest-labs/flux, 2024

    2024

  20. [28]

    Controlnet++: Improving conditional controls with efficient consistency feedback: Project page: liming-ai

    Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, and Chen Chen. Controlnet++: Improving conditional controls with efficient consistency feedback: Project page: liming-ai. github. io/controlnet_- plus_plus. In European Conference on Computer Vision,...

    2024

  21. [29]

    Superedit: Rectifying and facilitating supervision for instruction-based image editing.arXiv preprint arXiv:2505.02370, 2025

    Ming Li, Xin Gu, Fan Chen, Xiaoying Xing, Longyin Wen, Chen Chen, and Sijie Zhu. Superedit: Rectifying and facilitating supervision for instruction-based image editing.arXiv preprint arXiv:2505.02370, 2025

    2025 arXiv

  22. [30]

    Exploring how generative mllms perceive more than clip with the same vision encoder.arXiv preprint arXiv:2411.05195, 2024

    Siting Li, Pang Wei Koh, and Simon Shaolei Du. Exploring how generative mllms perceive more than clip with the same vision encoder.arXiv preprint arXiv:2411.05195, 2024

    2024 arXiv

  23. [31]

    An inverse scaling law for clip training.Advancesin Neural Information Processing Systems, 36:49068–49087, 2023

    Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for clip training.Advancesin Neural Information Processing Systems, 36:49068–49087, 2023

    2023

  24. [32]

    Mogao: An omni foundation model for interleaved multi-modal generation.arXiv preprint arXiv:2505.05472, 2025

    Chao Liao, Liyang Liu, Xun Wang, Zhengxiong Luo, Xinyu Zhang, Wenliang Zhao, Jie Wu, Liang Li, Zhi Tian, and Weilin Huang. Mogao: An omni foundation model for interleaved multi-modal generation.arXiv preprint arXiv:2505.05472, 2025

    2025 arXiv

  25. [33]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  26. [34]

    Flow-grpo: Training flow matching models via online rl.arXiv preprint arXiv:2505.05470, 2025

    Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl.arXiv preprint arXiv:2505.05470, 2025

    2025 arXiv

  27. [35]

    Improving video generation with human feedback.arXiv preprint arXiv:2501.13918, 2025

    Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, et al. Improving video generation with human feedback.arXiv preprint arXiv:2501.13918, 2025

    2025 arXiv

  28. [36]

    Inference-time scaling for generalist reward modeling.arXiv preprint arXiv:2504.02495, 2025

    Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling.arXiv preprint arXiv:2504.02495, 2025

    2025

  29. [37]

    lumalabs.https://lumalabs.ai/, 2024

    lumalabs. lumalabs.https://lumalabs.ai/, 2024

    2024

  30. [38]

    Step-video-t2v technical report: The practice, challenges, and future of video foundation model.arXiv preprint arXiv:2502.10248, 2025

    Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, et al. Step-video-t2v technical report: The practice, challenges, and future of video foundation model.arXiv preprint arXiv:2502.10248, 2025

    2025 arXiv

  31. [39]

    Inference-time scaling for diffusion models beyond scaling denoising steps

    Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, et al. Inference-time scaling for diffusion models beyond scaling denoising steps. arXiv preprint arXiv:2501.09732, 2025. 17

    2025 arXiv

  32. [40]

    Hpsv3: Towards wide-spectrum human preference score

    Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. arXiv preprint arXiv:2508.03789, 2025

    2025 arXiv

  33. [41]

    midjourney.https://www.midjourney.com/home, 2024

    midjourney. midjourney.https://www.midjourney.com/home, 2024

    2024

  34. [42]

    Inference-time text-to-video alignment with diffusion latent beam search.arXiv preprint arXiv:2501.19252, 2025

    Yuta Oshima, Masahiro Suzuki, Yutaka Matsuo, and Hiroki Furuta. Inference-time text-to-video alignment with diffusion latent beam search.arXiv preprint arXiv:2501.19252, 2025

    2025

  35. [43]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

    2022

  36. [44]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  37. [45]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    2022 arXiv

  38. [46]

    What makes a reward model a good teacher? an optimization perspective.arXiv preprint arXiv:2503.15477, 2025

    Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D Lee, and Sanjeev Arora. What makes a reward model a good teacher? an optimization perspective.arXiv preprint arXiv:2503.15477, 2025

    2025

  39. [47]

    recraft.https://www.recraft.ai/, 2024

    recraft. recraft.https://www.recraft.ai/, 2024

    2024

  40. [48]

    Byteedit: Boost, comply and accelerate generative image editing

    Yuxi Ren, Jie Wu, Yanzuo Lu, Huafeng Kuang, Xin Xia, Xionghui Wang, Qianqian Wang, Yixing Zhu, Pan Xie, Shiyin Wang, et al. Byteedit: Boost, comply and accelerate generative image editing. InEuropean Conference on Computer Vision, pages 184–200. Springer, 2024

    2024

  41. [49]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  42. [50]

    Runway.https://runwayml.com/research/introducing-runway-gen-4, 2025

    Runway. Runway.https://runwayml.com/research/introducing-runway-gen-4, 2025

    2025

  43. [51]

    Seaweed-7b: Cost-effective training of video generation foundation model.arXiv preprint arXiv:2504.08685, 2025

    Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, et al. Seaweed-7b: Cost-effective training of video generation foundation model.arXiv preprint arXiv:2504.08685, 2025

    2025 arXiv

  44. [52]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InInternational conference on machine learning, pages 2256–2265. pmlr, 2015

    2015

  45. [53]

    Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

    2011 arXiv

  46. [54]

    Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    2024 arXiv

  47. [55]

    Diffusion model alignment using direct preference optimization

    Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and...

    2024

  48. [56]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianx- iao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    2025 arXiv

  49. [57]

    Worldpm: Scaling human preference modeling.arXiv preprint arXiv:2505.10527, 2025

    Binghai Wang, Runji Lin, Keming Lu, Le Yu, Zhenru Zhang, Fei Huang, Chujie Zheng, Kai Dang, Yang Fan, Xingzhang Ren, et al. Worldpm: Scaling human preference modeling.arXiv preprint arXiv:2505.10527, 2025

    2025 arXiv

  50. [58]

    Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    2024 arXiv

  51. [59]

    Unified multimodal chain-of-thought reward model through reinforcement fine-tuning.arXiv preprint arXiv:2505.03318, 2025

    Yibin Wang, Zhimin Li, Yuhang Zang, Chunyu Wang, Qinglin Lu, Cheng Jin, and Jiaqi Wang. Unified multimodal chain-of-thought reward model through reinforcement fine-tuning.arXiv preprint arXiv:2505.03318, 2025

    2025

  52. [60]

    Rethinking reward model evaluation: Are we barking up the wrong tree?arXiv preprint arXiv:2410.05584, 2024

    Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, Xing Yu, Xinyu Lu, Ben He, Xianpei Han, Debing Zhang, and Le Sun. Rethinking reward model evaluation: Are we barking up the wrong tree?arXiv preprint arXiv:2410.05584, 2024. 18

    2024 arXiv

  53. [61]

    Qwen-image technical report.arXiv preprint arXiv:2508.02324, 2025

    Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report.arXiv preprint arXiv:2508.02324, 2025

    2025 arXiv

  54. [62]

    Better aligning text-to-image models with human preference

    Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to-image models with human preference. arXiv preprint arXiv:2303.14420, 1(3), 2023

    2023 arXiv

  55. [63]

    Human preference score: Better aligning text-to-image models with human preference

    Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2096–2105, 2023

  56. [64]

    Imagereward: Learning and evaluating human preferences for text-to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903–15935, 2023

    2023

  57. [65]

    Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation

    Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, et al. Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation. arXiv preprint arXiv:2412.21059, 2024

    2024 arXiv

  58. [66]

    A unified pairwise framework for rlhf: Bridging generative reward modeling and policy optimization.arXiv preprint arXiv:2504.04950, 2025

    Wenyuan Xu, Xiaochen Zuo, Chao Xin, Yu Yue, Lin Yan, and Yonghui Wu. A unified pairwise framework for rlhf: Bridging generative reward modeling and policy optimization.arXiv preprint arXiv:2504.04950, 2025

    2025 arXiv

  59. [67]

    Dancegrpo: Unleashing grpo on visual generation.arXiv preprint arXiv:2505.07818, 2025

    Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation.arXiv preprint arXiv:2505.07818, 2025

    2025 arXiv

  60. [68]

    Schedule on the fly: Diffusion time prediction for faster and better image generation

    Zilyu Ye, Zhiyang Chen, Tiancheng Li, Zemin Huang, Weijian Luo, and Guo-Jun Qi. Schedule on the fly: Diffusion time prediction for faster and better image generation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 23412–23422, 2025

    2025

  61. [69]

    Make pixels dance: High-dynamic video generation

    Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8850–8860, 2024

    2024

  62. [70]

    Onlinevpo: Align video diffusion model with online video-centric preference optimization.arXiv preprint arXiv:2412.15159, 2024

    Jiacheng Zhang, Jie Wu, Weifeng Chen, Yatai Ji, Xuefeng Xiao, Weilin Huang, and Kai Han. Onlinevpo: Align video diffusion model with online video-centric preference optimization.arXiv preprint arXiv:2412.15159, 2024

    2024

  63. [71]

    Unifl: Improve latent diffusion model via unified feedback learning.Advances in Neural Information Processing Systems, 37:67355–67382, 2024

    Jiacheng Zhang, Jie Wu, Yuxi Ren, Xin Xia, Huafeng Kuang, Pan Xie, Jiashi Li, Xuefeng Xiao, Weilin Huang, Shilei Wen, et al. Unifl: Improve latent diffusion model via unified feedback learning.Advances in Neural Information Processing Systems, 37:67355–67382, 2024

    2024

  64. [72]

    Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593, 2019

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593, 2019. 19

    1909 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.