Reward as An Agent for Embodied World Models

Fei Wang; Pu Li; Qiang Wu; Shan You; Yongxuan Lv; Zhigang Lin

arxiv: 2606.19990 · v1 · pith:MDEHAP4Tnew · submitted 2026-06-18 · 💻 cs.AI

Reward as An Agent for Embodied World Models

Pu Li , Zhigang Lin , Qiang Wu , Yongxuan Lv , Fei Wang , Shan You This is my paper

Pith reviewed 2026-06-26 17:20 UTC · model grok-4.3

classification 💻 cs.AI

keywords reinforcement learningworld modelsreward hackingembodied AIrollout diversificationagentic rewardsdynamic sampling

0 comments

The pith

Unifying an agentic reward evaluator with dynamic rollout diversification allows RL to refine embodied world models with less reward hacking and higher accuracy.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the main obstacle to scaling RL for world models is not insufficient exploration but the absence of trustworthy verification methods that can support wider sampling without falling into reward hacking. It proposes Reward as an Agent, an active evaluation framework that judges generated behaviors to supply more stable signals even under distribution shifts, paired with DynDiff-GRPO, which deliberately widens action-space coverage and trajectory variety. When these two components operate together in embodied settings, where physical plausibility and task success can be directly checked, the combined system reduces exploitation of flawed rewards and produces measurable accuracy improvements across several open-source models. A sympathetic reader would care because the work suggests that broader exploration becomes viable once verification is made robust rather than by restricting rollouts to conservative regimes near the training data.

Core claim

The paper claims that treating reward as an active agent for behavior evaluation, when unified with Dynamic-Aware Rollout Diversification through DynDiff-GRPO, supplies a reliable foundation for RL that supports substantially diversified sampling, mitigates reward hacking under distribution shifts, and delivers significant accuracy gains across multiple open-source embodied world models.

What carries the argument

Reward as an Agent, an agentic reward framework that actively evaluates generated behaviors to deliver robust signals and curb hacking, together with DynDiff-GRPO for explicit expansion of action-space exploration and state-action coverage.

If this is right

Broader exploration can scale successfully once grounded in robust verification strategies.
Reward hacking is effectively mitigated under distribution shifts in embodied settings.
Significant accuracy gains are achieved across multiple open-source world models.
Dynamic diversification of rollouts produces richer embodied behaviors beyond conservative regimes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same pairing of active reward evaluation and rollout diversification might be tested in non-embodied RL domains where verification is harder to define.
If the approach holds, it could support world models that discover novel dynamics rather than only refining known ones.
Future experiments could check whether the agentic reward component alone reduces hacking when paired with simpler exploration methods.

Load-bearing premise

Physical plausibility and task completion in embodied environments supply a reliable testbed that can detect reward hacking and distribution-shift failures.

What would settle it

Finding that the combined method still permits reward hacking or fails to produce accuracy gains when applied to a fresh set of world models or dynamics outside the tested open-source ones would falsify the central claim.

read the original abstract

While RL has become a promising tool for refining world models, existing methods largely rely on conservative rollouts near the training distribution, limiting exploration, behavioral diversity, and richer dynamic discovery. In this work, we challenge this conservative paradigm. We argue that the core limitation is not exploration itself, but the lack of reliable verification strategies to support broader exploration. Without reliable verification, expanded exploration becomes highly susceptible to reward hacking, where policies exploit imperfect rewards without achieving genuine improvement. To evaluate this motivation, we instantiate our method in embodied world models, where physical plausibility, and task completion provide a rigorous testbed for scalable RL under complex dynamics. On the verification side, we introduce Reward as an Agent, an agentic reward framework that actively evaluates generated behaviors to provide robust reward signals and mitigate reward hacking under distribution shifts. On the exploration side, we introduce Dynamic-Aware Rollout Diversification through DynDiff-GRPO, which explicitly expands action-space exploration to diversify trajectories, broaden state-action coverage, and encourage richer embodied behaviors beyond conservative rollout regimes. By unifying Reward as an Agent with DynDiff-GRPO, we enable RL on a more reliable reward foundation with substantially diversified sampling, effectively mitigating reward hacking while yielding significant accuracy gains across multiple open-source world models, thereby demonstrating that broader exploration can scale successfully when grounded in robust verification.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The abstract frames reward hacking as the real blocker to broader RL exploration in embodied world models and offers Reward as an Agent plus DynDiff-GRPO as the fix, but supplies zero equations, methods, or results to check whether either component actually works.

read the letter

The punchline is that this paper bets the limitation in world-model RL is not exploration volume but the absence of reliable verification. It therefore splits the problem into an agentic reward module that actively scores behaviors to resist hacking under shifts, and a dynamic-aware diversification step (DynDiff-GRPO) that widens action-space coverage beyond conservative rollouts.

What is new is the explicit pairing of those two pieces and the claim that embodied domains (physical plausibility plus task completion) give a strong enough testbed to validate the approach. The motivation section is clear and directly targets the reward-hacking failure mode that has plagued earlier model-based methods.

The soft spot is obvious and large: the text is still only an abstract. There are no derivations, no pseudocode, no baseline comparisons, and no numbers. Without those, the stated “significant accuracy gains across multiple open-source world models” cannot be assessed, and it is impossible to tell whether the agentic reward actually mitigates distribution-shift failures or simply adds another learned component that can itself be hacked. The assumption that physical-plausibility checks will reliably expose reward hacking is also untested in the provided text.

This is aimed at the model-based RL and robotics simulation crowd. A reader already working on verification or exploration in world models would get the most from it once the full methods and experiments appear. The work shows clear thinking about the verification-versus-exploration trade-off, so it deserves a serious referee to see whether the concrete implementations hold up.

Referee Report

2 major / 0 minor

Summary. The paper claims that existing RL methods for world models are limited by conservative rollouts near the training distribution, leading to insufficient exploration. It introduces 'Reward as an Agent,' an agentic framework for robust reward evaluation to mitigate reward hacking under distribution shifts, and 'DynDiff-GRPO' for dynamic-aware rollout diversification to expand action-space exploration and trajectory diversity. Unifying the two is asserted to enable reliable RL with broader sampling, yielding significant accuracy gains across open-source embodied world models while addressing reward hacking, with physical plausibility and task completion as the evaluation testbed.

Significance. If the unification demonstrably mitigates reward hacking via agentic verification while enabling diversified exploration that produces measurable accuracy improvements, the work would address a core tension in scaling RL for world models. The embodied testbed framing could provide a concrete domain for validating robustness claims. However, the absence of any supporting derivations, algorithms, or empirical results in the manuscript prevents assessment of whether these benefits are realized.

major comments (2)

[Abstract] Abstract: The central claim that unifying Reward as an Agent with DynDiff-GRPO 'effectively mitigat[es] reward hacking while yielding significant accuracy gains across multiple open-source world models' is presented without any methods section, equations, experimental protocol, baselines, metrics, or results. This absence makes the load-bearing assertion of empirical success impossible to evaluate for soundness or support.
[Abstract] Abstract (evaluation motivation paragraph): The assertion that 'physical plausibility and task completion provide a rigorous testbed for scalable RL under complex dynamics' is used to motivate the approach, but no description of how this testbed is operationalized, what metrics quantify hacking versus genuine improvement, or how distribution-shift failures are detected is supplied. Without these details the verification strategy cannot be assessed.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their thoughtful review and for identifying key areas where additional clarity is needed in the manuscript. We agree that the abstract, as presented, makes strong claims without accompanying details on methods or experiments, which limits the ability to evaluate the work. The full manuscript is intended to provide these details, but to address the concerns directly, we will revise the paper to include explicit sections or expansions on the methods, algorithms, experimental protocols, and evaluation metrics.

read point-by-point responses

Referee: [Abstract] Abstract: The central claim that unifying Reward as an Agent with DynDiff-GRPO 'effectively mitigat[es] reward hacking while yielding significant accuracy gains across multiple open-source world models' is presented without any methods section, equations, experimental protocol, baselines, metrics, or results. This absence makes the load-bearing assertion of empirical success impossible to evaluate for soundness or support.

Authors: The referee is correct that the abstract alone does not include these elements. The manuscript's full text contains dedicated sections describing the Reward as an Agent framework, the DynDiff-GRPO algorithm with equations, the experimental protocol, baselines, metrics for accuracy and reward hacking mitigation, and results on open-source embodied world models. However, since the absence is noted, we will make a revision to ensure the abstract references these sections more clearly or to include a high-level overview of the methods and results within the abstract constraints. revision: yes
Referee: [Abstract] Abstract (evaluation motivation paragraph): The assertion that 'physical plausibility and task completion provide a rigorous testbed for scalable RL under complex dynamics' is used to motivate the approach, but no description of how this testbed is operationalized, what metrics quantify hacking versus genuine improvement, or how distribution-shift failures are detected is supplied. Without these details the verification strategy cannot be assessed.

Authors: We acknowledge this point. The abstract motivates the testbed but does not detail the operationalization. The full manuscript provides descriptions of how physical plausibility and task completion are evaluated, including specific metrics for detecting reward hacking (e.g., through verification of behaviors under distribution shifts) and genuine improvements. We will revise the manuscript to incorporate these details into the abstract or add a pointer to the relevant sections in the evaluation methodology. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The supplied abstract contains no equations, parameter fits, derivations, or self-citations. The central claims introduce two new frameworks (Reward as an Agent; DynDiff-GRPO) and assert unification benefits, but no load-bearing step reduces by construction to its own inputs. The full manuscript is referenced externally yet not reproduced here; absent any visible derivation chain or quoted reduction, the paper is scored as self-contained with no circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 2 invented entities

Review based on abstract only; full methods and equations unavailable so ledger is minimal.

axioms (1)

domain assumption Physical plausibility and task completion provide a rigorous testbed for scalable RL under complex dynamics.
Invoked in abstract to justify the embodied setting as sufficient verification.

invented entities (2)

Reward as an Agent no independent evidence
purpose: Actively evaluates generated behaviors to provide robust reward signals and mitigate reward hacking under distribution shifts.
New framework introduced to address verification gap.
DynDiff-GRPO no independent evidence
purpose: Explicitly expands action-space exploration to diversify trajectories and broaden state-action coverage.
New diversification method introduced.

pith-pipeline@v0.9.1-grok · 5773 in / 1338 out tokens · 21925 ms · 2026-06-26T17:20:34.262135+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

30 extracted references

[1]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, ...

2025
[2]

Qwen3-vl technical report, 2025

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

2025
[3]

Flow-grpo: Training flow matching models via online rl, 2025

Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl, 2025

2025
[4]

Dancegrpo: Unleashing grpo on visual generation, 2025

Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, and Ping Luo. Dancegrpo: Unleashing grpo on visual generation, 2025

2025
[5]

Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde, 2026

Junzhe Li, Yutao Cui, Tao Huang, Yinping Ma, Chun Fan, Yiming Cheng, Miles Yang, Zhao Zhong, and Liefeng Bo. Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde, 2026

2026
[6]

Pref-grpo: Pairwise preference reward-based grpo for stable text-to-image reinforcement learning, 2026

Yibin Wang, Zhimin Li, Yuhang Zang, Yujie Zhou, Jiazi Bu, Chunyu Wang, Qinglin Lu, Cheng Jin, and Jiaqi Wang. Pref-grpo: Pairwise preference reward-based grpo for stable text-to-image reinforcement learning, 2026

2026
[7]

Manifold-aware exploration for reinforcement learning in video generation.arXiv preprint arXiv:2603.21872, 2026

Mingzhe Zheng, Weijie Kong, Yue Wu, Dengyang Jiang, Yue Ma, Xuanhua He, Bin Lin, Kaixiong Gong, Zhao Zhong, Liefeng Bo, Qifeng Chen, and Harry Yang. Manifold-aware exploration for reinforcement learning in video generation.arXiv preprint arXiv:2603.21872, 2026. 17

arXiv 2026
[8]

Unified personalized reward model for vision generation.arXiv preprint arXiv:2602.02380, 2026

Yibin Wang, Yuhang Zang, Feng Han, Jiazi Bu, Yujie Zhou, Cheng Jin, and Jiaqi Wang. Unified personalized reward model for vision generation.arXiv preprint arXiv:2602.02380, 2026

arXiv 2026
[9]

Genie: Generative interactive environments, 2024

Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Si...

2024
[10]

Gaia-1: A generative world model for autonomous driving, 2023

Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving, 2023

2023
[11]

Unisim: Learning interactive real-world simulators, 2023

Mengjiao Yang, Yifan Lu, Danfei Xu, et al. Unisim: Learning interactive real-world simulators, 2023

2023
[12]

NVIDIA, :, Arslan Ali, Junjie Bai, Maciej Bala, Yogesh Balaji, Aaron Blakeman, Tiffany Cai, Jiaxin Cao, Tianshi Cao, Elizabeth Cha, Yu-Wei Chao, Prithvijit Chattopadhyay, Mike Chen, Yongxin Chen, Yu Chen, Shuai Cheng, Yin Cui, Jenna Diamond, Yifan Ding, Jiaojiao Fan, Linxi Fan, Liang Feng, Francesco Ferroni, Sanja Fidler, Xiao Fu, Ruiyuan Gao, Yunhao Ge, ...

2026
[13]

V-jepa: Video joint embedding predictive architecture, 2024

Adrien Bardes, Quentin Garrido, Jean-Baptiste Alayrac, et al. V-jepa: Video joint embedding predictive architecture, 2024

2024
[14]

V-jepa 2: Self-supervised video models enable understanding, prediction and planning, 2025

Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba, Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xia...

2025
[15]

Scaling rectified flow transformers for high- resolution image synthesis, 2024

Patrick Esser, Sumith Kulal, Andreas Blattmann, et al. Scaling rectified flow transformers for high- resolution image synthesis, 2024

2024
[16]

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

2024
[17]

Coefficients-preserving sampling for reinforcement learning with flow matching, 2025

Feng Wang and Zihao Yu. Coefficients-preserving sampling for reinforcement learning with flow matching, 2025

2025
[18]

Unigrpo: Unified policy optimization for reasoning-driven visual generation, 2026

Jie Liu, Zilyu Ye, Linxiao Yuan, Shenhan Zhu, Yu Gao, Jie Wu, Kunchang Li, Xionghui Wang, Xiaonan Nie, Weilin Huang, and Wanli Ouyang. Unigrpo: Unified policy optimization for reasoning-driven visual generation, 2026

2026
[19]

Improving video generation with human feedback, 2025

Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Menghan Xia, Xintao Wang, Xiaohong Liu, Fei Yang, Pengfei Wan, Di Zhang, Kun Gai, Yujiu Yang, and Wanli Ouyang. Improving video generation with human feedback, 2025

2025
[20]

Pai-bench: A comprehensive benchmark for physical ai, 2025

Fengzhe Zhou, Jiannan Huang, Jialuo Li, Deva Ramanan, and Humphrey Shi. Pai-bench: A comprehensive benchmark for physical ai, 2025. 18

2025
[21]

Rethinking video generation model for the embodied world

Yufan Deng, Zilin Pan, Hongyu Zhang, Xiaojie Li, Ruoqing Hu, Yufei Ding, Yiming Zou, Yan Zeng, and Daquan Zhou. Rethinking video generation model for the embodied world. arXiv preprint arXiv:2601.15282, 2026

arXiv 2026
[22]

Worldcompass: Reinforcement learning for long-horizon world models, 2026

Zehan Wang, Tengfei Wang, Haiyu Zhang, Xuhui Zuo, Junta Wu, Haoyuan Wang, Wenqiang Sun, Zhenwei Wang, Chenjie Cao, Hengshuang Zhao, Chunchao Guo, and Zhou Zhao. Worldcompass: Reinforcement learning for long-horizon world models, 2026

2026
[23]

Agibot world colosseum.https://github.com/OpenDriveLab/ AgiBot-World, 2024

AgiBot World Colosseum contributors. Agibot world colosseum.https://github.com/OpenDriveLab/ AgiBot-World, 2024

2024
[24]

Humanoid everyday: A comprehensive robotic dataset for open-world humanoid manipulation, 2025

Zhenyu Zhao, Hongyi Jing, Xiawei Liu, Jiageng Mao, Abha Jha, Hanwen Yang, Rong Xue, Sergey Zakharor, Vitor Guizilini, and Yue Wang. Humanoid everyday: A comprehensive robotic dataset for open-world humanoid manipulation, 2025

2025
[25]

Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karam- cheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Y...

2025
[26]

Kairos: A native world model stack for physical ai, 2026

Kairos Team, Fei Wang, Shan You, Qiming Zhang, Tao Huang, Zuoyi Fu, Zhisheng Zheng, Yunlong Xi, Feng Lv, Xiaoming Wu, Zeyu Liu, Cong Wan, Pu Li, Ruiqing Yang, Xiaoou Li, Wei Wang, Kangkang Zhu, Yuwei Zhang, Shi Fu, Zheng Zhang, Xiaoning Wu, Xuzeng Fan, Dacheng Tao, and Xiaogang Wang. Kairos: A native world model stack for physical ai, 2026

2026
[27]

Oriane Sim’eoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michael Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien...

2025
[28]

Classifier-free diffusion guidance, 2022

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022

2022
[29]

Qwen3.5: Towards native multimodal agents, February 2026

Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026

2026
[30]

video_quality_category

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 19 A Appendix: Detailed Experimental Cases of Reward Hacking ...

2023

[1] [1]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, ...

2025

[2] [2]

Qwen3-vl technical report, 2025

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

2025

[3] [3]

Flow-grpo: Training flow matching models via online rl, 2025

Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl, 2025

2025

[4] [4]

Dancegrpo: Unleashing grpo on visual generation, 2025

Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, and Ping Luo. Dancegrpo: Unleashing grpo on visual generation, 2025

2025

[5] [5]

Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde, 2026

Junzhe Li, Yutao Cui, Tao Huang, Yinping Ma, Chun Fan, Yiming Cheng, Miles Yang, Zhao Zhong, and Liefeng Bo. Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde, 2026

2026

[6] [6]

Pref-grpo: Pairwise preference reward-based grpo for stable text-to-image reinforcement learning, 2026

Yibin Wang, Zhimin Li, Yuhang Zang, Yujie Zhou, Jiazi Bu, Chunyu Wang, Qinglin Lu, Cheng Jin, and Jiaqi Wang. Pref-grpo: Pairwise preference reward-based grpo for stable text-to-image reinforcement learning, 2026

2026

[7] [7]

Manifold-aware exploration for reinforcement learning in video generation.arXiv preprint arXiv:2603.21872, 2026

Mingzhe Zheng, Weijie Kong, Yue Wu, Dengyang Jiang, Yue Ma, Xuanhua He, Bin Lin, Kaixiong Gong, Zhao Zhong, Liefeng Bo, Qifeng Chen, and Harry Yang. Manifold-aware exploration for reinforcement learning in video generation.arXiv preprint arXiv:2603.21872, 2026. 17

arXiv 2026

[8] [8]

Unified personalized reward model for vision generation.arXiv preprint arXiv:2602.02380, 2026

Yibin Wang, Yuhang Zang, Feng Han, Jiazi Bu, Yujie Zhou, Cheng Jin, and Jiaqi Wang. Unified personalized reward model for vision generation.arXiv preprint arXiv:2602.02380, 2026

arXiv 2026

[9] [9]

Genie: Generative interactive environments, 2024

Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Si...

2024

[10] [10]

Gaia-1: A generative world model for autonomous driving, 2023

Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving, 2023

2023

[11] [11]

Unisim: Learning interactive real-world simulators, 2023

Mengjiao Yang, Yifan Lu, Danfei Xu, et al. Unisim: Learning interactive real-world simulators, 2023

2023

[12] [12]

NVIDIA, :, Arslan Ali, Junjie Bai, Maciej Bala, Yogesh Balaji, Aaron Blakeman, Tiffany Cai, Jiaxin Cao, Tianshi Cao, Elizabeth Cha, Yu-Wei Chao, Prithvijit Chattopadhyay, Mike Chen, Yongxin Chen, Yu Chen, Shuai Cheng, Yin Cui, Jenna Diamond, Yifan Ding, Jiaojiao Fan, Linxi Fan, Liang Feng, Francesco Ferroni, Sanja Fidler, Xiao Fu, Ruiyuan Gao, Yunhao Ge, ...

2026

[13] [13]

V-jepa: Video joint embedding predictive architecture, 2024

Adrien Bardes, Quentin Garrido, Jean-Baptiste Alayrac, et al. V-jepa: Video joint embedding predictive architecture, 2024

2024

[14] [14]

V-jepa 2: Self-supervised video models enable understanding, prediction and planning, 2025

Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba, Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xia...

2025

[15] [15]

Scaling rectified flow transformers for high- resolution image synthesis, 2024

Patrick Esser, Sumith Kulal, Andreas Blattmann, et al. Scaling rectified flow transformers for high- resolution image synthesis, 2024

2024

[16] [16]

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

2024

[17] [17]

Coefficients-preserving sampling for reinforcement learning with flow matching, 2025

Feng Wang and Zihao Yu. Coefficients-preserving sampling for reinforcement learning with flow matching, 2025

2025

[18] [18]

Unigrpo: Unified policy optimization for reasoning-driven visual generation, 2026

Jie Liu, Zilyu Ye, Linxiao Yuan, Shenhan Zhu, Yu Gao, Jie Wu, Kunchang Li, Xionghui Wang, Xiaonan Nie, Weilin Huang, and Wanli Ouyang. Unigrpo: Unified policy optimization for reasoning-driven visual generation, 2026

2026

[19] [19]

Improving video generation with human feedback, 2025

Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Menghan Xia, Xintao Wang, Xiaohong Liu, Fei Yang, Pengfei Wan, Di Zhang, Kun Gai, Yujiu Yang, and Wanli Ouyang. Improving video generation with human feedback, 2025

2025

[20] [20]

Pai-bench: A comprehensive benchmark for physical ai, 2025

Fengzhe Zhou, Jiannan Huang, Jialuo Li, Deva Ramanan, and Humphrey Shi. Pai-bench: A comprehensive benchmark for physical ai, 2025. 18

2025

[21] [21]

Rethinking video generation model for the embodied world

Yufan Deng, Zilin Pan, Hongyu Zhang, Xiaojie Li, Ruoqing Hu, Yufei Ding, Yiming Zou, Yan Zeng, and Daquan Zhou. Rethinking video generation model for the embodied world. arXiv preprint arXiv:2601.15282, 2026

arXiv 2026

[22] [22]

Worldcompass: Reinforcement learning for long-horizon world models, 2026

Zehan Wang, Tengfei Wang, Haiyu Zhang, Xuhui Zuo, Junta Wu, Haoyuan Wang, Wenqiang Sun, Zhenwei Wang, Chenjie Cao, Hengshuang Zhao, Chunchao Guo, and Zhou Zhao. Worldcompass: Reinforcement learning for long-horizon world models, 2026

2026

[23] [23]

Agibot world colosseum.https://github.com/OpenDriveLab/ AgiBot-World, 2024

AgiBot World Colosseum contributors. Agibot world colosseum.https://github.com/OpenDriveLab/ AgiBot-World, 2024

2024

[24] [24]

Humanoid everyday: A comprehensive robotic dataset for open-world humanoid manipulation, 2025

Zhenyu Zhao, Hongyi Jing, Xiawei Liu, Jiageng Mao, Abha Jha, Hanwen Yang, Rong Xue, Sergey Zakharor, Vitor Guizilini, and Yue Wang. Humanoid everyday: A comprehensive robotic dataset for open-world humanoid manipulation, 2025

2025

[25] [25]

Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karam- cheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Y...

2025

[26] [26]

Kairos: A native world model stack for physical ai, 2026

Kairos Team, Fei Wang, Shan You, Qiming Zhang, Tao Huang, Zuoyi Fu, Zhisheng Zheng, Yunlong Xi, Feng Lv, Xiaoming Wu, Zeyu Liu, Cong Wan, Pu Li, Ruiqing Yang, Xiaoou Li, Wei Wang, Kangkang Zhu, Yuwei Zhang, Shi Fu, Zheng Zhang, Xiaoning Wu, Xuzeng Fan, Dacheng Tao, and Xiaogang Wang. Kairos: A native world model stack for physical ai, 2026

2026

[27] [27]

Oriane Sim’eoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michael Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien...

2025

[28] [28]

Classifier-free diffusion guidance, 2022

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022

2022

[29] [29]

Qwen3.5: Towards native multimodal agents, February 2026

Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026

2026

[30] [30]

video_quality_category

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 19 A Appendix: Detailed Experimental Cases of Reward Hacking ...

2023