Pith. sign in

REVIEW 3 major objections 5 minor 44 references

VPT fine-tunes video diffusion models so agents, controlled objects, passive objects, and background keep physically coherent motion without collapsing visual quality.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 15:43 UTC pith:N7VIFVWI

load-bearing objection Solid fine-tuning recipe with real VideoPhy gains; the role-map is the main lever and also the main open risk, not a math failure. the 3 major comments →

arxiv 2607.04653 v2 pith:N7VIFVWI submitted 2026-07-06 cs.CV

Enhancing Video Physical Consistency via Role-aware Joint Training and Modality-decoupled Denoising

Guangting Zheng , Haojing Chen , Hao Li , Jingtao Zhang , Zhen Yang , Xiaosong Jia , Xue Yang , Shaofeng Zhang
show 1 more author
Yanyong Zhang
This is my paper
classification cs.CV
keywords video diffusionphysical consistencyrole-aware representationmodality-decoupled denoisingoptical flowcross-step auto-guidancetext-to-videofine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Modern video generators look sharp frame by frame but still break physics: poured wine flies wrong, gloves miss balls, ripples look fake. Earlier fixes that jointly train on optical flow with the video itself often blur the pretrained look, treat every moving region the same, and accumulate prediction errors at sampling time. VPT is a lightweight fine-tuning recipe that tags entities by physical role, denoises the video and the auxiliary signals at independent noise levels so the auxiliaries act as soft constraints, anneals their loss weight so they do not overpower appearance, and uses an earlier training checkpoint to steer the final model toward stronger dynamics. On standard physical-commonsense benchmarks the method raises semantic adherence and physical scores while VBench quality stays competitive, showing that role structure plus decoupled supervision can inject motion priors without wrecking the original generator.

Core claim

VPT claims that a pretrained video diffusion model can be made substantially more physically consistent by (1) concatenating a four-role map (agent, controlled object, passive object, background) with optical flow, (2) training with independent noise schedules and decaying auxiliary loss so auxiliaries guide rather than dominate, and (3) applying cross-step auto-guidance at inference. On Wan2.1 backbones this lifts VideoPhy SA/PC from 47.7/21.2 to 66.5/25.0 (1.3B) and yields consistent gains on VideoPhy-2 while preserving or improving VBench quality.

What carries the argument

Role-aware joint representation plus modality-decoupled denoising: optical flow and a scalar role map are VAE-encoded, concatenated with video latents, and each modality receives its own diffusion timestep; auxiliary losses are cosine-annealed so they become soft physical constraints rather than hard synchronized targets.

Load-bearing premise

The automatically extracted optical flow, vision-language role labels, and segmentation masks are accurate enough that training on them teaches real physics rather than label noise.

What would settle it

Re-run the same fine-tuning protocol after deliberately corrupting or randomizing the role maps and optical flow; if SA/PC gains on VideoPhy disappear or reverse while visual quality stays high, the claimed physical priors are not the source of the improvement.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Lightweight post-training can raise physical-commonsense scores on material-interaction and action-centric benchmarks without full model re-pretraining.
  • Treating different entity roles separately yields larger gains than optical-flow-only joint training on the same backbone.
  • Independent noise levels and loss annealing reduce the visual collapse that naive joint appearance-motion training produces.
  • Cross-step auto-guidance from an intermediate checkpoint can strengthen dynamics more reliably than guiding on recursively predicted auxiliary latents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If role maps prove brittle on long or multi-agent clips, the same decoupling machinery could be reused with denser 4-D geometry or force-field auxiliaries.
  • The capacity-conflict diagnosis suggests any multi-modal fine-tune of large video models may benefit from independent schedules rather than shared timesteps.
  • A natural next measurement is whether the learned role bias transfers to zero-shot prompts outside the WISA training distribution.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes VPT, a lightweight fine-tuning framework that improves physical consistency of pretrained video diffusion models (Wan2.1-T2V 1.3B/14B). It augments RGB latents with optical flow and a four-class role-aware map (agent, controlled object, passive object, background), trains with modality-decoupled timesteps and cosine-annealed auxiliary losses so auxiliaries act as soft constraints, and applies cross-step auto-guidance at inference using an intermediate checkpoint. On VideoPhy the 1.3B model improves SA/PC from 47.7/21.2 to 66.5/25.0 (relative +39.4%/+17.9%), with further gains on VideoPhy-2 and competitive or improved VBench quality; ablations attribute most of the SA lift to the role map and show naive joint training collapses.

Significance. If the gains reflect genuine entity-level physical priors rather than label or layout artifacts, VPT is a practical and timely contribution: it targets a recognized failure mode of video diffusion (long-range dynamics and contact) with a modular fine-tuning recipe that preserves pretrained visual quality, is demonstrated at two model scales, and is supported by clear motivation experiments (capacity conflict, inference error accumulation) and component ablations. Strengths include external-benchmark evaluation (VideoPhy, VideoPhy-2, VBench), explicit comparison to VideoJAM and full fine-tuning, planned code/checkpoint release, and a limitations section that states dependence on RAFT/Qwen3-VL/SAM3. The work is of clear interest to the video-generation and world-model communities even if some claims need tighter causal support.

major comments (3)
  1. Table 3(b) attributes the dominant SA gain to the role-aware map (baseline 45.1/20.9 → +flow 49.3/22.4 → +role 66.5/25.0). The central physical-prior claim therefore depends on Qwen3-VL role labels and SAM3 masks being accurate enough to teach agent/controlled/passive dynamics rather than VLM/segmentation artifacts. Limitations only notes this dependency; Table 3(c) and Fig. 5 only measure VAE reconstruction fidelity of the maps, not label correctness or causal physical content. Please add (i) quantitative or human validation of role maps on a VideoPhy-relevant subset (especially solid–fluid and multi-object contact), and/or (ii) a control that randomizes or shuffles role labels while keeping masks/flow fixed, to show that the SA/PC lift requires correct role semantics rather than extra spatial structure.
  2. §3.4–3.5 and Appendix A: modality-decoupled training is motivated as converting auxiliaries into soft constraints and mitigating recursive inference error, yet inference still constructs a joint latent with shared timesteps (tv=tf=tr) and recursively predicts flow/role channels (Eqs. 3–4; Inner-Guidance comparison). Table 3(a) shows cross-step auto-guidance improves PC (25.0→26.5) but slightly lowers SA (66.5→64.5). Please clarify what is actually sampled at inference (are auxiliary channels still denoised end-to-end?), report a pure text-only / auxiliary-dropped inference baseline under the same trained weights, and state whether the claimed reduction of recursive error is mainly from annealing during training or from auto-guidance.
  3. Table 1 and §4.3: full fine-tuning of Wan2.1-1.3B can lower PC (21.2→20.9) while VPT raises it, but there is no matched compute/data ablation that applies the same WISA-80K data and LoRA budget without role/flow supervision under the same decoupled schedule. Without that control, part of the SA jump could be domain adaptation to WISA rather than role-aware physics. A “data-only / LoRA-only” row (or re-use of the Table 3(b) baseline with identical protocol) should be stated explicitly next to VPT.
minor comments (5)
  1. No error bars, confidence intervals, or multi-seed variance are reported (checklist item 7). Even a small multi-seed or multi-prompt subsample on VideoPhy would strengthen the large SA claim.
  2. Fig. 1c–d and Fig. 4 are informative but axis labels and the meaning of “standard vs oracle” in 1d could be stated more explicitly in the caption for readers skimming.
  3. Role scalar encoding {0, 85, 170, 255} and auxiliary dropout p are free design choices (§3.3); a short sensitivity note (or appendix) would help reproducibility.
  4. Related work could more sharply contrast VPT with VideoREPA, PhyT2V, and WISA beyond a short list, given WISA-80K is the training set.
  5. Typo/consistency: abstract and intro use “VPT” expanded as Video Physical-consistency Tuning; ensure hyphenation and capitalization are uniform throughout.

Circularity Check

0 steps flagged

No significant circularity: empirical fine-tuning with external extractors and external benchmarks; no prediction reduces by construction to its inputs.

full rationale

VPT is a fine-tuning recipe (role-aware maps + optical flow, modality-decoupled timesteps, auxiliary loss annealing, cross-step auto-guidance) whose claims rest on measured gains on VideoPhy, VideoPhy-2, and VBench. Optical flow (RAFT) and role maps (Qwen3-VL + SAM3) are extracted offline from the WISA-80K training set and used as standard velocity-MSE targets; evaluation metrics are independent automatic evaluators on separate prompts. Equations (1)–(4) define ordinary joint/decoupled denoising and guidance; none equates a reported SA/PC number to a fitted constant or to a quantity defined from the same data. Citations (VideoJAM, Auto-Guidance, RAFT, etc.) supply baselines or tools, not load-bearing uniqueness theorems that force the result. Label-quality risk is an assumption/correctness issue, not circular derivation. Hence score 0 with empty steps.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 3 invented entities

Central claim rests on standard diffusion/flow-matching training plus several design choices and external extractors. Free parameters are training/inference knobs chosen by the authors; axioms are domain assumptions about what optical flow and discrete roles represent; invented entities are the four-role taxonomy and the joint latent construction used as supervision.

free parameters (5)
  • initial auxiliary loss weight λ0 = 0.2
    Set to 0.2 with cosine decay to 0 over K=2000 steps; controls how strongly auxiliary modalities influence fine-tuning.
  • LoRA rank and alpha = 32
    Both set to 32 for attention adapters; capacity of the fine-tune is a free design choice.
  • cross-step auto-guidance strength γ and intermediate checkpoint = γ=1.2, step 1600
    γ=1.2 with 1600-step intermediate model used for best PC; chosen by ablation, not derived.
  • auxiliary dropout probability p = 0.1
    Random zeroing of flow/role latents (p≈0.1 in appendix) for CFG compatibility; hand-set.
  • role scalar encoding values = 0/85/170/255
    Fixed map {A:0, Oc:85, Op:170, B:255} replicated to 3 channels; arbitrary discrete encoding of roles.
axioms (4)
  • domain assumption Pixel-level optical flow plus discrete entity roles are sufficient soft physical priors for long-range interaction consistency in latent video diffusion.
    Core premise of §3.3–3.4; not proven, motivated by failure modes of RGB-only and flow-only training.
  • domain assumption The pretrained video VAE latent space preserves optical-flow and role-map structure well enough for joint supervision (supported by high PSNR/SSIM, Table 3c).
    Justifies encoding F and R with the original video VAE rather than a dedicated encoder.
  • ad hoc to paper Independent timesteps (tv, tf, tr) convert auxiliary modalities from hard synchronized targets into soft constraints that reduce capacity conflict and inference error accumulation.
    Design claim of modality-decoupled denoising (§3.4); empirically supported by ablation vs naive joint training but not theoretically derived.
  • standard math Standard flow-matching / velocity-prediction diffusion training and classifier-free guidance remain valid when input/output projections are expanded and LoRA is applied.
    Inherited from Wan2.1 / flow-matching literature used as backbone.
invented entities (3)
  • Four-role physical role map (agent, controlled object, passive object, background) no independent evidence
    purpose: Provide entity-level physical structure so heterogeneous motion patterns can be supervised separately from dense optical flow.
    Defined in §3.3 with fixed scalar encoding; independent evidence is only the ablation gain on VideoPhy, not an external physical theory.
  • Modality-decoupled joint latent with independent noise levels and auxiliary loss annealing no independent evidence
    purpose: Avoid capacity conflict and recursive auxiliary error while still injecting motion/role priors.
    Training objective in Eq. (2) and §3.4; effectiveness is empirical within this paper.
  • Cross-step auto-guidance for video physical dynamics no independent evidence
    purpose: Amplify physical direction of the final fine-tuned model relative to an earlier checkpoint without relying on noisy auxiliary CFG.
    Adapted from image Auto-Guidance; video-specific use and intermediate-step choice are paper-specific.

pith-pipeline@v1.1.0-grok45 · 25141 in / 3430 out tokens · 28967 ms · 2026-07-11T15:43:34.923256+00:00 · methodology

0 comments
read the original abstract

While modern video diffusion models excel in visual fidelity, maintaining long-range physical consistency remains a formidable challenge. Conventional pixel-reconstruction objectives mainly focus on appearance details and often fail to capture the underlying dynamics of a scene. To mitigate this, recent efforts have integrated auxiliary modalities (e.g., optical flow) to introduce physics priors via joint training with video appearance. However, these methods have three main limitations: (1) they do not distinguish the different motion patterns of different entity types; (2) joint modeling of visual and auxiliary modalities can cause capacity conflicts and weaken the pretrained visual prior; and (3) auxiliary modalities may accumulate errors during inference. To address these issues, we propose \textbf{VPT}, a fine-tuning framework for improving physical consistency in video diffusion models. VPT introduces a role-aware signal that groups entities into agents, controlled objects, passive objects, and background, so that different physical roles can be modeled more clearly. We further propose a modality-decoupled denoising strategy, where the visual and auxiliary channels are assigned independent noise levels. Together with a loss-weight decay strategy, this design makes auxiliary modalities serve as soft constraints rather than strong dependencies, mitigating recursive prediction errors during inference. We also introduce cross-step auto-guidance to further strengthen physical dynamics. Experiments show that VPT improves physical consistency while preserving visual quality, achieving relative gains of 39.4\% in SA and 17.9\% in PC on VideoPhy benchmark over Wan2.1-T2V-1.3B, and consistent improvements on VideoPhy-2 benchmark. The project page is available at https://tom-zgt.github.io/VPT.

Figures

Figures reproduced from arXiv: 2607.04653 by Guangting Zheng, Haojing Chen, Hao Li, Jingtao Zhang, Shaofeng Zhang, Xiaosong Jia, Xue Yang, Yanyong Zhang, Zhen Yang.

Figure 1
Figure 1. Figure 1: Overview of qualitative examples and motivation experiments. modalities provide useful motion priors, existing approaches still face three key limitations. First, they usually model optical flow uniformly and ignore the heterogeneous dynamics of different entities. For example, an agent, a manipulated object, and a passive object often follow different motion patterns. As shown in Figure 1a, the human init… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of VPT. VPT consists of three stages: data preparation, modality-decoupled fine￾tuning, and cross-step auto-guidance inference. First, optical flows and role-aware maps are extracted from training videos as auxiliary physical cues. Then, VPT finetunes the video diffusion model with modality-decoupled denoising and auxiliary loss annealing, decoupling visual and auxiliary denoising while gradually … view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison on physically grounded video generation. Wan2.1-T2V-1.3B and VideoJAM (based on Wan2.1-T2V-1.3B) often suffer from physically implausible artifacts, such as unstable deformation, inconsistent ripples, and incoherent interaction dynamics. VPT (based on Wan2.1-T2V-1.3B) better preserves object geometry and generates more realistic fluid responses and motion trajectories, demonstrating … view at source ↗
Figure 4
Figure 4. Figure 4: Ablation study on classifier-free guidance (CFG) scales. Left: text-only CFG sweep. Right: CFG combinations over text, optical-flow, and role-aware latents. Directly applying CFG to auxiliary modalities does not consistently outperform text-only guidance, as inference-time auxiliary predictions can contain larger errors and thus fail to provide reliable guidance. Effect of training and inference strategy … view at source ↗
Figure 5
Figure 5. Figure 5: VAE reconstruction of RGB video, optical flow, and role-aware maps. For each modality, we show the original input and its VAE reconstruction. The reconstructed optical flow retains the original motion direction and temporal trajectory, and the reconstructed role-aware map preserves entity regions and role labels. structure needed for role-aware physical supervision. Visual comparisons of the original and r… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative results. The first row displays the outcomes of Wan2.1-1.3B, the second row [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative results. The first row displays the outcomes of Wan2.1-1.3B, the second row [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 12 linked inside Pith

  1. [1]

    L. S. Aira, A. Montanaro, E. Aiello, D. Valsesia, and E. Magli. Motioncraft: Physics-based zero-shot video generation, 2024

    2024

  2. [2]

    S. Bai, Y . Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y . Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y . Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang,...

    2025

  3. [3]

    Bansal, Z

    H. Bansal, Z. Lin, T. Xie, Z. Zong, M. Yarom, Y . Bitton, C. Jiang, Y . Sun, K.-W. Chang, and A. Grover. Videophy: Evaluating physical commonsense for video generation, 2024

    2024

  4. [4]

    Bansal, C

    H. Bansal, C. Peng, Y . Bitton, R. Goldenberg, A. Grover, and K.-W. Chang. Videophy-2: A challenging action-centric physical commonsense evaluation in video generation, 2025

    2025

  5. [5]

    Bar-Tal, H

    O. Bar-Tal, H. Chefer, O. Tov, C. Herrmann, R. Paiss, S. Zada, A. Ephrat, J. Hur, G. Liu, A. Raj, Y . Li, M. Rubinstein, T. Michaeli, O. Wang, D. Sun, T. Dekel, and I. Mosseri. Lumiere: A space-time diffusion model for video generation, 2024

    2024

  6. [6]

    Blattmann, T

    A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y . Levi, Z. English, V . V oleti, A. Letts, V . Jampani, and R. Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    Blattmann, R

    A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22563–22575, 2023

    2023

  8. [8]

    Carion, L

    N. Carion, L. Gustafson, Y .-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V . Alwala, H. Khedr, A. Huang, J. Lei, T. Ma, B. Guo, A. Kalla, M. Marks, J. Greer, M. Wang, P. Sun, R. Rädle, T. Afouras, E. Mavroudi, K. Xu, T.-H. Wu, Y . Zhou, L. Momeni, R. Hazra, S. Ding, S. Vaze, F. Porcher, F. Li, S. Li, A. Kamath, H. K. Cheng, P. Dollár, N. Ravi, K. Sae...

    2026

  9. [9]

    Chefer, U

    H. Chefer, U. Singer, A. Zohar, Y . Kirstain, A. Polyak, Y . Taigman, L. Wolf, and S. Sheynin. Videojam: Joint appearance-motion representations for enhanced motion generation in video models, 2025

    2025

  10. [10]

    S. Chen, C. Ge, Y . Zhang, Y . Zhang, F. Zhu, H. Yang, H. Hao, H. Wu, Z. Lai, Y . Hu, et al. Goku: Flow based video generative foundation models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 23516–23527, 2025

    2025

  11. [11]

    Z. Chen, T. Liu, L. Zhuo, J. Ren, Z. Tao, H. Zhu, F. Hong, L. Pan, and Z. Liu. 4dnex: Feed-forward 4d generative modeling made easy.arXiv preprint arXiv:2508.13154, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    H. Deng, T. Pan, H. Diao, Z. Luo, Y . Cui, H. Lu, S. Shan, Y . Qi, and X. Wang. Autoregressive video generation without vector quantization.arXiv preprint arXiv:2412.14169, 2024. 10

    Pith/arXiv arXiv 2024

  13. [13]

    Y . Guo, C. Yang, A. Rao, Z. Liang, Y . Wang, Y . Qiao, M. Agrawala, D. Lin, and B. Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning, 2024

    2024

  14. [14]

    J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, and T. Salimans. Imagen video: High definition video generation with diffusion models.arXiv preprint arXiv:2210.02303, 2022

    Pith/arXiv arXiv 2022

  15. [15]

    J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models, 2020

    2020

  16. [16]

    Ho and T

    J. Ho and T. Salimans. Classifier-free diffusion guidance, 2022

    2022

  17. [17]

    J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458, 2022

    Pith/arXiv arXiv 2022

  18. [18]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models, 2021

    2021

  19. [19]

    Huang, Y

    J. Huang, Y . Zhang, X. He, Y . Gao, Z. Cen, B. Xia, Y . Zhou, X. Tao, P. Wan, and J. Jia. Uni- tyvideo: Unified multi-modal multi-task learning for enhancing world-aware video generation. arXiv preprint arXiv:2512.07831, 2025

    arXiv 2025

  20. [20]

    Huang, Y

    Z. Huang, Y . He, J. Yu, F. Zhang, C. Si, Y . Jiang, Y . Zhang, T. Wu, Q. Jin, N. Chanpaisit, Y . Wang, X. Chen, L. Wang, D. Lin, Y . Qiao, and Z. Liu. Vbench: Comprehensive benchmark suite for video generative models, 2023

    2023

  21. [21]

    B. Kang, Y . Yue, R. Lu, Z. Lin, Y . Zhao, K. Wang, G. Huang, and J. Feng. How far is video generation from world model: A physical law perspective, 2024

    2024

  22. [22]

    Karras, M

    T. Karras, M. Aittala, T. Kynkäänniemi, J. Lehtinen, T. Aila, and S. Laine. Guiding a diffusion model with a bad version of itself, 2024

    2024

  23. [23]

    W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    B. Lin, Y . Ge, X. Cheng, Z. Li, B. Zhu, S. Wang, X. He, Y . Ye, S. Yuan, L. Chen, et al. Open-sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Lipman, R

    Y . Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling, 2023

    2023

  26. [26]

    J. Liu, J. Han, B. Yan, F. Zhu, X. Wang, Y . Jiang, B. PENG, Z. Yuan, et al. Infinitystar: Unified spacetime autoregressive modeling for visual generation. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems

  27. [27]

    S. Liu, Z. Ren, S. Gupta, and S. Wang. Physgen: Rigid-body physics-grounded image-to-video generation. InEuropean Conference on Computer Vision, pages 360–378, 2024

    2024

  28. [28]

    Long, Y .-C

    X. Long, Y .-C. Guo, C. Lin, Y . Liu, Z. Dou, L. Liu, Y . Ma, S.-H. Zhang, M. Habermann, C. Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9970–9980, 2024

    2024

  29. [29]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization, 2019

    2019

  30. [30]

    X. Ma, Y . Wang, G. Jia, X. Chen, Z. Liu, Y .-F. Li, C. Chen, and Y . Qiao. Latte: Latent diffusion transformer for video generation.arXiv preprint arXiv:2401.03048, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    F. Meng, J. Liao, X. Tan, W. Shao, Q. Lu, K. Zhang, Y . Cheng, D. Li, Y . Qiao, and P. Luo. To- wards world simulator: Crafting physical commonsense-based benchmark for video generation, 2024

    2024

  32. [32]

    Motamed, L

    S. Motamed, L. Culp, K. Swersky, P. Jaini, and R. Geirhos. Do generative video models learn physical principles from watching videos?, 2025. 11

    2025

  33. [33]

    Peebles and S

    W. Peebles and S. Xie. Scalable diffusion models with transformers, 2023

    2023

  34. [34]

    Rombach, A

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models, 2022

    2022

  35. [35]

    Singer, A

    U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni, D. Parikh, S. Gupta, and Y . Taigman. Make-a-video: Text-to-video generation without text-video data.arXiv preprint arXiv:2209.14792, 2022

    Pith/arXiv arXiv 2022

  36. [36]

    Teed and J

    Z. Teed and J. Deng. Raft: Recurrent all-pairs field transforms for optical flow, 2020

    2020

  37. [37]

    J. Wang, A. Ma, K. Cao, J. Zheng, Z. Zhang, J. Feng, S. Liu, Y . Ma, B. Cheng, D. Leng, Y . Yin, and X. Liang. Wisa: World simulator assistant for physics-aware text-to-video generation, 2025

    2025

  38. [38]

    S. Wang, R. Qian, Y . Lyu, Q. Liu, L. Zou, J. Qin, S. Liu, and C. Shan. Runawayevil: Jailbreaking the image-to-video generative models, 2025

    2025

  39. [39]

    WanTeam, A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng, J. Wang, J. Zhang, J. Zhou, J. Wang, J. Chen, K. Zhu, K. Zhao, K. Yan, L. Huang, M. Feng, N. Zhang, P. Li, P. Wu, R. Chu, R. Feng, S. Zhang, S. Sun, T. Fang, T. Wang, T. Gui, T. Weng, T. Shen, W. Lin, W. Wang, W. Wang, W. Zhou, W. Wang, W. Shen, W. Yu, X. Shi, X...

    Pith/arXiv arXiv 2025

  40. [40]

    D. Xi, J. Wang, Y . Liang, X. Qiu, Y . Huo, R. Wang, C. Zhang, and X. Li. Omnivdiff: Omni controllable video diffusion for generation and understanding.arXiv preprint arXiv:2504.10825, 2025

    arXiv 2025

  41. [41]

    Q. Xue, X. Yin, B. Yang, and W. Gao. Phyt2v: Llm-guided iterative self-refinement for physics- grounded text-to-video generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025

    2025

  42. [42]

    Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y . Yang, W. Hong, X. Zhang, G. Feng, D. Yin, Y . Zhang, W. Wang, Y . Cheng, B. Xu, X. Gu, Y . Dong, and J. Tang. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Limitations

    X. Zhang, J. Liao, S. Zhang, F. Meng, X. Wan, J. Yan, and Y . Cheng. Videorepa: Learning physics for video generation through relational alignment with foundation models.arXiv preprint arXiv:2505.23656, 2025. 12 A Comparison with Inner-Guidance Inner-Guidance.Following VideoJAM [ 9], we adapt Inner-Guidance to VPT by applying separate guidance weights to ...

    Pith/arXiv arXiv 2025

  44. [44]

    Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or ...