REVIEW 5 major objections 6 minor 22 references
Fine-tuning a small optical-flow decoder on simulated fluid videos gives a frozen video generator physically coherent pouring and sloshing.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 02:46 UTC pith:TEQXN2N5
load-bearing objection The paper has a real dataset and a clever architecture, but the central claim—that flow supervision drives the physics gains—is untested because the only baseline is the frozen backbone; the missing RGB-only fine-tune ablation leaves the result unproven. the 5 major comments →
Physics-Grounded Fluid Video Generation with a Simulation Dataset and Dual-Stream Optical-Flow Supervision
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that the persistent failure of video diffusion models to respect fluid physics stems from a supervision gap, not an architectural limit: without explicit motion targets, models imitate fluid appearance instead of dynamics. The discovery is that this gap can be closed by decoupling a frozen pretrained backbone from a trainable dual-stream decoder—an RGB decoder plus a lightweight optical-flow decoder, fused through zero-initialized convolutions so the pretrained generation starts undisturbed. Training only the two decoders on a mixed dataset of physics-simulated pouring and sloshing videos and curated real footage, with a joint reconstruction and flow loss, yields
What carries the argument
The central mechanism is the dual-stream decoder: the standard RGB decoder plus a parameter-light Optical-Flow Decoder that consumes the same diffusion-transformer latent, with the flow branch's intermediate features injected into the RGB stream through zero-initialized convolutions. Because the fusion convolutions start at zero, the flow branch has no effect at initialization and is gradually blended in as training proceeds, preserving the pretrained backbone's output quality. The flow decoder is supervised with a per-pixel end-point-error loss, an edge-aware spatial smoothness term, and a temporal smoothness term, using optical-flow pseudo-ground-truth extracted from simulated and real vid
Load-bearing premise
The load-bearing premise is that the optical-flow estimates used as pseudo-ground-truth supervision accurately capture true fluid motion; if the estimator is unreliable at fluid boundaries, on transparent or occluded regions, the model may learn to imitate the estimator's errors rather than physics.
What would settle it
Cross-check generated videos with a different optical-flow estimator than the one used for supervision, or directly test mass conservation: if the flow decoder's output diverges sharply from the second estimator at fluid–container boundaries, or if the liquid volume implied by the flow field disagrees with visible level changes, then the model has internalized the supervisor's biases rather than physics.
If this is right
- If correct, video generators can gain physical plausibility without retraining the backbone: only about 2% of decoder parameters need fine-tuning.
- The improvement holds across a 10x scale change (1.3B to 14B parameters), suggesting the approach is scale-agnostic.
- Physical-commonsense gains persist on a domain-shifted benchmark whose first frames come from a different text-to-image model, indicating generalization beyond the training distribution.
- The sub-pixel flow self-consistency implies the model carries an explicit, readable motion representation, which could support controllable generation or physics verification.
- Human raters prefer the fine-tuned videos for realism, so the effect is perceptible to observers, not only reflected in automated metrics.
Where Pith is reading between the lines
- Because the supervision is pseudo-ground-truth from an optical-flow estimator, the 'physics' the model learns is bounded by that estimator's accuracy; a testable extension is training with simulator-derived ground-truth flow (e.g., projected particle velocities) to see whether the in-distribution end-point error drops further or whether estimator biases are being imitated.
- The zero-initialized fusion likely acts as an implicit curriculum where the flow branch's influence ramps up over training; ablating the fusion schedule would reveal how much of the gain comes from the motion objective versus the fine-tuning itself.
- The same dual-stream recipe—frozen backbone plus a supervised motion-decoding branch—could transfer to other physics domains such as deformable solids, smoke, or granular flows, wherever a simulation or estimator can provide dense motion targets.
- The automated physical-commonsense score used for evaluation is itself a proxy; using it as a reward for reinforcement-learning post-training, as the authors suggest, would test whether these physical-plausibility gains compound or plateau.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses physical implausibility in fluid video generation by (1) constructing a training dataset of 1,638 MPM-simulated pouring/sloshing clips and 2,320 keyword-filtered real pouring clips, with a 1,515-video real test set and an 18-prompt first-frame generalization benchmark, and (2) fine-tuning a frozen Wan2.1 image-to-video diffusion transformer with an additional lightweight optical-flow decoder branch, fused into the RGB decoder via zero-initialized convolutions. The joint objective combines RGB reconstruction with end-point-error and smoothness losses on RAFT-extracted pseudo-ground-truth flow. Experiments report VideoPhy-2 Physical-Commonsense and Video-Quality gains at 1.3B and 14B scales, a comparison against CogVideoX-5B on a 51-video sample, a 36-participant human study, and an optical-flow self-consistency probe with EPE values down to 0.54 pixels.
Significance. The core idea — injecting explicit optical-flow supervision through a separate decoding branch with zero-convolution fusion — is architecturally appealing and consistent with the stated goal of improving motion consistency without retraining a large backbone. The simulated dataset with swept physical parameters is a potentially reusable asset, and the two-scale evaluation is a positive feature. If the central attribution claim were supported, the method would be a practical, inexpensive way to improve physical plausibility in frozen video diffusion models. However, the current evidence does not isolate the effect of the flow branch from the effect of fine-tuning on the new dataset, and the flow-accuracy evaluation is partly circular because RAFT supplies both the training supervision and the evaluation reference. These issues are load-bearing for the novelty and interpretation of the results.
major comments (5)
- [Sec. 3.2, Sec. 4.2, Table 1] The claimed attribution to dual-stream flow supervision is not tested. The only baseline is the completely frozen Wan2.1 backbone. Fine-tuning updates the RGB decoder, the optical-flow decoder, and the zero-convolution fusion, while Wan2.1 updates nothing. The sentence in Sec. 4.2 calling this 'an internal ablation' is therefore misleading: the comparison confounds dataset fine-tuning with the addition of the flow branch and flow loss. A control that fine-tunes only the RGB decoder on the same 3,958 training videos, with the same partial freezing and hyperparameters while omitting the flow decoder and L_f, is required to support the claim that the flow branch is responsible for the improvements in Tables 1–2. Without this control, the observed gains could be entirely due to exposure to the fluid dataset, and the paper's central novelty would collapse to 'fine-tuning on a curated dataset
- [Sec. 3.1, Sec. 4.5, Table 4] The optical-flow accuracy evaluation is partly circular. RAFT is used to generate pseudo-ground-truth flow for training supervision (Sec. 3.1), and in Sec. 4.5 RAFT extracted from the generated video is used as the reference to measure the flow decoder's EPE. Agreement between the flow decoder and RAFT on generated video can reflect the model's success at imitating RAFT's estimator, not an independent verification of physical correctness. The abstract's claim that low EPE 'confirms the model has internalized a coherent motion prior' overstates what the experiment can show. A non-circular check for the simulated portion of the dataset would use simulator-known ground-truth velocities or an independently trained flow estimator; for real videos, the claim should be softened to self-consistency with a reference estimator.
- [Tables 1–3, Sec. 4.2, Sec. 4.4] No error bars, confidence intervals, or significance tests are reported for any of the quantitative comparisons. This matters because the reported gains are often small: e.g., the iStock-Fluid 1.3B PC gain is 3.85 points and the Flux-Fluid 1.3B PC gain is 5.88 points on a 17-prompt benchmark. Table 2 uses only 51 videos, and Table 3 reports a 4-point human rating with standard deviations up to 1.31 for a mean of 2.87. Without variance estimates or paired tests, the claimed 'consistent gains' across scales and test sets cannot be distinguished from sampling noise. I request either per-benchmark confidence intervals, or the raw scores and a statistical comparison, for at least Tables 1 and 2.
- [Sec. 4.5, 'Flow accuracy' discussion] The EPE numbers in Table 4 are presented as evidence that the flow branch is 'not decoupled from what the RGB stream actually renders.' But the comparison is between the flow decoder's output and RAFT re-extracted from the RGB output. Since both are computed on the same model's output, low EPE is expected if the flow branch has learned to output a function of the latent that correlates with the RGB decoder's output, even if the motion itself is not physically correct. The >1px percentages are nontrivial (14.7% in-distribution, 21.8% out-of-distribution), and the claim that these are 'concentrated at high-frequency fluid–container boundaries' is not supported by a per-pixel breakdown. Please provide a location-wise error map or a quantitative boundary-region analysis, and discuss what this implies for the physical-grounding claim.
- [Sec. 3.1, 'Real video collection'] The description of the real-video curation is underspecified for reproducibility. The 'large stock-footage corpus' is said to contain 10,000 professionally shot videos with automatically generated captions, but the source is not named, and the keyword filter yields 3,835 videos of which 2,320 are used for training and 1,515 for testing. This split is reasonable, but the absence of a URL or provider, and the reliance on caption quality, makes it impossible for others to reconstruct the benchmark. If the stock corpus is proprietary, the paper should at least release the video identifiers or the keyword list and filtering procedure, and should state whether the 1,515-video test set is permanently available.
minor comments (6)
- [Sec. 1, Abstract] The abstract states 'improves VideoPhy-2 Physical-Commonsense and Video-Quality scores over the frozen backbone by up to 8.75 and 4.65 points.' The 4.65-point VQ gain should be checked: Table 1 shows VQ improvements of 3.08 (1.3B iStock), 3.75 (14B iStock), and 5.88 (Flux-Fluid). The 4.65 value does not match any number in Table 1, unless it refers to a different comparison. Please clarify.
- [Sec. 4.2, Table 2] The 51-video sample used for the comparison with CogVideoX-5B should be described: how were the 51 videos selected from the 1,515-video iStock-Fluid set? Was this selection random, and is it fixed for all models? Without this detail, the 'outperforms a leading open competitor' claim is hard to interpret.
- [Sec. 4.4, Table 3] The human study reports only means and standard deviations. Standard deviations are large relative to mean differences (Ours vs. Wan2.1 differ by 0.62, while std devs are 1.05–1.31). A paired test would strengthen the conclusion, and the study design (10 scenes, 36 raters, 3 methods per scene) is well suited to a mixed-effects model.
- [Sec. 3.4, Algorithm 1] The algorithm mentions 'the timestep-dependent loss weighting scheme of the base model [1]' but does not specify how w(t) is computed. Since the proposed loss adds a flow term, it is important to know whether the same weighting is applied to L_f. Please state this explicitly.
- [Sec. 3.2, Conditioning] The conditioning mechanism uses the 10th frame as an intermediate conditioning frame via a binary mask. It is not clear whether this mask is also used at inference for all generations, or whether the model can generate without the intermediate frame. Please clarify the inference-time protocol.
- [Fig. 2] The figure's caption says 'Representative frame sequences from the simulated (top) and real (bottom) training data,' but the top row is likely the RGB frames and the bottom row the masked flow fields. The ordering and labels should be made more explicit to avoid confusion.
Circularity Check
The optical-flow 'internalized motion prior' evaluation is partly circular: RAFT supplies both the training target for the flow decoder and the reference at test time; the central PC/VQ gains remain externally grounded.
specific steps
-
fitted input called prediction
[Sec. 3.1 (pseudo-flow supervision) and Sec. 4.5 (Optical Flow Accuracy)]
"We then extract dense optical flow with a pretrained RAFT model [3] for every training video to serve as pseudo ground-truth motion supervision. ... We extract RAFT flow from our model’s own generated video and compare it, as a reference, to the flow predicted by the Optical-Flow Decoder for the same sample."
The Optical-Flow Decoder is trained to minimize L_epe(f, \hat f) (Eq. 3), where f is the RAFT-extracted pseudo-ground-truth flow on every training video. The Sec. 4.5 'direct optical-flow read-out' then measures EPE between the decoder's \hat f and RAFT re-extracted from the model's own generated video. For in-distribution samples this is largely a consistency check of the learned RAFT-mimicking mapping, not an independent test of physical correctness. The abstract's conclusion that the model has 'internalized a coherent motion prior rather than merely improving surface appearance' therefore rests in part on comparing the model to its own training target.
full rationale
The one clear circular element is the flow-accuracy probe: the same RAFT estimator defines the pseudo-ground-truth supervision in Sec. 3.1 and serves as the reference in the Sec. 4.5 EPE evaluation, so low in-distribution EPE is partly guaranteed by construction. The central quantitative claims, however, are anchored to the external VideoPhy-2 PC/VQ benchmark and a 36-participant blind human study, which do not depend on the paper's own fitted values, and the comparison to CogVideoX-5B provides additional independent reference points. The paper's 'internal ablation' wording in Sec. 4.2 is not a circularity but a confound: Ours differs from the frozen Wan2.1 in both RGB-decoder fine-tuning and the new flow branch, so the PC/VQ gains are not isolated to the optical-flow supervision without an RGB-only fine-tune control. Similarly, the use of the authors' own VersaGauss framework [6] for simulation is a dependency rather than a circular derivation. Overall the central contribution has independent empirical content, so the circularity score is moderate (4), reflecting the self-referential flow evaluation rather than a collapse of the main result into its inputs.
Axiom & Free-Parameter Ledger
free parameters (8)
- lambda_ss =
0.01
- lambda_ts =
0.05
- lambda_kl =
0.1
- conditioning_frame_index =
10
- dataset_mix =
1,638 simulated + 2,320 real = 3,958
- flow_decoder_width =
narrower (unspecified)
- train_epochs =
20
- learning_rate =
1e-5
axioms (6)
- domain assumption RAFT optical-flow estimates provide accurate pseudo-ground-truth motion for training and evaluation on fluid videos.
- domain assumption The MPM-based multiphase simulator of [6] produces physically correct pouring/sloshing dynamics.
- domain assumption VideoPhy-2's automated PC/VQ scorer is a valid measure of physical commonsense.
- ad hoc to paper Zero-initialized convolutions fuse the flow branch into the RGB decoder without destabilizing the pretrained model at the start of training.
- ad hoc to paper The frozen encoder, DiT blocks, and text encoder contain enough information about fluid motion in their latent that decoder-only fine-tuning can recover physically plausible dynamics.
- domain assumption Optical flow is a physically meaningful training signal for fluid dynamics.
read the original abstract
Video diffusion models generate visually compelling content but routinely violate elementary physics when the subject involves fluids: liquid columns break apart in mid-air, container water levels fail to rise as liquid is poured in, and splashes disperse without regard to momentum or gravity. We attribute this gap to the fact that large-scale video-text corpora contain almost no explicit motion supervision, so models learn to imitate fluid appearance rather than dynamics. We address this with two contributions. First, we build a physics-simulation fluid dataset combining 1,638 MPM-simulated pouring/sloshing videos with 2,320 keyword-filtered real pouring videos mined from stock footage, plus two held-out test sets: a 1,515-video real-video benchmark and an 18-prompt text-to-first-frame generalization benchmark. Second, we introduce a dual-stream image-to-video architecture built on a pretrained diffusion-transformer video generator. It augments the standard RGB decoder with a lightweight Optical-Flow Decoder branch trained with explicit end-point-error and smoothness losses, fused into the RGB stream via zero-initialized convolutions so the pretrained backbone starts undisturbed. Only the two decoders are updated; the encoder, temporal transformer, and text encoder remain frozen. Across two model scales (1.3B and 14B) and two test sets, our method improves VideoPhy-2 Physical-Commonsense and Video-Quality scores over the frozen backbone by up to 8.75 and 4.65 points, outperforms a leading open competitor, and is preferred by human raters in a blind study. A direct optical-flow read-out evaluation further shows an end-point error as low as 0.54 pixels in-distribution, confirming the model has internalized a coherent motion prior rather than merely improving surface appearance.
Figures
Reference graph
Works this paper leans on
-
[1]
T. Wan, A. Wang, B. Ai, et al. Wan: Open and ad- vanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025
Pith/arXiv arXiv 2025
-
[2]
Z. Yang, J. Teng, W. Zheng, et al. CogVideoX: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024
Pith/arXiv arXiv 2024
-
[3]
Teed and J
Z. Teed and J. Deng. RAFT: Recurrent all-pairs field trans- forms for optical flow. InEuropean Conference on Computer Vision (ECCV), pages 402–419, 2020
2020
-
[4]
H. Bansal, C. Peng, Y . Bitton, et al. VideoPhy-2: A chal- lenging action-centric physical commonsense evaluation in video generation.arXiv preprint arXiv:2503.06800, 2025
Pith/arXiv arXiv 2025
-
[5]
Batifol, A
S. Batifol, A. Blattmann, F. Boesel, et al. FLUX.1 Kontext: Flow matching for in-context image generation and editing in latent space.arXiv e-prints, arXiv:2506, 2025
2025
-
[6]
R. Su, L. Yang, X. Xie, J.-H. Lai, et al. VersaGauss: A ver- satile framework for generating multiphase dynamics with 3D Gaussians.IEEE International Conference on Multime- dia and Expo (ICME), in press
-
[7]
S. Liu, Z. Ren, S. Gupta, et al. PhysGen: Rigid-body physics-grounded image-to-video generation. InEuropean Conference on Computer Vision (ECCV), pages 360–378. Springer, 2024
2024
-
[8]
T. Xie, Z. Zong, Y . Qiu, et al. PhysGaussian: Physics- integrated 3D Gaussians for generative dynamics. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4389–4398, 2024
2024
-
[9]
Y . Feng, X. Feng, Y . Shang, et al. Gaussian Splashing: Uni- fied particles for versatile motion synthesis and rendering. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 518–529, 2025
2025
-
[10]
Zhang, H.-X
T. Zhang, H.-X. Yu, R. Wu, et al. PhysDreamer: Physics- based interaction with 3D objects via video generation. In European Conference on Computer Vision (ECCV), pages 388–406. Springer, 2024
2024
-
[11]
Huang, H
T. Huang, H. Zhang, Y . Zeng, et al. DreamPhysics: Learn- ing physics-based 3D dynamics with video diffusion priors. InProceedings of the AAAI Conference on Artificial Intelli- gence, volume 39, pages 3733–3741, 2025
2025
-
[12]
J. Lin, Z. Wang, D. Xu, et al. Phys4DGen: Physics- compliant 4D generation with multi-material composition perception. InProceedings of the 33rd ACM International Conference on Multimedia, pages 10398–10407, 2025
2025
-
[13]
J. Ho, T. Salimans, A. Gritsenko, et al. Video diffusion mod- els. InAdvances in Neural Information Processing Systems (NeurIPS), volume 35, pages 8633–8646, 2022
2022
-
[14]
U. Singer, A. Polyak, T. Hayes, et al. Make-A-Video: Text- to-video generation without text-video data.arXiv preprint arXiv:2209.14792, 2022
Pith/arXiv arXiv 2022
- [15]
-
[16]
Liang, Y
Y . Liang, Y . Fang, R. Li, et al. Integrating reinforcement learning with visual generative models: foundations and ad- vances.Vicinagearth, 3(1):2, 2026
2026
-
[17]
Y . Liang, X. Zhan, H. Huang, C. Zhang, and X. Li. From world action models to embodied brains: A roadmap for open-world physical intelligence.arXiv preprint arXiv:2607.11689, 2026
Pith/arXiv arXiv 2026
-
[18]
R. Li, Y . Liang, Z. Ni, H. Huang, C. Zhang, and X. Li. Grow- ing with the generator: Self-paced GRPO for video genera- tion.arXiv preprint arXiv:2511.19356, 2025
Pith/arXiv arXiv 2025
-
[19]
Z. Ni, Y . Liang, R. Li, Y . Zhou, H. Huang, C. Zhang, and X. Li. Seeing what matters: Visual preference policy optimiza- tion for visual generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27260–27269, 2026
2026
-
[20]
R. Li, K. Hao, Y . Liang, H. Huang, C. Zhang, Y . Gu, and X. Li. Learning to credit the right steps: Objective-aware process optimization for visual generation.arXiv preprint arXiv:2604.19234, 2026
Pith/arXiv arXiv 2026
-
[21]
R. Li, B. Li, Y . Liang, H. Huang, C. Zhang, and X. Li. Reward-aware trajectory shaping for few-step visual genera- tion.arXiv preprint arXiv:2604.14910, 2026
Pith/arXiv arXiv 2026
-
[22]
R. Liu, Y . Liang, H. Huang, T. Yu, and C. Zhang. Learn- ing what to trust: Bayesian prior-guided optimization for visual generation. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 34408–34417, 2026. 8
2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.