{"id":"200565b7-0125-4744-a1de-72d1f1a86a7a","arxiv_id":"2411.15540","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"MotionPrompt improves temporal consistency in text-to-video diffusion models by optimizing learnable prompt tokens during sampling, guided by an optical-flow discriminator.","lead":"This paper describes a way to make AI-generated videos look more consistent between frames by adjusting the text prompt during generation. It uses a discriminator trained on real and AI-made optical flows to guide the tweaking, and reports gains on several standard video-quality metrics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The optical-flow discriminator is trained on clean, complete flows but applied to partially denoised intermediate estimates; no evidence shows its gradients transfer, and the Table 3 ablation makes its marginal benefit ambiguous.","rationale":"The reader identified the train/inference gap of the discriminator as the load-bearing premise, and I agree. The method's novelty is the use of a trained optical-flow discriminator to guide prompt embeddings; if that discriminator is not informative on partially denoised Tweedie estimates, the claimed causal mechanism is unsupported. The paper itself acknowledges the gap in Section 4.1 but offers no transfer analysis. The Table 3 ablation weakens confidence further: adding the discriminator yields negligible or negative metric differences. A random-discriminator control and an AUC measurement on intermediate flows would settle causality. These concerns do not require changing the reader's CONDITIONAL verdict: the paper is plausible and addressable, but the central mechanism needs direct evidence before the claim can be accepted. I therefore keep the verdict unchanged.","tokens_in":13612,"tokens_out":8092,"duration_ms":76658,"concrete_test":"Run the MotionPrompt pipeline on AnimateDiff with a fixed randomly initialized discriminator in place of the trained one, keeping all hyperparameters in Table 5 and the same seeds. If VBench subject/temporal-flicker/motion-smoothness gains over baseline persist, the trained discriminator is not causally responsible for the improvements; if they vanish, train/inference transfer matters. As a second diagnostic, compute the trained discriminator's AUC on optical flows from the exact Tweedie-decoded intermediate latents at t = 5, 10, 15 (real vs generated); AUC near 0.5 on these inputs would directly confirm the out-of-distribution gap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"MotionPrompt's central mechanism is the gradient of ℓdisc (Eq. 13) with respect to the learnable tokens T, computed through VAE-decoded Tweedie estimates at intermediate timesteps. But the discriminator is trained (Sec. 4.1) only on optical flows of complete, clean real/generated videos. At inference, Algorithm 2 feeds it flows of partially denoised latents at t in the optimization range (e.g., 3 < t < 15), which contain residual noise, blur, and artifacts. The authors explicitly note this gap but supply no analysis that the discriminator remains informative on this out-of-distribution input. If the discriminator's decisions do not transfer, the gradient in Eq. 13 is essentially arbitrary and the prompt optimization is guided by noise. This concern is not merely formal: Table 3 rows (a) vs (c) show that adding the discriminator (λ1 = 0 → λ1 = 1) improves subject consistency by only 0.0024 while decreasing motion smoothness (0.9643 → 0.9599) and dynamic degree (0.4365 → 0.4125), so the empirical evidence that the trained discriminator is the driver of the reported gains is thin.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MotionPrompt, an inference-time guidance method for text-to-video diffusion models. It appends learnable token embeddings to the text prompt and optimizes them during reverse sampling using gradients from a discriminator that distinguishes optical flows of real videos from those of generated videos. The method is evaluated on LaVie, AnimateDiff, and VideoCrafter2 using VBench metrics, ablations, and a user study. The paper reports improvements in subject consistency, motion smoothness, and temporal flickering, while dynamic degree decreases for all models and overall consistency decreases for two of the three models.","tokens_in":13938,"tokens_out":8515,"duration_ms":71119,"significance":"If the results hold, the idea of controlling global video properties by optimizing prompt tokens guided by an optical-flow discriminator is a novel and potentially useful addition to inference-time video guidance. Strengths include: no retraining of the video diffusion model; gradient computation on only a subset of frames; orthogonality to existing methods such as FreeInit; and the inclusion of ablations, cross-discriminator generalization, token-variation analysis, and a user study. However, the quantitative evidence is currently thin: gains are small, no error bars or significance tests are reported, hyperparameters are tuned per model, and the discriminator is applied to out-of-distribution inputs without validation. These issues must be addressed before the central claim can be accepted.","major_comments":[{"comment":"The discriminator is trained on optical flows of complete, clean videos, but at inference it is applied to flows computed on Tweedie estimates at intermediate timesteps (e.g., t between 3 and 20). The paper explicitly notes this distinction in Sec. 4.1 but provides no analysis that the discriminator remains informative on these out-of-distribution inputs. This is load-bearing because the entire guidance signal is the gradient of Eq. (13) with respect to the token embeddings. The ablation in Table 3, rows (a) vs (c), shows that adding ℓ_disc changes subject consistency by only +0.0024 while decreasing motion smoothness (0.9643→0.9599) and dynamic degree (0.4365→0.4125), so the empirical evidence that the discriminator drives the reported gains is weak. Please report discriminator accuracy or calibration on flows from Tweedie estimates at the optimization timesteps, and include an ablation where the discriminator gradient is replaced by a random or constant direction.","section":"Sec. 4.1, Sec. 3.3, Eq. (13)"},{"comment":"The abstract claims the method improves temporal consistency \"without compromising the fidelity of the generated content,\" but the paper's own metrics contradict this. Dynamic degree decreases for all three models (LaVie 0.5150→0.3963, AnimateDiff 0.4700→0.4125, VideoCrafter2 0.4088→0.3938), and overall consistency decreases for LaVie (0.2506→0.2415) and VideoCrafter2 (0.2498→0.2451). The text acknowledges a trade-off but does not quantify it or justify why these losses are acceptable. The abstract and conclusion should state the trade-off accurately and the claims should be scaled to the evidence.","section":"Table 1 and Abstract"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any VBench metric. Many reported differences are small (e.g., subject consistency gains of +0.002 to +0.004) and could be within run-to-run or prompt-set variability, even when seeds are fixed. Please provide standard errors over multiple seeds or a paired bootstrap over the 800 prompts, and report a significance test for the user study (e.g., a test on the win/tie/lose counts in Table 2).","section":"Tables 1 and 3, Sec. 4.2"},{"comment":"Hyperparameters (λ1, λ2, λ3, η, optimization range, number of frames) are tuned separately for each model, and the discriminator is trained on videos generated by the same model and the same 800-prompt set used for evaluation. This creates a risk of overfitting to the benchmark and makes it difficult to assess the method's generality. Please describe how hyperparameters were selected (e.g., whether a separate validation prompt set was used) and provide a sensitivity analysis or results with a fixed hyperparameter set across models.","section":"Sec. 4.1, Table 5"},{"comment":"Motion smoothness in VBench is computed from optical flow, and the guidance objective explicitly pushes generated optical flow toward a 'real' distribution. Consequently, the reported motion-smoothness gains may partly reflect optimizing for the evaluation metric rather than a general improvement in temporal coherence. Please add evaluation metrics not directly derived from optical flow (e.g., human ratings of physical plausibility, or other temporal-consistency metrics) and discuss the extent of the overlap between the guidance objective and the VBench metrics.","section":"VBench metric overlap"}],"minor_comments":[{"comment":"Equation (1) defines q(z_t|z_0) = N(z_t; sqrt(1−ᾱ_t) z_0, ᾱ_t I), which is inconsistent with the standard forward process used in Eqs. (3), (4), and (10), where z_t = sqrt(ᾱ_t) z_0 + sqrt(1−ᾱ_t) ε. Please correct the notation.","section":"Eq. (1)"},{"comment":"Algorithm 1 ends with \"return D(z0)\", which would return the discriminator's prediction rather than the generated video. This appears to be a typo; it should return z0.","section":"Algorithm 1"},{"comment":"The text states that increasing λ2 results in smoother motion, but the values are non-monotonic: motion smoothness is 0.9648 for λ2=0, 0.9599 for λ2=5, and 0.9658 for λ2=10. Please revise the description.","section":"Sec. 5.1, Table 3"},{"comment":"The statement that using a cross-dataset discriminator gives \"a general improvement in performance\" overstates the results: the discriminator trained on LaVie data improves subject consistency relative to the AD-default discriminator but decreases overall consistency (0.2529→0.2473). Please qualify the claim.","section":"Sec. 5.2, Table 4"},{"comment":"The note \"t = 0 represents the initial noise\" is confusing; in the diffusion formulation used in the paper, t = 0 denotes the clean image and t = T denotes initial noise. Please rephrase the table note.","section":"Table 3"},{"comment":"Equation (14) defines the TV loss with sums over H and W, but f is introduced as a 2×H×W optical flow; please clarify whether the loss is summed over the two channels or computed on a single-channel magnitude.","section":"Sec. 3.3, Eq. (14)"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a novel and timely idea, but the quantitative support does not yet match the strength of the claims. The main risks are the unvalidated train/inference gap for the discriminator and the possibility that the reported gains are partly due to optimizing for VBench metrics. I would encourage the authors to conduct the transfer validation and add statistical rigor; if these are addressed, the paper could be accepted. No ethical concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a sensible, clearly described extension of MinorityPrompt to video, using an optical-flow discriminator as the guidance loss. The idea is a neat combination and the writing is honest. But the quantitative evidence that the discriminator itself is doing the work is weak, and the abstract oversells fidelity preservation. It's a credible paper, worth reviewing, not a breakthrough.\n\nWhat's actually new: first to apply on-the-fly prompt optimization to video diffusion, and using a learned optical-flow realism score as the optimization target. The method is training-free for the base video model, works across three T2V models (LaVie, AnimateDiff, VideoCrafter2), and the paper includes a user study, ablations, a cross-dataset discriminator generalization check, and a synergy experiment with FreeInit. They also report computational cost, which is more than many papers do. The limitations section is candid: they admit the objective isn't physics-based and that dynamic degree drops.\n\nThe soft spots. First, the discriminator train/inference gap. The discriminator is trained on flows of complete, clean videos but at inference gets flows from VAE-decoded Tweedie estimates at intermediate timesteps. The authors note this but give no analysis showing the discriminator's predictions transfer. This is a real concern, though not fatal: at timesteps 3–15 of 50, the denoised estimates are fairly close to clean, and the gradient averages over several decoded frame pairs. What bothers me more is Table 3: adding the discriminator (rows a vs c) improves subject consistency by only 0.0024 while decreasing motion smoothness and dynamic degree. That suggests the discriminator's marginal contribution is small, and the gains over baseline might come mostly from token optimization plus TV loss.\n\nSecond, the quantitative evidence is thin. No error bars, no significance tests, and hyperparameters are tuned separately per model (Table 5). The abstract says 'without compromising fidelity,' but dynamic degree drops for all models and overall consistency drops for two of three. They do acknowledge a trade-off in the text, so this is more a wording problem than a deep flaw.\n\nThird, the method needs a per-model trained discriminator and per-model hyperparameters, which limits the 'training-free' selling point. It's training-free for the base model, but you still need to train a discriminator and tune several knobs. That's fine, but should be stated louder.\n\nCitation pattern looks fine; they build on MinorityPrompt and cite relevant video guidance and optical flow work. The math is straightforward and the pseudo-code is clear.\n\nWho this is for: people working on inference-time control of video diffusion models. A useful incremental result. It deserves a serious referee; with added significance tests, an analysis of discriminator transfer, and a cleaner ablation, it could be a solid paper. My recommendation: send it to review, and if it comes to revision, push on the ablation and the fidelity claim.","headline":"Solid incremental extension of prompt optimization to video with an optical-flow discriminator, but the evidence for the discriminator's marginal value is thinner than the paper claims.","tokens_in":14407,"tokens_out":2387,"would_cite":false,"duration_ms":20844,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MotionPrompt claims that appending learnable token embeddings to the text prompt and optimizing them during reverse sampling with an optical-flow discriminator improves temporal coherence and motion realism in text-to-video diffusion…","keywords":["MotionPrompt","text-to-video diffusion","prompt optimization","optical flow","temporal consistency","guidance","discriminator","inference-time optimization"],"falsifier":"Train the same discriminator on optical flows of partially denoised intermediate estimates and measure its accuracy: if it is near chance, the guidance gradients are arbitrary. A cheaper check is to randomly shuffle the discriminator's labels while keeping the rest of the pipeline; if the VBench improvements persist, the discriminator is not the source of the gain.","tokens_in":13453,"feed_emoji":"🎬","tokens_out":4197,"duration_ms":38067,"temperature":0.7,"pith_summary":"MotionPrompt is an inference-time method for making text-to-video diffusion models produce more temporally coherent videos. The paper's central claim is that a few learnable tokens appended to the text prompt, optimized during reverse sampling using gradients from an optical-flow discriminator, improve motion smoothness and subject consistency across several video models without retraining the diffusion model. Instead of backpropagating through all frames of the video latent, the optimization updates only the token embeddings, which influence the whole generated sequence. The reported VBench scores and a user study support the claim that temporal quality improves while text alignment stays roughly constant.","feed_headline":"Hidden prompt tokens smooth AI video motion","feed_subtitle":"An optical-flow discriminator steers token embeddings during sampling, improving coherence without retraining video models.","key_machinery":"The central object is a learnable token embedding $T$ appended to the text prompt, optimized on-the-fly during reverse diffusion sampling. Because the text conditioning $c(T)$ influences every frame, gradients computed from optical flow on a small subset of frames indirectly steer the whole video, avoiding per-frame backpropagation through the video latent. The guidance signal comes from a discriminator trained on optical flows of real versus generated videos, supplemented by a total-variation term that encourages smooth flow fields and an $\\ell^2$ penalty that keeps the optimized embeddings near their initial text-space location.","core_discovery":"The discovery is that optical-flow realism can serve as a usable guidance signal for prompt optimization in latent video diffusion. The paper trains a discriminator $\\phi_\\theta$ to distinguish optical flows computed between random pairs of frames from real videos and flows from generated videos. At inference, it appends placeholder tokens $S$ to the prompt, and at selected timesteps computes the denoised estimate $\\hat{x}_t(c(T))$, extracts optical flow $f(\\hat{x}_t(c(T)))$ from a subset of decoded frames, and optimizes the token embeddings $T$ against $\\ell_{\\text{total}} = \\lambda_1\\log(1-\\phi_{\\theta^*}(f)) + \\lambda_2\\,\\mathrm{TV}(f) + \\lambda_3\\|T-T_0\\|_2^2$. The result is videos whose motion is judged more natural and whose object appearance and background remain consistent, with only a small reduction in dynamic degree.","pith_inferences":["Because the objective is not physics-based, the same framework could become a general video-quality prompt controller by swapping the discriminator for any differentiable evaluator of a desired sequence property.","The optimized token embeddings may drift into reusable prompt-space descriptors of motion style; inspecting them after generation could reveal interpretable motion words.","The clean-video-versus-denoised-estimate gap could be closed by explicitly training the discriminator on noised inputs, which would likely change the effective strength of the guidance signal.","If the discriminator's signal transfers across frame distances, sampling optical flows from distant frame pairs rather than adjacent ones might extend the method to longer videos."],"forward_implications":["Pretrained text-to-video models can be guided toward smoother, more consistent motion without fine-tuning the diffusion weights.","Guidance cost is bounded by optimizing a small embedding rather than the full video latent, so the approach is cheaper than directing every frame separately.","The loss terms are modular: the discriminator pushes flows toward real-video statistics, the total-variation term enforces smooth flow fields, and the $\\ell^2$ penalty keeps embeddings near the text space the model was trained on.","The method works across different backbone models (Lavie, AnimateDiff, VideoCrafter2) and extends to image-to-video generation with DynamiCrafter.","There is an expected trade-off between motion dynamics and coherence; the paper reports that its chosen hyperparameters balance the two."],"supporting_citations":[{"why":"Supplies the on-the-fly prompt optimization mechanism with learnable tokens that MotionPrompt extends from minority image generation to video guidance.","marker":"[35]"},{"why":"Provides the DPS-style guidance formulation and Tweedie's formula used to compute the denoised estimate at each optimization step.","marker":"[6]"},{"why":"Supplies the VBench benchmark metrics used for the quantitative evaluation of consistency, flickering, smoothness, and dynamics.","marker":"[18]"},{"why":"Provides the RAFT optical flow estimator used to compute flows from real and generated frame pairs for discriminator training and inference.","marker":"[34]"},{"why":"Supplies the pretrained Vision Transformer backbone fine-tuned as the optical-flow discriminator's image encoder.","marker":"[9]"},{"why":"Serves as one of the text-to-video baseline models evaluated with and without MotionPrompt.","marker":"[4]"},{"why":"Serves as another text-to-video baseline and as the primary model for the ablation studies.","marker":"[14]"},{"why":"Serves as the third text-to-video baseline used to demonstrate the method's generality across different diffusion backbones.","marker":"[36]"}],"fun_headline_variants":["Optical flow guides prompt tokens for coherent video","Flow-based guidance tweaks video prompt embeddings","Motion-aware prompt optimization for stable video","Flow-discriminated prompt tokens reduce video jitter"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a discriminator trained on optical flows of complete, clean videos can still tell real from generated motion when it is applied to optical flows of partially denoised frames during sampling; the paper notes this gap but does not verify that the signals transfer.","fun_headline_variants_meta":{"raw":{"variants":["Optical flow guides prompt tokens for coherent video","Flow-based guidance tweaks video prompt embeddings","Motion-aware prompt optimization for stable video","Flow-discriminated prompt tokens reduce video jitter"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000436,"raw_usage":{"total_tokens":2184,"prompt_tokens":875,"completion_tokens":1309,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":1252}},"tokens_in":491,"tokens_out":1309,"duration_ms":9086,"temperature":1.0,"reasoning_tokens":1252,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:09:44.553800+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same discriminator on optical flows of partially denoised intermediate estimates and measure its accuracy: if it is near chance, the guidance gradients are arbitrary. A cheaper check is to randomly shuffle the discriminator's labels while keeping the rest of the pipeline; if the VBench improvements persist, the discriminator is not the source of the gain.","supporting_citations":[{"cited_title":"Diffusion posterior sampling for general noisy inverse problems","cited_arxiv_id":null,"evidence_quote":"Provides the DPS-style guidance formulation and Tweedie's formula used to compute the denoised estimate at each optimization step."},{"cited_title":"VBench: Com- prehensive benchmark suite for video generative models","cited_arxiv_id":null,"evidence_quote":"Supplies the VBench benchmark metrics used for the quantitative evaluation of consistency, flickering, smoothness, and dynamics."},{"cited_title":"Raft: Recurrent all-pairs field transforms for optical flow","cited_arxiv_id":null,"evidence_quote":"Provides the RAFT optical flow estimator used to compute flows from real and generated frame pairs for discriminator training and inference."},{"cited_title":"Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024","cited_arxiv_id":null,"evidence_quote":"Serves as one of the text-to-video baseline models evaluated with and without MotionPrompt."},{"cited_title":"Animatediff: Animate your personalized text-to-image diffusion models without specific tuning","cited_arxiv_id":null,"evidence_quote":"Serves as another text-to-video baseline and as the primary model for the ablation studies."}],"review_version":1}