Pith. sign in

REVIEW 5 major objections 6 minor 41 references

Optical-Flow Guided Prompt Optimization for Coherent Video Generation

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MotionPrompt claims that appending learnable token embeddings to the text prompt and optimizing them during reverse sampling with an optical-flow discriminator improves temporal coherence and motion realism in text-to-video diffusion…

desk verdict Solid incremental extension of prompt optimization to video with an optical-flow discriminator, but the evidence for the discriminator's marginal value is thinner than the paper claims. read the letter →

arxiv 2411.15540 v2 pith:JMASYGNA submitted 2024-11-23 cs.CV cs.AIcs.LGeess.IV

Hyelin Nam , Jaemin Kim , Dohun Lee , Jong Chul Ye This is my paper · ORCID
classification cs.CVcs.AIcs.LGeess.IV
keywords MotionPrompttext-to-videodiffusionoptimizationopticalflowtemporalconsistencyguidancediscriminatorinference-time
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MotionPrompt is an inference-time method for making text-to-video diffusion models produce more temporally coherent videos. The paper's central claim is that a few learnable tokens appended to the text prompt, optimized during reverse sampling using gradients from an optical-flow discriminator, improve motion smoothness and subject consistency across several video models without retraining the diffusion model. Instead of backpropagating through all frames of the video latent, the optimization updates only the token embeddings, which influence the whole generated sequence. The reported VBench scores and a user study support the claim that temporal quality improves while text alignment stays roughly constant.

What carries the argument

The central object is a learnable token embedding $T$ appended to the text prompt, optimized on-the-fly during reverse diffusion sampling. Because the text conditioning $c(T)$ influences every frame, gradients computed from optical flow on a small subset of frames indirectly steer the whole video, avoiding per-frame backpropagation through the video latent. The guidance signal comes from a discriminator trained on optical flows of real versus generated videos, supplemented by a total-variation term that encourages smooth flow fields and an $\ell^2$ penalty that keeps the optimized embeddings near their initial text-space location.

What would settle it

Train the same discriminator on optical flows of partially denoised intermediate estimates and measure its accuracy: if it is near chance, the guidance gradients are arbitrary. A cheaper check is to randomly shuffle the discriminator's labels while keeping the rest of the pipeline; if the VBench improvements persist, the discriminator is not the source of the gain.

Watch

Extended reading notes

Core claim

The discovery is that optical-flow realism can serve as a usable guidance signal for prompt optimization in latent video diffusion. The paper trains a discriminator $\phi_\theta$ to distinguish optical flows computed between random pairs of frames from real videos and flows from generated videos. At inference, it appends placeholder tokens $S$ to the prompt, and at selected timesteps computes the denoised estimate $\hat{x}_t(c(T))$, extracts optical flow $f(\hat{x}_t(c(T)))$ from a subset of decoded frames, and optimizes the token embeddings $T$ against $\ell_{\text{total}} = \lambda_1\log(1-\phi_{\theta^*}(f)) + \lambda_2\,\mathrm{TV}(f) + \lambda_3\|T-T_0\|_2^2$. The result is videos whose motion is judged more natural and whose object appearance and background remain consistent, with only a small reduction in dynamic degree.

Load-bearing premise

The load-bearing premise is that a discriminator trained on optical flows of complete, clean videos can still tell real from generated motion when it is applied to optical flows of partially denoised frames during sampling; the paper notes this gap but does not verify that the signals transfer.

Editorial extensions

If this is right

  • Pretrained text-to-video models can be guided toward smoother, more consistent motion without fine-tuning the diffusion weights.
  • Guidance cost is bounded by optimizing a small embedding rather than the full video latent, so the approach is cheaper than directing every frame separately.
  • The loss terms are modular: the discriminator pushes flows toward real-video statistics, the total-variation term enforces smooth flow fields, and the $\ell^2$ penalty keeps embeddings near the text space the model was trained on.
  • The method works across different backbone models (Lavie, AnimateDiff, VideoCrafter2) and extends to image-to-video generation with DynamiCrafter.
  • There is an expected trade-off between motion dynamics and coherence; the paper reports that its chosen hyperparameters balance the two.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the objective is not physics-based, the same framework could become a general video-quality prompt controller by swapping the discriminator for any differentiable evaluator of a desired sequence property.
  • The optimized token embeddings may drift into reusable prompt-space descriptors of motion style; inspecting them after generation could reveal interpretable motion words.
  • The clean-video-versus-denoised-estimate gap could be closed by explicitly training the discriminator on noised inputs, which would likely change the effective strength of the guidance signal.
  • If the discriminator's signal transfers across frame distances, sampling optical flows from distant frame pairs rather than adjacent ones might extend the method to longer videos.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes MotionPrompt, an inference-time guidance method for text-to-video diffusion models. It appends learnable token embeddings to the text prompt and optimizes them during reverse sampling using gradients from a discriminator that distinguishes optical flows of real videos from those of generated videos. The method is evaluated on LaVie, AnimateDiff, and VideoCrafter2 using VBench metrics, ablations, and a user study. The paper reports improvements in subject consistency, motion smoothness, and temporal flickering, while dynamic degree decreases for all models and overall consistency decreases for two of the three models.

Significance. If the results hold, the idea of controlling global video properties by optimizing prompt tokens guided by an optical-flow discriminator is a novel and potentially useful addition to inference-time video guidance. Strengths include: no retraining of the video diffusion model; gradient computation on only a subset of frames; orthogonality to existing methods such as FreeInit; and the inclusion of ablations, cross-discriminator generalization, token-variation analysis, and a user study. However, the quantitative evidence is currently thin: gains are small, no error bars or significance tests are reported, hyperparameters are tuned per model, and the discriminator is applied to out-of-distribution inputs without validation. These issues must be addressed before the central claim can be accepted.

major comments (5)
  1. [Sec. 4.1, Sec. 3.3, Eq. (13)] The discriminator is trained on optical flows of complete, clean videos, but at inference it is applied to flows computed on Tweedie estimates at intermediate timesteps (e.g., t between 3 and 20). The paper explicitly notes this distinction in Sec. 4.1 but provides no analysis that the discriminator remains informative on these out-of-distribution inputs. This is load-bearing because the entire guidance signal is the gradient of Eq. (13) with respect to the token embeddings. The ablation in Table 3, rows (a) vs (c), shows that adding ℓ_disc changes subject consistency by only +0.0024 while decreasing motion smoothness (0.9643→0.9599) and dynamic degree (0.4365→0.4125), so the empirical evidence that the discriminator drives the reported gains is weak. Please report discriminator accuracy or calibration on flows from Tweedie estimates at the optimization timesteps, and include an ablation where the discriminator gradient is replaced by a random or constant direction.
  2. [Table 1 and Abstract] The abstract claims the method improves temporal consistency "without compromising the fidelity of the generated content," but the paper's own metrics contradict this. Dynamic degree decreases for all three models (LaVie 0.5150→0.3963, AnimateDiff 0.4700→0.4125, VideoCrafter2 0.4088→0.3938), and overall consistency decreases for LaVie (0.2506→0.2415) and VideoCrafter2 (0.2498→0.2451). The text acknowledges a trade-off but does not quantify it or justify why these losses are acceptable. The abstract and conclusion should state the trade-off accurately and the claims should be scaled to the evidence.
  3. [Tables 1 and 3, Sec. 4.2] No error bars, confidence intervals, or significance tests are reported for any VBench metric. Many reported differences are small (e.g., subject consistency gains of +0.002 to +0.004) and could be within run-to-run or prompt-set variability, even when seeds are fixed. Please provide standard errors over multiple seeds or a paired bootstrap over the 800 prompts, and report a significance test for the user study (e.g., a test on the win/tie/lose counts in Table 2).
  4. [Sec. 4.1, Table 5] Hyperparameters (λ1, λ2, λ3, η, optimization range, number of frames) are tuned separately for each model, and the discriminator is trained on videos generated by the same model and the same 800-prompt set used for evaluation. This creates a risk of overfitting to the benchmark and makes it difficult to assess the method's generality. Please describe how hyperparameters were selected (e.g., whether a separate validation prompt set was used) and provide a sensitivity analysis or results with a fixed hyperparameter set across models.
  5. [VBench metric overlap] Motion smoothness in VBench is computed from optical flow, and the guidance objective explicitly pushes generated optical flow toward a 'real' distribution. Consequently, the reported motion-smoothness gains may partly reflect optimizing for the evaluation metric rather than a general improvement in temporal coherence. Please add evaluation metrics not directly derived from optical flow (e.g., human ratings of physical plausibility, or other temporal-consistency metrics) and discuss the extent of the overlap between the guidance objective and the VBench metrics.
minor comments (6)
  1. [Eq. (1)] Equation (1) defines q(z_t|z_0) = N(z_t; sqrt(1−ᾱ_t) z_0, ᾱ_t I), which is inconsistent with the standard forward process used in Eqs. (3), (4), and (10), where z_t = sqrt(ᾱ_t) z_0 + sqrt(1−ᾱ_t) ε. Please correct the notation.
  2. [Algorithm 1] Algorithm 1 ends with "return D(z0)", which would return the discriminator's prediction rather than the generated video. This appears to be a typo; it should return z0.
  3. [Sec. 5.1, Table 3] The text states that increasing λ2 results in smoother motion, but the values are non-monotonic: motion smoothness is 0.9648 for λ2=0, 0.9599 for λ2=5, and 0.9658 for λ2=10. Please revise the description.
  4. [Sec. 5.2, Table 4] The statement that using a cross-dataset discriminator gives "a general improvement in performance" overstates the results: the discriminator trained on LaVie data improves subject consistency relative to the AD-default discriminator but decreases overall consistency (0.2529→0.2473). Please qualify the claim.
  5. [Table 3] The note "t = 0 represents the initial noise" is confusing; in the diffusion formulation used in the paper, t = 0 denotes the clean image and t = T denotes initial noise. Please rephrase the table note.
  6. [Sec. 3.3, Eq. (14)] Equation (14) defines the TV loss with sums over H and W, but f is introduced as a 2×H×W optical flow; please clarify whether the loss is summed over the two channels or computed on a single-channel magnitude.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MotionPrompt's optical-flow discriminator is a standard inference-time guidance objective, and its results are tested against external VBench and user-study benchmarks, so the reported improvements are not equivalent to the training inputs by construction.

full rationale

The paper's derivation chain is: train an optical-flow discriminator via Eq. 12 on RAFT flows from real videos (DAVIS/WebVid) and generated videos from the baseline model; define the guidance loss as log(1 - D(f(\hat{x}_t(c(T)))) in Eq. 13; optimize only the learnable prompt tokens T via Eq. 11; and evaluate on VBench metrics and a user study. None of the reported VBench metrics is identical to the training objective: subject consistency, background consistency, temporal flickering, motion smoothness, and dynamic degree are external evaluation functions computed on final videos, whereas the guidance loss is a classifier score on optical flow of intermediate Tweedie estimates. The overlap between the discriminator's fake training videos and the baseline models used in evaluation is the standard GAN setup, not a construction-level equivalence. The prompt-optimization mechanism is attributed to Um and Ye [35], a same-group citation, but the paper independently validates that component with ablations (Table 6 and Fig. 4), so the citation is methodological rather than load-bearing. The acknowledged train/inference gap (clean flows for training versus denoised estimates during sampling) is a distribution-shift validity risk, not a circular argument, and the hyperparameter selection on VBench is an evaluation-conduct concern rather than a reduction of the prediction to its inputs. Accordingly, no circular step is exhibited.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper's contribution consists mainly of a new inference-time optimization scheme; it inherits diffusion background from prior work and introduces several per-model hyperparameters that the central results partly depend on.

free parameters (6)
  • λ1, λ2, λ3 (loss weights) = λ1=1.0 all; λ2=1.0/5.0/5.0 for Lavie/AnimateDiff/VideoCrafter2; λ3=10.0/10.0/3.0
    Regularization weights in ℓ_total are tuned per model based on VBench scores; see Tab. 5 and the ablation in Tab. 3.
  • Learning rate η for prompt optimization = 0.0005 (Lavie), 0.005 (AnimateDiff), 0.001 (VideoCrafter2)
    Chosen per model; reported in Tab. 5, supplementary A.2.
  • Optimization range (timesteps) = 5<t<15 (Lavie), 3<t<15 (AnimateDiff), 3<t<20 (VideoCrafter2)
    Range of DDIM steps during which the prompt embedding is optimized; tuned per model, Tab. 5.
  • Number of decoded frames for flow = 6 (Lavie), 2 (AnimateDiff), 6 (VideoCrafter2)
    Number of frames decoded into pixels to compute optical-flow losses; per-model choice in Tab. 5.
  • K (optimization iterations per step) = 3 for all models
    Number of inner-loop token-embedding updates; chosen in ablation Tab. 3 as a balance between quality and cost.
  • Token initialization word = 'authentic'
    Initial embedding for the learnable token; the authors state it was selected as a word that can help improve video quality (Section 3.2).
assumptions (5)
  • standard math Tweedie's formula yields the posterior mean of the clean latent.
    Used in Eqs. (4) and (10) to estimate the clean sample during reverse sampling; standard in diffusion guidance.
  • standard math DDIM reverse sampling and classifier-free guidance are taken as background.
    Eqs. (3)-(5); no new derivation is given for these.
  • domain assumption Improving the realism of optical flow in generated videos improves temporal consistency.
    Stated in Section 2 as the core hypothesis; the whole guidance loss is built on it.
  • domain assumption A discriminator trained on clean video optical flows transfers to denoised estimates at intermediate timesteps.
    Explicitly acknowledged in Section 4.1; no validation of the transfer is provided.
  • ad hoc to paper The total loss with tuned weights and TV regularization is an appropriate objective for temporal coherence.
    The loss in Section 3.3 is constructed for this paper and the weights are tuned per model without a principled derivation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optical-Flow Guided Prompt Optimization for Coherent Video Generation." pith.science (2026). https://pith.science/paper/JMASYGNA

@misc{pith2026241115540,
  author       = {Pith},
  title        = {Pith review of: Optical-Flow Guided Prompt Optimization for Coherent Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JMASYGNA}},
  note         = {Machine review of arXiv:2411.15540}
}
read the original abstract

While text-to-video diffusion models have made significant strides, many still face challenges in generating videos with temporal consistency. Within diffusion frameworks, guidance techniques have proven effective in enhancing output quality during inference; however, applying these methods to video diffusion models introduces additional complexity of handling computations across entire sequences. To address this, we propose a novel framework called MotionPrompt that guides the video generation process via optical flow. Specifically, we train a discriminator to distinguish optical flow between random pairs of frames from real videos and generated ones. Given that prompts can influence the entire video, we optimize learnable token embeddings during reverse sampling steps by using gradients from a trained discriminator applied to random frame pairs. This approach allows our method to generate visually coherent video sequences that closely reflect natural motion dynamics, without compromising the fidelity of the generated content. We demonstrate the effectiveness of our approach across various models.

Figures

Figures reproduced from arXiv: 2411.15540 by the authors.

Figure 1
Figure 1. MotionPrompt enhances temporal consistency and motion smoothness in text-to-video diffusion models by combining optical flow guidance with prompt optimization. It can be combined with a range of text-to-video diffusion models to produce visually coherent video sequences that closely align with intended motion while preserving content fidelity. Best viewed with Acrobat Reader. Click each image to play the video clip.… view at source ↗
Figure 2
Figure 2. Overall pipeline of MotionPrompt. MotionPrompt enhances temporal consistency in text-to-video diffusion models by combining prompt optimization with an optical flow-based discriminator. Leveraging gradients from a subset of frames and aligning optical flow with real-world motion patterns, MotionPrompt efficiently generates videos with smooth, realistic motion and strong contextual coherence. 3.2. Prompt Optimization… view at source ↗
Figure 3
Figure 3. Qualitative comparison against three baselines. Additional results are provided in the supplementary material. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Cosine similarity between learnable and initial token [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Comparison of video results generated by the vanilla [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparison between the baseline, FreeInit, [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Additional results of DynamicCrafter [40]. the results obtained by using both methods together, high￾lighting their combined potential. While FreeInit significantly improves temporal quality, it does so at the expense of overall video quality. Specifically, Tab. 7 demo…
Figure 8
Figure 8. Figure 8: Computational cost. We compute GPU memory usage and computing time, aver￾aging over 100 gener￾ations. C. Additional Results In this section, we provide additional result images to further demonstrate the performance and effectiveness of our approach across different mo…
Figure 9
Figure 9. Figure 9: Additional results of AnimateDiff [14]. 4 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Additional results of Lavie [36]. 5 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Additional results of VideoCrafter2 [4]. 6 [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 25 canonical work pages

  1. [1]

    Frozen in time: A joint video and image encoder for end-to- end retrieval

    Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to- end retrieval. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1728–1738, 2021. 6

    work page 2021

  2. [2]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 22563–22575, 2023. 2

    2023

  3. [3]

    Videocrafter1: Open diffusion models for high-quality video generation, 2023

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter1: Open diffusion models for high-quality video generation, 2023. 2

    2023

  4. [4]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024. 2, 5, 7, 3, 6

    work page 2024

  5. [5]

    Unictrl: Improving the spatiotemporal consistency of text-to-video diffusion mod- els via training-free unified attention control

    Xuweiyi Chen, Tian Xia, and Sihan Xu. Unictrl: Improving the spatiotemporal consistency of text-to-video diffusion mod- els via training-free unified attention control. arXiv preprint arXiv:2403.02332, 2024. 2, 3

    arXiv 2024

  6. [6]

    Diffusion posterior sampling for general noisy inverse problems

    Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In International Conference on Learning Representations, 2023. 2, 3

    work page 2023

  7. [7]

    Prompt-tuning latent diffusion models for inverse problems

    Hyungjin Chung, Jong Chul Ye, Peyman Milanfar, and Mauri- cio Delbracio. Prompt-tuning latent diffusion models for inverse problems. In ICML. OpenReview.net, 2024. 3

    work page 2024

  8. [8]

    Diffusion models beat GANs on image synthesis

    Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. InAdvances in Neural Information Processing Systems, 2021. 2, 3

    work page 2021

Show all 41 references
  1. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  2. [10]

    Tweedie’s formula and selection bias

    Bradley Efron. Tweedie’s formula and selection bias. Journal of the American Statistical Association, 106(496):1602–1614,

  3. [11]

    Preserve your own correlation: A noise prior for video diffusion models

    Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Comp...

    2023

  4. [12]

    Motion guidance: Diffusion-based image editing with differentiable motion esti- mators

    Daniel Geng and Andrew Owens. Motion guidance: Diffusion-based image editing with differentiable motion esti- mators. In The Twelfth International Conference on Learning Representations, 2024. 3

    2024

  5. [14]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yao- hui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. International Con- ference on Learning Representations , 2024. 2, ...

    2024

  6. [15]

    Latent video diffusion models for high-fidelity long video generation

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221 ,

    arXiv

  7. [16]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. 3

    2021

  8. [17]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. In The Eleventh International Conference on Learning Representations, 2023. 2

    2023

  9. [18]

    VBench: Com- prehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Com- prehensive benchmark suite for video generative models....

    2024

  10. [19]

    Dif- fusionclip: Text-guided diffusion models for robust image manipulation

    Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Dif- fusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426– 2435, 2022. 2

    2022

  11. [20]

    Dream- sampler: Unifying diffusion sampling and score distillation for image manipulation

    Jeongsol Kim, Geon Yeong Park, and Jong Chul Ye. Dream- sampler: Unifying diffusion sampling and score distillation for image manipulation. arXiv preprint arXiv:2403.11415,

    arXiv

  12. [21]

    Videoguide: Improving video diffusion models without training through a teacher’s guide, 2024

    Dohun Lee, Bryan S Kim, Geon Yeong Park, and Jong Chul Ye. Videoguide: Improving video diffusion models without training through a teacher’s guide, 2024. 2, 3

    2024

  13. [23]

    Flow-grounded spatial-temporal video prediction from still images

    Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Flow-grounded spatial-temporal video prediction from still images. In European Conference on Computer Vision, 2018. 3 9

    2018

  14. [24]

    FlowVid: Taming im- perfect optical flows for consistent video-to-video synthesis

    Feng Liang, Bichen Wu, Jialiang Wang, Licheng Yu, Kunpeng Li, Yinan Zhao, Ishan Misra, Jia-Bin Huang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. FlowVid: Taming im- perfect optical flows for consistent video-to-video synthesis. In Proceedings of the IEEE/CVF Conference ...

    2024

  15. [25]

    Conditional image-to-video gener- ation with latent flow diffusion models

    Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image-to-video gener- ation with latent flow diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18444–18455, 2023. 3

    2023

  16. [26]

    Video generation from single semantic label map

    Junting Pan, Chengyu Wang, Xu Jia, Jing Shao, Lu Sheng, Junjie Yan, and Xiaogang Wang. Video generation from single semantic label map. arXiv preprint arXiv:1903.04480,

    1903 arXiv

  17. [27]

    Energy-based cross attention for bayesian context update in text-to-image diffusion models

    Geon Yeong Park, Jeongsol Kim, Beomsu Kim, Sang Wan Lee, and Jong Chul Ye. Energy-based cross attention for bayesian context update in text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024. 3

    2024

  18. [28]

    The 2017 davis challenge on video object segmentation

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Ar- beláez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675, 2017. 6

    2017 arXiv

  19. [29]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022. 2

    2022

  20. [30]

    Animating arbitrary objects via deep motion transfer

    Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 3

    2019

  21. [31]

    First order motion model for image animation

    Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In Conference on Neural Information Pro- cessing Systems (NeurIPS), 2019. 3

    2019

  22. [32]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502,

    2010 arXiv

  23. [33]

    Pseudoinverse-guided diffusion models for inverse problems

    Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guided diffusion models for inverse problems. In International Conference on Learning Repre- sentations, 2023. 2

    2023

  24. [34]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020. 6

    2020

  25. [35]

    Minorityprompt: Text to minority image generation via prompt optimization

    Soobin Um and Jong Chul Ye. Minorityprompt: Text to minority image generation via prompt optimization. arXiv preprint arXiv:2410.07838, 2024. 2, 3, 4

    2024 arXiv

  26. [36]

    Lavie: High-quality video gener- ation with cascaded latent diffusion models

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video gener- ation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023. 2, 5, 7, 3

    2023 arXiv

  27. [37]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large lan- guage models. In Advances in Neural Information Processing Systems, 2022. 3

    2022

  28. [38]

    Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery

    Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. In Thirty-seventh Conference on Neural Informa- tion Processing Systems, 2023. 3

    2023

  29. [39]

    Freeinit: Bridging initialization gap in video diffusion models

    Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu. Freeinit: Bridging initialization gap in video diffusion models. In European Conference on Computer Vision (ECCV), 2024. 2

    2024

  30. [40]

    Dynamicrafter: Animating open- domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open- domain images with video diffusion priors. In European Conference on Computer Vision, pages 399–417. Springer,

  31. [41]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 2

    2024 arXiv

  32. [42]

    Conditional prompt learning for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 3

    2022

  33. [43]

    Choose the more natural and smoothly moving video

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Zi- wei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision (IJCV), 2022. 3 10 Optical-Flow Guided Prompt Optimization for Coherent Video Generation Supplementary Material The supplement...

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.