Pith. sign in

REVIEW 5 major objections 6 minor 44 references

TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Models

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TITAN-Guide shows that forward gradients can replace backpropagation for classifier guidance, cutting GPU memory roughly in half while keeping or improving video quality.

desk verdict Forward-mode AD for diffusion guidance is a genuinely useful memory saver, but the quality gains are under-evidenced and the score-based tangent needs better justification. read the letter →

arxiv 2508.00289 v1 pith:INYF2CRO submitted 2025-08-01 cs.CV

Christian Simon , Masato Ishii , Akio Hayakawa , Zhi Zhong , Shusuke Takahashi , Takashi Shibuya , Yuki Mitsufuji This is my paper
classification cs.CV
keywords training-freeguidanceclassifiertext-to-videodiffusionforwardgradientdescentlatentoptimizationmemory-efficientinference-timealignmentvideogeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TITAN-Guide aims to make inference-time guidance practical for text-to-video (T2V) diffusion models, which are too memory-hungry for existing classifier-guidance methods. The paper's central claim is that forward gradients — directional derivatives of the guidance loss with respect to the video latent, computed in a single forward pass — can replace backpropagation when updating latents during sampling. Because no backward pass is needed, memory for the optimizer drops by roughly half, and the method can guide AnimateDiff and CogVideoX on a single consumer GPU. On FVD and several alignment metrics, TITAN-Guide matches or beats prior training-free guidance baselines while also supporting higher resolutions. If correct, this makes off-the-shelf predictors (aesthetic scorers, style encoders, audio-video aligners) a practical steering mechanism for video generation without any fine-tuning.

What carries the argument

The load-bearing object is the forward-gradient update $G_t = \langle \nabla f, V_t \rangle V_t$, computed with forward-mode automatic differentiation (a Jacobian-vector product) in a single forward pass. $V_t$ is a "gradient guess" direction; the machinery is that projecting the true gradient onto $V_t$ and multiplying back by $V_t$ gives an unbiased but high-variance estimate of $\nabla f$ when $V_t$ is random, and a more deterministic estimate when $V_t$ is the normalized score estimate from the denoiser. This estimate is used in the update $Z_t \leftarrow Z_t - \lambda_t G_t$, avoiding the memory cost of storing the full computational graph for backpropagation. The second component is the traversal to $t=0$ via iterative sampling rather than the Tweedie estimate, which the paper argues gives the guidance classifier a clean video to score at every step.

What would settle it

Measure the cosine similarity between TITAN-Guide's update direction $G_t$ and the true gradient $\nabla f$ at the same latent on a handful of VGG-Sound prompts: if the similarity is zero or negative at early denoising steps while reported FVD gains persist, then the forward-gradient mechanism is not the source of the improvement.

Watch

Extended reading notes

Core claim

The paper proposes that a guided text-to-video diffusion step should not rely on backpropagation or on a single-step Tweedie estimate of the clean video. Instead, at each denoising step it fully samples the latent forward to t=0 with the denoiser, decodes the result, scores it with an off-the-shelf predictor, and then updates the original latent by the forward-gradient estimate $G_t = \langle \nabla f, V_t \rangle V_t$. Three choices of the guess direction $V_t$ are studied: isotropic Gaussian noise, the normalized denoiser output $\epsilon_\theta(Z_t,t,c)/\|\epsilon_\theta(Z_t,t,c)\|$, and a gradient sampled from a small subset of frames. The paper claims this update eliminates the backward pass, cuts GPU memory by about half relative to DOODL, TFG, MPGD, and FreeDoM, and produces lower FVD and higher alignment scores on audio-video, aesthetic, style, and frame-interpolation tasks.

Load-bearing premise

The paper assumes the forward-gradient direction $G_t = \langle \nabla f, V_t \rangle V_t$ points downhill enough to improve the latent, most speculatively when $V_t$ is the score-based guess $\epsilon_\theta(Z_t,t,c)/\|\epsilon_\theta(Z_t,t,c)\|$; this is heuristic and only supported by the toy study and qualitative results.

Editorial extensions

If this is right

  • Guidance of T2V models becomes feasible on 24GB consumer GPUs; at 256x256 the method uses about 20.6GB versus 40-67GB for prior baselines.
  • Higher resolution is unlocked: 384x384 guidance runs in 42GB where baselines exceed 90GB.
  • The same latent-update rule works across multiple off-the-shelf predictors (ImageBind, DOVER, Style-CLIP) and across tasks including audio-video alignment, aesthetic scoring, style transfer, and frame interpolation.
  • FVD and alignment metrics improve or match the strongest baselines, with sampled-gradient guesses giving the best quality in the paper's comparisons.
  • Because the method avoids backpropagation, it also extends to image-domain guidance tasks such as super-resolution, deblurring, and attribute conditioning, where it is competitive on LPIPS, FID, and KID.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the forward-gradient estimator's variance is the main quality bottleneck, so replacing $V_t$ with a lower-variance control variate (e.g., a learned or ensemble direction) should further improve guidance without adding memory.
  • Beyond the paper: because the update is forward-only, the same scheme could steer other memory-intensive generative models — high-resolution image diffusion, 3D/NeRF, or long-video generators — where backpropagation through sampling is prohibitive.
  • Beyond the paper: the paper's $t=0$ traversal costs about 2 minutes per video versus roughly 40 seconds for single-step estimators; caching trajectory states or amortizing the clean-sample computation could close that speed gap while keeping the memory advantage.
  • Beyond the paper: the score-based guess is only heuristically motivated; formalizing when $\langle \nabla f, \epsilon_\theta/\|\epsilon_\theta\| \rangle \epsilon_\theta/\|\epsilon_\theta\|$ is a descent direction (e.g., connecting it to denoising score matching) would put the method on firmer ground.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes TITAN-Guide, a training-free classifier-guidance method for text-to-video (T2V) diffusion models. Instead of backpropagating through the denoising and decoding pipeline, the method estimates the gradient of a guidance loss by forward-mode automatic differentiation, computing G_t = <∇f, V_t>V_t for a tangent direction V_t. Three choices of V_t are studied: isotropic Gaussian noise (random guesses), the normalized output of the denoiser (score-based guesses), and a normalized gradient computed on a small subset of frames (sampled-gradient guesses). The method is evaluated on AnimateDiff and CogVideoX for audio–video alignment, aesthetic guidance, style guidance, frame interpolation, and several image-domain tasks. Reported GPU memory usage is roughly half that of backpropagation-based baselines such as DOODL, enabling guidance at resolutions that baselines cannot handle on consumer GPUs.

Significance. The memory-efficiency claim is directly measured and is a practically valuable contribution: controlling memory-intensive T2V models without backpropagation would broaden the applicability of guidance methods. The paper ships code and includes memory measurements at multiple resolutions. However, the central performance claim of 'more optimal control' is not robustly established. The reported quality gains over baselines are small (1–3 FVD points), are shown without error bars or multiple seeds, and partly rely on evaluating with the same models used as guidance objectives. Furthermore, the best-performing variant (sampled-gradient guesses) appears to require backpropagation to compute its tangent direction, which weakens the paper's stated novelty. If the authors can address these concerns with additional experiments and a more careful characterization of the forward-gradient approximation, the method could be a useful addition to the guided-diffusion toolbox.

major comments (5)
  1. [§4.2, Eq. (9) and Algorithm 1] The score-based guess V_t = εθ(Z_t,t,c)/||εθ(Z_t,t,c)|| is a deterministic direction, so G_t = ⟨∇f, V_t⟩V_t is a rank-one projection of the true gradient, not an unbiased estimator. The update magnitude equals ||∇f|| times the cosine between ∇f and V_t, which can be near zero in high-dimensional latent spaces. The paper provides no measurement of this alignment; the toy experiment in Figure 4 is qualitative and single-seed, and Tables 1–2 report single runs. Without evidence that the denoiser output correlates with the guidance-loss gradient, the claimed quality advantage over DOODL and other baselines is not explained.
  2. [Tables 1, 2, and 4] All results are reported as single runs without error bars or multiple seeds. The FVD differences between TITAN-Guide and DOODL are 1–3 points (Table 1: 334.21 vs 333.42 and 331.11), and the IB-AV/IB-TV scores are essentially flat across methods (e.g., 0.218 vs 0.215–0.216). Without variance estimates or significance testing, these differences do not support the claim of 'more optimal control'. Please report means and standard deviations over at least three seeds.
  3. [§4.2, Sampled gradient guesses] This variant defines V_t by sampling from ∇_{Z^F_0|t} f(Dϕ(Z^F_0|t), y, ψ), which is a backpropagation through the decoder and classifier, even though F is small. Therefore the abstract's statement that TITAN-Guide 'eliminates the need for backpropagation' is true only for the random- and score-based guesses, not for the variant that achieves the best scores in Tables 1–2. The authors should clarify this and report the memory and runtime overhead of computing this guess separately.
  4. [§5.2, Metrics] The guidance objectives (ImageBind, DOVER, Style-CLIP) are also used as evaluation metrics (IB-AV, IB-TV, DOVER, ED in Tables 1–2). Hence part of the reported alignment improvement simply measures how well the optimizer minimized its own loss. Because FVD and KVD are the only external metrics, the authors should either add independent metrics (e.g., human preference or a different video-quality model) or qualify the alignment-based claims accordingly.
  5. [Figure 4] The toy experiment is presented as evidence that score-based guesses approximate ground-truth gradients, but it is qualitative, uses a single seed, and does not quantify the cosine similarity or step-size behavior between G_t and ∇f. Given the central role of this assumption in the paper, a quantitative comparison would be more convincing.
minor comments (6)
  1. [Abstract] The first sentence is grammatically incomplete: 'In the recent development of conditional diffusion models still require heavy supervised fine-tuning' should be rephrased, for instance as 'Recent conditional diffusion models still require heavy supervised fine-tuning...'.
  2. [Equation (4)] Equation (4) contains a spurious negative sign: 'Zt ← -Zt - λt∇ZtL(...)' should presumably be 'Zt ← Zt - λt∇ZtL(...)', consistent with Algorithm 1 and the rest of the paper.
  3. [Section 5.1] The text says the T2V models are 'noth effective' — this is a typo and should read 'both effective'.
  4. [Section 9] The phrase 'ig.8' should be 'Fig. 8'.
  5. [Table 3 and surrounding text] The ablation over iterations is non-monotonic (FVD 348.26 at 20, 340.21 at 50, 350.52 at 100) but the text says improvements are marginal; please explain the degradation at 100 iterations.
  6. [Appendix 8] The hyperparameter section states λ_t = 0.1 and F = 2 for sampled-gradient guesses, but no sensitivity analysis for F is provided, although F affects the gradient-approximation quality and memory cost.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation-chain circularity; one minor evaluation overlap from using guidance losses as metrics.

  1. other [Section 5.1 (Guidance tasks and Metrics); Tables 1-2]
    "For classifier guidance, we employed off-the-shelf discriminative models: 1) ImageBind [10] for audio-text-video alignment tasks, 2) DOVER [29] to enhance aesthetic aspects e.g., semantics and composition, and 3) Style-CLIP [21] to control the style of generated videos. // For multi-modal alignment, we used ImageBind scores [10] to measure text-video (IB-TV) and audio-video (IB-A V) correspondences."

    The guidance update in Algorithm 1 minimizes f(D(hat Z_0|t), y, psi) with psi equal to ImageBind, DOVER, or Style-CLIP, and Tables 1-2 then report IB-TV, IB-AV, DOVER, and style-feature ED as evidence of success. Because these scores are the same cosine-similarity losses being optimized, the reported alignment and aesthetic gains partially measure the optimizer's own objective by construction. This is a real but minor evaluation overlap: FVD/KVD and AV-Align are external, no constants are fitted to the benchmarks, and the forward-gradient derivation itself does not reduce to these metrics.

full rationale

TITAN-Guide's central derivation is self-contained rather than circular. Equation (6) defines the forward-gradient estimator G(Z) = <nabla f(Z), V>V as the standard rank-one projection, and Eq. (9) is an explicit heuristic choice of tangent, not a parameter fitted to the reported data. The memory-efficiency claim follows from forward-mode AD and is verified by direct measurement, and FVD/KVD/AV-Align are external benchmarks. The score-based guess is an accuracy/robustness risk (a projection can be small or misaligned), but that is not circularity. The only mild issue is that the guidance models double as evaluation metrics, so those specific numbers partly reflect the optimizer minimizing its own loss. That does not invalidate the central method, which is an empirical optimization result rather than a derivation that reduces to its inputs.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the validity of forward-gradient approximations for latent updates, the differentiability of off-the-shelf classifiers, and the usefulness of unrolled clean latents as guidance signals. No new physical or architectural entities are postulated; the free parameters are standard guidance hyperparameters chosen by hand.

free parameters (3)
  • guidance learning rate lambda_t = 0.1
    Set to 0.1 for all experiments in Section 8; no sensitivity analysis is shown, and the value is chosen by hand.
  • number of guidance iterations = 20 (ablated 20/50/100 in Table 3)
    The ablation shows marginal improvement with more iterations, so 20 is used as a fixed setting; the exact role of 'iterations' in the algorithm is not fully specified.
  • number of sampled frames F for sampled-gradient guess = 2
    Section 8 states F=2 for the sampled-gradient guess, chosen without detailed analysis of its impact.
assumptions (3)
  • ad hoc to paper The forward gradient G = ⟨∇f, V⟩V approximates the true gradient ∇f for the chosen guess vectors V.
    For random V ~ N(0,I) this is unbiased but high variance; for score-based guesses V = εθ/||εθ|| no such guarantee is given. Used in Eq. (6) and Algorithm 1.
  • domain assumption Unrolling the denoiser from Z_t to t=0 produces a clean latent whose decoded video is a valid input for off-the-shelf classifiers.
    The paper acknowledges in Fig. 2 that early clean estimates are incomplete, yet the method relies on the unrolled output for gradients at every step. Section 4.1, Eq. (7-8).
  • domain assumption Off-the-shelf classifiers ψ (ImageBind, DOVER, Style-CLIP) are differentiable with respect to the decoded video input.
    Required for computing JVP through the full pipeline; not verified in the paper, though standard for such models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Models." pith.science (2026). https://pith.science/paper/INYF2CRO

@misc{pith2026250800289,
  author       = {Pith},
  title        = {Pith review of: TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/INYF2CRO}},
  note         = {Machine review of arXiv:2508.00289}
}
read the original abstract

In the recent development of conditional diffusion models still require heavy supervised fine-tuning for performing control on a category of tasks. Training-free conditioning via guidance with off-the-shelf models is a favorable alternative to avoid further fine-tuning on the base model. However, the existing training-free guidance frameworks either have heavy memory requirements or offer sub-optimal control due to rough estimation. These shortcomings limit the applicability to control diffusion models that require intense computation, such as Text-to-Video (T2V) diffusion models. In this work, we propose Taming Inference Time Alignment for Guided Text-to-Video Diffusion Model, so-called TITAN-Guide, which overcomes memory space issues, and provides more optimal control in the guidance process compared to the counterparts. In particular, we develop an efficient method for optimizing diffusion latents without backpropagation from a discriminative guiding model. In particular, we study forward gradient descents for guided diffusion tasks with various options on directional directives. In our experiments, we demonstrate the effectiveness of our approach in efficiently managing memory during latent optimization, while previous methods fall short. Our proposed approach not only minimizes memory requirements but also significantly enhances T2V performance across a range of diffusion guidance benchmarks. Code, models, and demo are available at https://titanguide.github.io.

Figures

Figures reproduced from arXiv: 2508.00289 by the authors.

Figure 1
Figure 1. Various guided diffusion tasks with objectives to steer the output of generated videos and performance comparison with the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Projection of clean data estimation {\boldsymbol {z}}_{0|t} and latent noise {\boldsymbol {z}}_{t} onto video {\boldsymbol {X}}. Since off-the-shelf classifier models are not specifically designed for corrupted visual data, evaluating clean estimation {\boldsymbol {z}}_{0|t} in early iterations propagates false signals, as indicated by low text alignment scores (e.g., using ImageBind [10]). However, steering diffusi… view at source ↗
Figure 3
Figure 3. Comparison of TITAN-Guide with previous diffusion guidance methods. (a) Standard classifier guidance, with which latents are [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Experiments on toy data (i.e., Moons) using diffusion models, with original data in blue and generated samples in orange. We compare guided diffusion against true gradients: (a) Original distribution and generated samples, with objective function in red. TITAN￾Guide wi…
Figure 5
Figure 5. Figure 5: Results of our proposed approach compared with Seeing-and-Hearing [ [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparisons on T2V guidance: (a) Aesthetic [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Memory consumption (in GB) for various classifier [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Qualitative results of TITAN-Guide on style guidance. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Qualitative results of TITAN-Guide on frame interpo [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Qualitative results of TITAN-Guide for audio-video alignment at 384 [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Qualitative results of TITAN-Guide for audio-video alignment at 384 [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 31 canonical work pages

  1. [1]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 6

    arXiv 2023

  2. [2]

    Universal guidance for diffusion models

    Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geip- ing, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 843–852,

  3. [3]

    Gradients without backprop- agation

    Atılım G ¨unes ¸ Baydin, Barak A Pearlmutter, Don Syme, Frank Wood, and Philip Torr. Gradients without backprop- agation. arXiv preprint arXiv:2202.08587 , 2022. 2, 3, 5, 10

    arXiv 2022

  4. [4]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zis- serman. Vggsound: A large-scale audio-visual dataset. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020. 1, 5, 6, 11

    work page 2020

  5. [5]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shil- iang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023. 6

    arXiv 2023

  6. [6]

    Flatten: optical flow- guided attention for consistent text-to-video editing

    Yuren Cong, Mengmeng Xu, Christian Simon, Shoufa Chen, Jiawei Ren, Yanping Xie, Juan-Manuel Perez-Rua, Bodo Rosenhahn, Tao Xiang, and Sen He. Flatten: optical flow- guided attention for consistent text-to-video editing. arXiv preprint arXiv:2310.05922, 2023. 2

    arXiv

  7. [7]

    Tweedie’s formula and selection bias

    Bradley Efron. Tweedie’s formula and selection bias. Jour- nal of the American Statistical Association , 106:1602 – 1614, 2011. 2, 3, 5

    work page 2011

  8. [8]

    Douceur, Jon Howell, and Jared Saul

    Jeremy Elson, John R. Douceur, Jon Howell, and Jared Saul. Asirra: a captcha that exploits interest-aligned manual image categorization. In Conference on Computer and Communi- cations Security, 2007. 12

    work page 2007

Show all 44 references
  1. [9]

    Can forward gra- dient match backpropagation? In International Conference on Machine Learning, pages 10249–10264

    Louis Fournier, St ´ephane Rivaud, Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Can forward gra- dient match backpropagation? In International Conference on Machine Learning, pages 10249–10264. PMLR, 2023. 5

    2023

  2. [10]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, 2023. 3, 4, 6, 11

    2023

  3. [11]

    Animatediff: Animate your personalized text-to- image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to- image diffusion models without specific tuning. Interna- tional Conference on Learning Representations, 2024. 4, 6, 8, 10

    2024

  4. [12]

    Manifold preserving guided diffusion

    Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, J Zico Kolter, Ruslan Salakhutdinov, and Ste- fano Ermon. Manifold preserving guided diffusion. In The Twelfth International Conference on Learning Represent...

    2024

  5. [13]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 10, 12

    2020

  6. [14]

    Deep learning face attributes in the wild

    Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015. 12

    2015

  7. [15]

    Sdedit: Guided image synthesis and editing with stochastic differential equa- tions

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jia- jun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equa- tions. arXiv preprint arXiv:2108.01073, 2021. 2

    2021 arXiv

  8. [16]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI conference on artificial intelligence, pages 4296–4304, 2024. 2

    2024

  9. [17]

    Patel, and Tim K

    Nithin Gopalakrishnan Nair, Anoop Cherian, Suhas Lo- hit, Ye Wang, Toshiaki Koike-Akino, Vishal M. Patel, and Tim K. Marks. Steered diffusion: A generalized framework for plug-and-play conditional image synthesis. In Proceed- ings of the IEEE/CVF International Conference on Co...

    2023

  10. [18]

    Steered diffusion: A generalized framework for plug- and-play conditional image synthesis

    Nithin Gopalakrishnan Nair, Anoop Cherian, Suhas Lohit, Ye Wang, Toshiaki Koike-Akino, Vishal M Patel, and Tim K Marks. Steered diffusion: A generalized framework for plug- and-play conditional image synthesis. In Proceedings of the IEEE/CVF International Conference on Compute...

    2023

  11. [19]

    Ditto: Diffusion inference-time t- optimization for music generation

    Zachary Novack, Julian McAuley, Taylor Berg-Kirkpatrick, and Nicholas J Bryan. Ditto: Diffusion inference-time t- optimization for music generation. In ICML, 2024. 2, 4

    2024

  12. [20]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Al- ban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017. 10

    2017

  13. [21]

    Styleclip: Text-driven manipulation of stylegan imagery

    Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 2085–2094,

  14. [22]

    Scaling forward gradient with local losses

    Mengye Ren, Simon Kornblith, Renjie Liao, and Geoffrey Hinton. Scaling forward gradient with local losses. arXiv preprint arXiv:2210.03310, 2022. 5

    2022 arXiv

  15. [23]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 2, 12

    2010 arXiv

  16. [24]

    Loss-guided diffusion models for plug-and-play controllable generation

    Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mar- dani, Ming-Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. In Proceedings of the 40th Inter- national Conference on Machine Learning , pages 32483–...

    2023

  17. [25]

    Score-based generative modeling through stochastic differential equa- tions

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions. arXiv preprint arXiv:2011.13456, 2020. 2, 4

    2011 arXiv

  18. [26]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 6

    2018 arXiv

  19. [27]

    Edict: Ex- act diffusion inversion via coupled transformations

    Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Ex- act diffusion inversion via coupled transformations. arXiv preprint arXiv:2211.12446, 2022. 2

    2022 arXiv

  20. [28]

    End-to-end diffusion latent optimization improves classifier guidance

    Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent optimization improves classifier guidance. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 7280–7290,

  21. [29]

    Exploring video quality assessment on user generated contents from aesthetic and technical perspectives

    Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jing- wen Hou Hou, Annan Wang, Wenxiu Sun Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. In International Conference on Computer Vision (ICCV) ,

  22. [30]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...

    2023

  23. [31]

    Cvpr 2023 text guided video editing competition, 2023

    Jay Zhangjie Wu, Xiuyu Li, Difei Gao, Zhen Dong, Jin- bin Bai, Aishani Singh, Xiaoyu Xiang, Youzeng Li, Zuwei Huang, Yuanxi Sun, Rui He, Feng Hu, Junhua Hu, Hai Huang, Hanyu Zhu, Xu Cheng, Jie Tang, Mike Zheng Shou, Kurt Keutzer, and Forrest Iandola. Cvpr 2023 text guided vide...

    2023

  24. [32]

    Seeing and hearing: Open-domain visual- audio generation with diffusion latent aligners

    Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open-domain visual- audio generation with diffusion latent aligners. In CVPR,

  25. [33]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 4, 6

    2024 arXiv

  26. [34]

    Diverse and aligned audio-to-video genera- tion via text-to-video model adaptation, 2023

    Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi. Diverse and aligned audio-to-video genera- tion via text-to-video model adaptation, 2023. 6

    2023

  27. [35]

    Tfg: Unified training-free guidance for diffusion models

    Haotian Ye, Haowei Lin, Jiaqi Han, Minkai Xu, Sheng Liu, Yitao Liang, Jianzhu Ma, James Zou, and Stefano Ermon. Tfg: Unified training-free guidance for diffusion models

  28. [36]

    Freedom: Training-free energy-guided condi- tional diffusion model

    Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-free energy-guided condi- tional diffusion model. Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV) , 2023. 2, 3, 4, 6, 7, 8, 12

    2023

  29. [37]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In IEEE International Conference on Computer Vision (ICCV),

  30. [40]

    Let f : Rm − →Rn

    The Details of Forward Gradient Descents In this section, we provide detailed explanation on using forward AD to estimate gradients. Let f : Rm − →Rn. The directional gradient along V evaluated at X can be defined as: f ′(X) = lim δ→0 f (X + δV ) − f (X) δ . (10) The forward g...

  31. [41]

    In all experiments, we em- ploy AnimateDiff [11] with epiCRealism 2 as the base text- to-image model to generate 16 frames 8fps

    Experimental Settings Text-to-Video (T2V) Models. In all experiments, we em- ploy AnimateDiff [11] with epiCRealism 2 as the base text- to-image model to generate 16 frames 8fps. We set the de- noising process to 20 iterations, as we found this to be suf- ficient for generatin...

  32. [42]

    A bird in a forest

    Additional Results on Video Generation In this section, we present our qualitative results, with addi- tional videos available in the supplementary material. We also provide our generated video samples at our project page3. Qualitative results. We present qualitative results i...

  33. [43]

    However, since our primary focus is text- to-video tasks, we do not explore this aspect in depth but provide evidence of its applicability

    Additional Results on Image Generation We also demonstrate that our proposed approach has the potential to be effectively applied to the image diffusion models [13, 23]. However, since our primary focus is text- to-video tasks, we do not explore this aspect in depth but provid...

  34. [44]

    Following [35], for super resolution, and deblurring tasks, we use the CAT-DDPM diffusion model trained on the CAT dataset [8]

    Super resolution, 2) CelebA (guided by gender and age specification), and deblurring. Following [35], for super resolution, and deblurring tasks, we use the CAT-DDPM diffusion model trained on the CAT dataset [8]. While, we use CelebA-DDPM trained on the CelebA dataset [14] fo...

  35. [2023]

    2, 4, 5, 6, 7, 8, 12

  36. [2024]

    2, 3, 4, 5, 6, 7, 8, 11, 12

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.