REVIEW 3 major objections 4 minor 13 cited by
StableAnimator: High-Quality Identity-Preserving Human Image Animation
T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read StableAnimator is an end-to-end video diffusion framework that animates a reference person through pose sequences while keeping the face recognizable, and reports beating post-processing-based systems on identity and video-fidelity metrics.
desk verdict A strong, honest empirical systems contribution whose HJB theory section does not survive scrutiny; treat it as a heuristic latent optimization and judge it on the numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Three components carry the argument. The global content-aware Face Encoder is a stack of cross-attention blocks that refines ArcFace embeddings by attending to CLIP image embeddings, giving identity features awareness of the reference layout. The distribution-aware ID Adapter aligns the face cross-attention output to the image cross-attention output by matching per-feature mean and standard deviation before feeding the sum to temporal layers, which prevents the temporal layers from shifting the identity features out of the diffusion latents' domain. The inference-time HJB equation-based optimization treats the predicted clean sample at each denoising step as the control variable of an optimal-control problem whose terminal cost is face similarity to the reference; solving that problem yields a drift $(1-t)\nabla\log p(X_t)\,dt + dw_t$, which the paper argues is structurally the score-based reverse SDE, so the optimized prediction can be folded into the EDM sampler as a steering gradient.
What would settle it
Re-derive the claimed equality with the diffusion noise coefficient $g^2$ retained: Eq. (12) as written omits $g^2$ and flips a sign relative to the standard reverse SDE, so a direct symbolic check of whether the optimizer's update is a solution of Eq. (4) would settle the theoretical claim. Empirically, one could run Algorithm 2 with the ArcFace gradient direction reversed or randomly perturbed; if identity scores stay high, the face-optimization loop is not what carries the reported identity preservation.
Extended reading notes
Core claim
In the paper's own framing, the discovery is that identity preservation and video fidelity can be treated as a distribution-matching problem rather than a post-processing problem. A global content-aware Face Encoder first lets face embeddings attend to the full-reference image embedding, so the identity signal carries context. A distribution-aware ID Adapter then computes separate cross-attention outputs for face and image embeddings and normalizes the face-attention features into the image-attention feature distribution before temporal layers act, so temporal modeling no longer corrupts the injected identity. At inference, the paper performs ten Adam steps on each denoised latent prediction, minimizing the ArcFace cosine distance to the reference, and derives this update from an HJB optimal-control problem whose solution it shows has the same structure as the score-based diffusion reverse SDE. The result, as reported, is end-to-end animation with no third-party face-swapping and the highest identity and video-fidelity scores among the compared methods.
Load-bearing premise
The load-bearing premise is that the per-step latent optimization genuinely solves the Hamilton-Jacobi-Bellman optimal-control problem and that the resulting drift is structurally the same as the diffusion reverse SDE; this equivalence is asserted rather than rigorously proven, and if it fails the method's theoretical foundation loses its basis even if the empirical face-similarity updates still help.
Editorial extensions
If this is right
- If the central claim is right, identity-consistent human animation can be produced in one pass through a video diffusion model, eliminating the quality loss and domain mismatch that face-swapping post-processing introduces.
- The distribution-alignment recipe gives a direct way to port image-domain identity adapters into video diffusion: align injected feature distributions with spatial latents before temporal layers.
- Because the HJB optimization is training-free and applied at inference, it can be attached to other diffusion backbones, as the paper demonstrates with its cross-backbone ablations.
- The reported gains, with CSIM rising from 0.360 to 0.831 on TikTok, indicate that identity-consistency headroom is large and that face-similarity metrics are not saturated by existing baselines.
- Long sequences, including animations of over 300 frames and multi-person scenes, are claimed to remain identity-consistent, extending the method beyond short single-person clips.
Reading between the lines
- A reasonable reading is that the empirical power of the inference step comes from the direct ArcFace-similarity gradient itself; the HJB derivation gives a principled framing, but the update would likely still improve identity even if the optimal-control equivalence fails.
- The distribution-alignment idea is not specific to faces: any video-diffusion task that injects per-frame conditions into a U-Net with temporal layers, for example audio-driven expression or text-driven editing, could use the same mean-variance alignment before temporal processing.
- A testable extension: measure identity preservation with a face embedder different from ArcFace, since the inference objective optimizes ArcFace similarity directly and part of the reported gain could be embedder-specific.
- The face-inpainting extension the paper mentions in its limitation could be combined with the same HJB steering, turning a stated failure case into a targeted adapter rather than a general post-processing step.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. StableAnimator proposes an end-to-end identity-preserving human image animation framework built on Stable Video Diffusion, with three main contributions: a global content-aware Face Encoder that refines face embeddings using reference-image context, a distribution-aware ID Adapter that aligns face and image cross-attention features before temporal modeling, and an inference-time face optimization that the authors formulate as a Hamilton-Jacobi-Bellman (HJB) equation-based control problem. The model is trained with a face-mask-weighted reconstruction loss and evaluated on the TikTok dataset and an additional Unseen100 benchmark, reporting state-of-the-art CSIM and FVD values. The paper includes extensive ablations showing that each component contributes positively and that the inference-time optimization also improves other backbones.
Significance. If the empirical claims hold, StableAnimator addresses a practical problem—identity consistency in pose-driven human animation—without relying on external face-swapping post-processing, and its distribution-aware adapter is a plausible mechanism for injecting identity features into video diffusion models. The paper provides a detailed algorithm description, comprehensive ablations, and an honest discussion of limitations and ethical concerns. However, the central theoretical contribution, the claimed integration of HJB-based optimal control with diffusion denoising, is not rigorously established; the derivation in Sec. 3.2 is internally inconsistent with the reverse SDE it invokes, and Algorithm 2 is a heuristic latent-space gradient descent rather than a solution of the stated control problem. The empirical results are valuable regardless of the HJB framing, but the current presentation overstates the theoretical novelty.
major comments (3)
- [Sec. 3.2, Eqs. (9)-(12)] The claimed equivalence between the derived SDE and the score-based reverse SDE is not established and is internally inconsistent. In Eq. (12), the Brownian term is dw_t, so the diffusion coefficient g^2 would be 1, in which case the drift coefficient of ∇log p in the standard reverse SDE (Eq. 16 in Sec. A.2) should be g^2 (up to sign), not (1−t). If one instead intends g^2=(1−t), the noise term must be √(1−t) dw_t, not dw_t. The step labeled 'certainty equivalence' (adding Brownian motion to the deterministic optimal-control solution) is asserted without a proof or a precise citation, so the derivation does not demonstrate that solving Eq. (4) corresponds to diffusion denoising. This undermines the claim in Sec. 3.2 that 'solving the HJB equation can be integrated with the diffusion denoising process.'
- [Sec. 3.2, Algorithm 1 (and Algorithm 2 in Sec. A.6)] Algorithm 2 does not solve the optimal-control problem in Eq. (4). The control variable c_t is never defined in the algorithm, and the optimization performed is 10 Adam steps on an ArcFace cosine loss applied to a trainable copy of the diffusion model's prediction xpred. The statement 'xpred in Algorithm 2 is equivalent to x1' is unjustified: x1 in Eq. (4) is the terminal target of the control problem, whereas xpred after optimization is a face-similarity-optimized latent that is neither the reference latent nor a Tweedie estimate of the data. No Hamiltonian, terminal condition, or trajectory constraint appears in the algorithm. Therefore, the paper does not prove that the proposed optimization is HJB-based; at best it is an inference-time face-similarity guidance heuristic.
- [Table 1 and Sec. 4.2] The reported comparisons on the TikTok dataset are based on numbers taken from the competitors' papers, and no error bars, confidence intervals, or repeated-seed statistics are provided for any metric. Since the headline claim is a large improvement (e.g., CSIM 0.831 vs. 0.360 for ControlNeXt), the lack of uncertainty quantification and the inconsistent evaluation protocols across cited results make the exact magnitude of the improvement difficult to verify. The authors should report variance across multiple runs or at least state the number of evaluation videos and run the main competitors under a unified protocol, as they did for Unseen100.
minor comments (4)
- [Sec. 3.2, notation] The time variable t is overloaded: it denotes both the normalized time in the control problem (running from 0 to 1) and the EDM noise level (with σ(t)=t). This makes equations such as Eq. (11) confusing; please use distinct symbols for the control-time and the diffusion time.
- [Eq. (7)] In Hamilton's equations, dγ/dt should equal −∂H*/∂X, not ∂H*/∂X as written; the result is the same here because ∂H*/∂X=0, but the sign error should be corrected.
- [Sec. 4.3, Table 5] The cross-backbone experiments are informative, but the hyperparameters of the proposed optimization (Adam learning rate η and the number of inner steps k=10) are never varied or analyzed. A short sensitivity study would clarify how robust the reported gains are to these choices.
- [Supplementary A.8] Two consecutive sentences are duplicated verbatim ('The reason is that applying the ArcFace ID loss requires...'). This should be fixed editorially.
Circularity Check
Partial circularity: the HJB 'rationale' in Sec. 3.2 installs its own conclusion by defining the terminal target x₁ as the face-optimized xpred of Algorithm 2 and renames the known score-SDE drift as an HJB solution; the Face Encoder, ID Adapter, and benchmark results are independent.
-
self definitional
[Sec. 3.2 'ID-preserving During Inference' (around Eqs. 3–4); Algorithm 2 (App. A.6)]
"The variable refers to the predicted sample by the diffusion model at each denoising iteration. ... Xt and xt refer to the groundtruth sample and the predicted sample by the model. Thus, xpred in Algorithm 2 is equivalent to x1. // Algorithm 2: loss = (1 − Cos(Arc(fpred), Arc(y))).abs().mean() ... xpred = xop"
The HJB cost in Eq. 4, min ∫½‖c‖²dt + (r/2)‖X₁ − x₁‖², contains no face-similarity term; identity enters the derivation only through the terminal target x₁, which the text declares 'equivalent to x1,' i.e., to xpred of Algorithm 2. But Algorithm 2 replaces xpred with ten Adam steps on (1 − Cos(Arc(Decoder(xop)), Arc(y))).abs().mean(), minimizing the reference-face distance. Consequently the r→∞ drift (x₁ − X_t)/(1 − t) points at the face-optimized latent, not at any state the HJB mathematics selected. The conclusion that solving the HJB equation 'constrains the denoising path and thus benefits ID preservation' is therefore installed by the definition x₁ := xpred := face-optimized latent: drop that identification and Eq.
-
renaming known result
[Sec. 3.2, Eqs. 9–12 and the closing assertion; cf. Eq. 16 (App. A.2)]
"It is evident that Eq. 12 and SDE formulation [42] are structurally the same, thus we can seamlessly incorporate the solution process of the HJB equation into the diffusion denoising for ID preservation."
The announced proof that 'solving the HJB equation can be integrated with the diffusion denoising process' reduces to (i) substituting the Tweedie identity X₁ = X_t + (1 − t)²∇log p(X_t) into the r→∞ optimal drift to obtain Eq. 12, and (ii) asserting 'It is evident that Eq. 12 and SDE formulation [42] are structurally the same.' Move (i) yields the standard score-SDE drift in normalized time — a known result [42], not a new implication of the HJB formalism. Move (ii) is inconsistent with the paper's own Eq. 16: with noise dw_t the volatility is g = 1, so the ∇log p drift coefficient would have to be g² = 1, not (1 − t).
full rationale
Most of StableAnimator is self-contained and honestly evaluated. The global content-aware Face Encoder, the Distribution-aware ID Adapter (the statistics alignment of Eq. 2), and the face-mask reconstruction loss (Eq. 13) are trainable modules ablated on the held-out Unseen100 set; no parameter is fitted to a benchmark subset and then re-reported as a prediction, and the appended limitations (A.12: hand-occluded faces fail; A.8: ArcFace ID loss omitted from training for memory reasons) are stated honestly. The authors' self-citations ([43, 44, 51, 53, 54, 55, 56]) appear only in related-work context and are not load-bearing, and no uniqueness or existence theorem is imported from the authors' own prior work, so the self-citation patterns do not apply. Circularity is confined to the HJB 'rationale' in Sec. 3.2. First, the HJB cost of Eq. 4 has no face term — min ∫½‖c‖²dt + (r/2)‖X₁ − x₁‖² — so identity preservation can enter only through the terminal target x₁, which the paper equates with xpred of Algorithm 2; but Algorithm 2 overwrites xpred with ten Adam steps on the ArcFace cosine loss (1 − Cos(Arc(Decoder(xop)), Arc(y))).abs().mean(). The conclusion that 'solving the HJB equation ... constrains the denoising path and thus benefits ID preservation' is therefore installed by definition (x₁ := xpred := face-optimized latent), and the r→∞ drift (x₁ − X_t)/(1 − t) merely points toward that optimized point. Second, Eq. 12 is the Tweedie identity rearranged into a score-SDE drift, and the asserted structural sameness with [42] is inconsistent with the paper's own reverse SDE (Eq. 16: g = 1 implies a ∇log p drift of 1, not (1 − t)). The HJB derivation is thus a post-hoc wrapper around a heuristic latent-space gradient loop, reversing the claimed direction of support. A metric-alignment note: the inference loss is the negative of the headline CSIM metric, so part of the CSIM gap is by construction of the test-time objective, though the w/o-optimization ablation (CSIM 0.782 vs 0.805) shows the trained model already carries most of the identity signal. Because the training-stage contributions and the benchmark numbers are independent of the HJB narrative, this is partial circularity, not a full collapse.
Assumptions & free parameters
free parameters (3)
- Adam learning rate η in HJB optimization =
not reported
- Inner optimization iterations k =
10
- EDM sampling hyperparameters (Snoise, Schurn, Stmin, Stmax) =
standard values from EDM [26]
assumptions (4)
- standard math Pontryagin's Maximum Principle gives the optimal control c_t = γ for the linear-quadratic terminal-cost problem in Eq. 4.
- standard math Tweedie's formula E[X1|Xt] = Xt + (1-t)^2 ∇ log p(Xt) holds for the scaled noise process Xt = X1 + (1-t)ε.
- ad hoc to paper The denoising process in SVD/EDM can be reparameterized as Xt = X1 + (1-t)ε with t = 1 - t', and the reverse SDE structurally matches the HJB-derived dynamics.
- domain assumption The temporal layers only distort the feature distribution in an affine (mean/variance) way, so aligning means/variances of face and image cross-attention outputs suffices to prevent ID loss.
Cite this review
Pith. "Pith review of StableAnimator: High-Quality Identity-Preserving Human Image Animation." pith.science (2026). https://pith.science/paper/TBRSO6JF
@misc{pith2026241117697,
author = {Pith},
title = {Pith review of: StableAnimator: High-Quality Identity-Preserving Human Image Animation},
year = {2026},
howpublished = {\url{https://pith.science/paper/TBRSO6JF}},
note = {Machine review of arXiv:2411.17697}
}
read the original abstract
Current diffusion models for human image animation struggle to ensure identity (ID) consistency. This paper presents StableAnimator, the first end-to-end ID-preserving video diffusion framework, which synthesizes high-quality videos without any post-processing, conditioned on a reference image and a sequence of poses. Building upon a video diffusion model, StableAnimator contains carefully designed modules for both training and inference striving for identity consistency. In particular, StableAnimator begins by computing image and face embeddings with off-the-shelf extractors, respectively and face embeddings are further refined by interacting with image embeddings using a global content-aware Face Encoder. Then, StableAnimator introduces a novel distribution-aware ID Adapter that prevents interference caused by temporal layers while preserving ID via alignment. During inference, we propose a novel Hamilton-Jacobi-Bellman (HJB) equation-based optimization to further enhance the face quality. We demonstrate that solving the HJB equation can be integrated into the diffusion denoising process, and the resulting solution constrains the denoising path and thus benefits ID preservation. Experiments on multiple benchmarks show the effectiveness of StableAnimator both qualitatively and quantitatively.
Figures
Figures from the paper (17 more)
Forward citations
Cited by 13 Pith papers
-
AgentHOI: Multi-Agent Reasoning for Human-Object-Interaction Video Generation via Implicit Representation Alignment
AgentHOI generates human-object interaction videos from text plus one human image and one object image, using multi-agent action planning and implicit text-to-motion feature alignment inside a video diffusion model.
-
PERSONA: Personalized Whole-Body 3D Avatar with Pose-Driven Deformations from a Single Image
PERSONA creates a personalized 3D avatar from one image by using diffusion-generated pose-rich videos to train a 3D Gaussian avatar with balanced sampling and geometry-weighted optimization.
-
DreamActor-H1: High-Fidelity Human-Product Demonstration Video Generation via Motion-designed Diffusion Transformers
A diffusion transformer model generates human-product demonstration videos from paired human and product images while preserving both identities through masked cross-attention and motion template guidance.
-
HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animation
HunyuanVideo-HOMA generates human-object interaction videos from weak, sparse inputs: one arm pose, an object center dot, a human photo, and an object photo.
-
AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models
A diffusion model animates a character into arbitrary dynamic backgrounds by conditioning on a rendered 3D-avatar video, reframing open-domain animation as a restoration problem.
-
DanceTogether! Identity-Preserving Multi-Person Interactive Video Generation
A diffusion model fuses per-person masks with pose keypoints to generate identity-preserving, two-person interactive videos from a single reference image, outperforming prior single-person-animation pipelines.
-
FlexiAct: Towards Flexible Action Control in Heterogeneous Scenarios
FlexiAct transfers actions from a reference video to an arbitrary target image, allowing changes in layout, skeleton, and viewpoint while keeping the target subject's appearance.
-
EchoVideo: Identity-Preserving Human Video Generation by Multimodal Feature Fusion
EchoVideo preserves identity in generated human videos by pre-fusing face, image, and text features, then training with stochastic shallow-feature dropout to reduce copy-paste artifacts.
-
StableAnimator++: Overcoming Pose Misalignment and Face Distortion for Human Image Animation
StableAnimator++ combines learnable SVD-guided pose alignment, a distribution-aware ID Adapter, and an HJB-based inference-time face optimizer to preserve identity in human image animation under severe pose misalignment.
-
Human Motion Video Generation: A Survey
A comprehensive survey with a five-phase pipeline model for human motion video generation, covering over 200 papers and adding a new benchmark comparison of nine pose-guided methods.
-
Preview WB-DH: Towards Whole Body Digital Human Bench for the Generation of Whole-body Talking Avatar Videos
The paper previews a claimed 2M-clip multimodal benchmark for whole-body talking avatar video generation, with standard metrics and an initial evaluation of eight open-source models.
-
JWB-DH-V1: Benchmark for Joint Whole-Body Talking Avatar and Speech Generation Version 1
A paper announcing a large-scale whole-body talking avatar benchmark and evaluation protocol, but with insufficient details to verify the dataset or the joint audio-video evaluation.
-
EgoAnimate: Generating Human Animations from Egocentric top-down Views
EgoAnimate synthesizes a frontal T-pose image from an egocentric top-down photo using a fine-tuned Stable Diffusion model, then animates it with off-the-shelf image-to-motion methods to produce an animatable avatar.
Reference graph
Works this paper leans on
-
[1]
Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models
Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233, 2024. 2
arXiv 2024
-
[2]
Optimal control and viscosity solutions of Hamilton-Jacobi-Bellman equa- tions
Martino Bardi, Italo Capuzzo Dolcetta, et al. Optimal control and viscosity solutions of Hamilton-Jacobi-Bellman equa- tions. Springer, 1997. 2, 4, 5
work page 1997
-
[3]
Stable video diffusion: Scaling latent video diffusion models to large datasets
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 , 2023. 2, 3, 5, 6
arXiv 2023
-
[4]
Video generation models as world simulators
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators
-
[5]
Genera- tive modeling with phase stochastic bridges
Tianrong Chen, Jiatao Gu, Laurent Dinh, Evangelos A Theodorou, Joshua Susskind, and Shuangfei Zhai. Genera- tive modeling with phase stochastic bridges. In ICLR, 2024. 4, 5
work page 2024
-
[6]
Tri Dao and Albert Gu. Transformers are SSMs: General- ized models and efficient algorithms through structured state space duality. In ICML, 2024. 3
work page 2024
-
[7]
Arcface: Additive angular margin loss for deep face recognition
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In CVPR, 2019. 2, 3, 5
work page 2019
-
[8]
Diffusion models beat gans on image synthesis
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021. 1, 2
work page 2021
Show all 67 references
-
[9]
Tweedie’s formula and selection bias
Bradley Efron. Tweedie’s formula and selection bias. Jour- nal of the American Statistical Association, 2011. 5
2011
-
[10]
Deterministic and stochastic optimal control
Wendell H Fleming and Raymond W Rishel. Deterministic and stochastic optimal control. Springer Science & Business Media, 2012. 5
2012
-
[11]
Generative adversarial networks
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commu- nications of the ACM, 2020. 3
2020
-
[12]
Livepor- trait: Efficient portrait animation with stitching and retarget- ing control
Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Livepor- trait: Efficient portrait animation with stitching and retarget- ing control. arXiv preprint arXiv:2407.03168, 2024. 2, 6
2024 arXiv
-
[13]
Animatediff: Animate your personalized text-to- image diffusion models without specific tuning
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to- image diffusion models without specific tuning. In ICLR,
-
[14]
Pulid: Pure and lightning id customization via con- trastive alignment
Zinan Guo, Yanze Wu, Zhuowei Chen, Lang Chen, and Qian He. Pulid: Pure and lightning id customization via con- trastive alignment. In NeurIPS, 2024. 2, 4
2024
-
[15]
Facefusion
Ruhs Henry. Facefusion. https://github.com/ facefusion/facefusion, 2024. 1, 2, 3, 4, 7
2024
-
[16]
Prompt-to-prompt im- age editing with cross attention control
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 1, 2
2022 arXiv
-
[17]
Denoising diffu- sion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In NeurIPS, 2020
2020
-
[18]
Cascaded diffusion models for high fidelity image generation
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. JMLR, 2022. 1, 2
2022
-
[19]
Cogvideo: Large-scale pretraining for text-to-video generation via transformers
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022. 2
2022 arXiv
-
[20]
Image quality metrics: Psnr vs
Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, 2010. 6
2010
-
[21]
Lora: Low-rank adaptation of large language models
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2021. 2
2021
-
[22]
Animate anyone: Consistent and controllable image- to-video synthesis for character animation
Li Hu. Animate anyone: Consistent and controllable image- to-video synthesis for character animation. In CVPR, 2024. 1, 3, 5, 6, 8
2024
-
[23]
Consistentid: Portrait generation with multimodal fine-grained identity preserving
Jiehui Huang, Xiao Dong, Wenhui Song, Hanhui Li, Jun Zhou, Yuhao Cheng, Shutao Liao, Long Chen, Yiqiang Yan, Shengcai Liao, et al. Consistentid: Portrait generation with multimodal fine-grained identity preserving. arXiv preprint arXiv:2404.16771, 2024. 2, 4
2024 arXiv
-
[24]
Few- shot human motion transfer by personalized geometry and texture modeling
Zhichao Huang, Xintong Han, Jia Xu, and Tong Zhang. Few- shot human motion transfer by personalized geometry and texture modeling. In CVPR, 2021. 3
2021
-
[25]
Learning high fidelity depths of dressed humans by watching social media dance videos
Yasamin Jafarian and Hyun Soo Park. Learning high fidelity depths of dressed humans by watching social media dance videos. In CVPR, 2021. 2, 5, 6 9
2021
-
[26]
Elucidating the design space of diffusion-based generative models
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In NeurIPS, 2022. 4, 5
2022
-
[27]
Auto-encoding variational bayes
Diederik P Kingma. Auto-encoding variational bayes. In ICLR, 2014. 3
2014
-
[28]
Optimal control theory: an introduction
Donald E Kirk. Optimal control theory: an introduction . Courier Corporation, 2004. 5
2004
-
[29]
Photomaker: Customizing re- alistic human photos via stacked id embedding
Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming- Ming Cheng, and Ying Shan. Photomaker: Customizing re- alistic human photos via stacked id embedding. In CVPR,
-
[30]
Latte: Latent diffusion transformer for video generation
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Zi- wei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024. 2
2024 arXiv
-
[31]
Sdedit: Guided image synthesis and editing with stochastic differential equa- tions
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jia- jun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equa- tions. In ICLR, 2021. 1, 2
2021
-
[32]
Improved denoising diffusion probabilistic models
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021. 2
2021
-
[33]
Scalable diffusion models with transformers
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023. 2
2023
-
[34]
Controlnext: Powerful and effi- cient control for image and video generation
Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Ming- Chang Yang, and Jiaya Jia. Controlnext: Powerful and effi- cient control for image and video generation. arXiv preprint arXiv:2408.06070, 2024. 1, 2, 3, 4, 6, 8
2024 arXiv
-
[35]
Stochastic hamilton–jacobi–bellman equations
Shige Peng. Stochastic hamilton–jacobi–bellman equations. SIAM Journal on Control and Optimization, 1992. 2, 4, 5
1992
-
[36]
Learn- ing transferable visual models from natural language super- vision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 2, 3
2021
-
[37]
High-resolution image syn- thesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022. 1, 2, 8
2022
-
[38]
First order motion model for image animation
Aliaksandr Siarohin, St ´ephane Lathuili`ere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In NeurIPS, 2019. 1, 3, 4
2019
-
[39]
Motion representations for ar- ticulated animation
Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for ar- ticulated animation. In CVPR, 2021. 1, 3, 6
2021
-
[40]
Make-a-video: Text-to-video generation without text-video data
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,
-
[41]
Denois- ing diffusion implicit models
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In ICLR, 2021. 1, 2
2021
-
[42]
Score-based generative modeling through stochastic differential equa- tions
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions. In ICLR, 2021. 1, 2, 5
2021
-
[43]
Motioneditor: Editing video motion via content-aware diffusion
Shuyuan Tu, Qi Dai, Zhi-Qi Cheng, Han Hu, Xintong Han, Zuxuan Wu, and Yu-Gang Jiang. Motioneditor: Editing video motion via content-aware diffusion. In CVPR, 2024. 2
2024
-
[44]
Motionfollower: Editing video motion via lightweight score-guided diffusion
Shuyuan Tu, Qi Dai, Zihao Zhang, Sicheng Xie, Zhi-Qi Cheng, Chong Luo, Xintong Han, Zuxuan Wu, and Yu-Gang Jiang. Motionfollower: Editing video motion via lightweight score-guided diffusion. arXiv preprint arXiv:2405.20325 ,
-
[45]
Plug-and-play diffusion features for text-driven image-to-image translation
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In CVPR, 2023. 1, 2
2023
-
[46]
Instantid: Zero-shot identity-preserving gener- ation in seconds
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and An- thony Chen. Instantid: Zero-shot identity-preserving gener- ation in seconds. arXiv preprint arXiv:2401.07519, 2024. 2, 4
2024 arXiv
-
[47]
Disco: Disentangled control for realistic human dance generation
Tan Wang, Linjie Li, Kevin Lin, Yuanhao Zhai, Chung- Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for realistic human dance generation. In CVPR, 2024. 1, 3, 5, 6, 8
2024
-
[48]
Magicvideo-v2: Multi- stage high-aesthetic video generation
Weimin Wang, Jiawei Liu, Zhijie Lin, Jiangqiao Yan, Shuo Chen, Chetwin Low, Tuyen Hoang, Jie Wu, Jun Hao Liew, Hanshu Yan, et al. Magicvideo-v2: Multi- stage high-aesthetic video generation. arXiv preprint arXiv:2401.04468, 2024. 2
2024 arXiv
-
[49]
To- wards real-world blind face restoration with generative fa- cial prior
Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. To- wards real-world blind face restoration with generative fa- cial prior. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 1
2021
-
[50]
Unianimate: Taming unified video diffusion mod- els for consistent human image animation
Xiang Wang, Shiwei Zhang, Changxin Gao, Jiayu Wang, Xiaoqiang Zhou, Yingya Zhang, Luxin Yan, and Nong Sang. Unianimate: Taming unified video diffusion mod- els for consistent human image animation. arXiv preprint arXiv:2406.01188, 2024. 1, 3, 5, 6, 8
2024 arXiv
-
[51]
Genrec: Unifying video generation and recognition with diffusion models
Zejia Weng, Xitong Yang, Zhen Xing, Zuxuan Wu, and Yu-Gang Jiang. Genrec: Unifying video generation and recognition with diffusion models. arXiv preprint arXiv:2408.15241, 2024. 1
2024 arXiv
-
[52]
Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In CVPR, 2023. 2
2023
-
[53]
Vidiff: Translating videos via multi-modal instructions with diffusion models
Zhen Xing, Qi Dai, Zihao Zhang, Hui Zhang, Han Hu, Zux- uan Wu, and Yu-Gang Jiang. Vidiff: Translating videos via multi-modal instructions with diffusion models. arXiv preprint arXiv:2311.18837, 2023. 2
2023 arXiv
-
[54]
Simda: Simple diffusion adapter for efficient video generation
Zhen Xing, Qi Dai, Han Hu, Zuxuan Wu, and Yu-Gang Jiang. Simda: Simple diffusion adapter for efficient video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7827– 7839, 2024. 1
2024
-
[55]
Aid: Adapting image2video diffusion mod- els for instruction-guided video prediction
Zhen Xing, Qi Dai, Zejia Weng, Zuxuan Wu, and Yu- Gang Jiang. Aid: Adapting image2video diffusion mod- els for instruction-guided video prediction. arXiv preprint arXiv:2406.06465, 2024
2024 arXiv
-
[56]
A survey on video dif- fusion models
Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. A survey on video dif- fusion models. ACM Computing Surveys, 57(2):1–42, 2024. 1
2024
-
[57]
Magicanimate: Temporally consistent human image animation using diffusion model
Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng 10 Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In CVPR, 2024. 1, 3, 5, 6, 8
2024
-
[58]
Videogpt: Video generation using vq-vae and trans- formers
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and trans- formers. arXiv preprint arXiv:2104.10157, 2021. 2
2021 arXiv
-
[59]
Facestudio: Put your face everywhere in seconds
Yuxuan Yan, Chi Zhang, Rui Wang, Yichao Zhou, Gege Zhang, Pei Cheng, Gang Yu, and Bin Fu. Facestudio: Put your face everywhere in seconds. arXiv preprint arXiv:2312.02663, 2023. 2
2023 arXiv
-
[60]
Effec- tive whole-body pose estimation with two-stages distillation
Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effec- tive whole-body pose estimation with two-stages distillation. In ICCV, 2023. 5
2023
-
[61]
Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arxiv:2308.06721 ,
-
[62]
Magvit: Masked generative video transformer
Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos ´e Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. InCVPR, pages 10459–10469,
-
[63]
Adding conditional control to text-to-image diffusion models
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023. 3
2023
-
[64]
Mim- icmotion: High-quality human motion video generation with confidence-aware pose guidance
Yuang Zhang, Jiaxi Gu, Li-Wen Wang, Han Wang, Junqi Cheng, Yuefeng Zhu, and Fangyuan Zou. Mim- icmotion: High-quality human motion video generation with confidence-aware pose guidance. arXiv preprint arXiv:2406.19680, 2024. 1, 2, 3, 4, 6, 8
2024 arXiv
-
[65]
Chan, Chongyi Li, and Chen Change Loy
Shangchen Zhou, Kelvin C.K. Chan, Chongyi Li, and Chen Change Loy. Towards robust blind face restoration with codebook lookup transformer. In NeurIPS, 2022. 1
2022
-
[66]
Champ: Controllable and consistent human image animation with 3d parametric guidance
Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu. Champ: Controllable and consistent human image animation with 3d parametric guidance. In EECV, 2024. 1, 3, 5, 6, 8 11 A. Supplementary Material A.1. Evaluation Metrics Following pr...
2024
-
[67]
We can see that our proposed components can sig- nificantly facilitate the performance of different backbone- based models, particularly in the facial regions
ControlNeXt and MagicAnimate are based on Stable Video Diffusion (SVD) and Stable Diffusion (SD), respec- tively. We can see that our proposed components can sig- nificantly facilitate the performance of different backbone- based models, particularly in the facial regions. Not...
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.