Pith. sign in

REVIEW 3 major objections 6 minor 63 references

DreamDance: Animating Character Art via Inpainting Stable Gaussian Worlds

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Animating a character illustration with large, coherent camera movement becomes tractable by first rebuilding its scene as a 3D Gaussian field and then inpainting a pose-driven character into the rendered background video.

desk verdict DreamDance is a coherent incremental advance over MikuDance that deserves referee time, but its camera-control guarantee rests on an unvalidated training-inference proxy. read the letter →

arxiv 2505.24733 v1 pith:EPNU2SLN submitted 2025-05-30 cs.CV

Jiaxu Zhang , Xianfang Zeng , Xin Chen , Wei Zuo , Gang Yu , Guosheng Lin , Zhigang Tu This is my paper
classification cs.CV
keywords characterartanimationvideoinpainting3DGaussiansplattingdiffusiontransformerpose-guidedcameratrajectorycontrolgatedconditioningscene
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that animating a character illustration with large, coherent camera movement becomes tractable if the job is split into two inpainting passes instead of generating character and scene together. In the first pass, a camera-aware scene-inpainting step fills in the single reference image from many viewpoints and rebuilds the background as a 3D Gaussian field, so any requested camera path can render a rough but geometrically consistent background video. In the second pass, a pose-aware video-inpainting model, built on a transformer-based video diffusion model with time-step-dependent gates, injects the character's appearance and the driving pose into that coarse video while also cleaning up its artifacts. The paper reports that this two-pass design outperforms prior image-animation and character-replacement methods on image and video quality metrics and in a user study. If the claim is right, one flat illustration can become a full animated shot with precise camera dynamics and no manual 3D modeling.

What carries the argument

The load-bearing object is the gated pose-aware video inpainting model, a diffusion model whose transformer backbone mixes several latent streams (the technical meaning of the paper's MM-DiT label). At each denoising step $t$ it forms $x'_t = f_b([\beta, x_t]) + \tanh(g_p(t))\cdot f_p(\gamma) + \tanh(g_c(t))\cdot f_c(\phi)$, where $\beta$ is the latent of the coarse background video plus inpainting mask, $\gamma$ is the pose latent, $\phi$ is the character-reference latent, $f_b/f_p/f_c$ are convolutional project-in layers, and the gates $\tanh(g_p(t))$ and $\tanh(g_c(t))$ fall as $t$ increases. This arrangement makes the character's identity and pose the dominant input early in denoising and lets later steps concentrate on harmonizing and sharpening the video, which is how one model both inserts the animated character and repairs the rough Gaussian-rendered background. The scene side of the machinery is the 3D Gaussian field produced by warp-and-inpaint along a spiral trajectory and extended along the custom camera trajectory; splatting that field yields the coarse but spatially consistent background video that carries the camera motion.

What would settle it

Render a held-out set of coarse background videos from reconstructed Gaussian fields, measure the distribution of artifacts such as black-void sizes, blur, and perspective distortion, and train the same gated video inpainting model on a mix of real renders and synthetic degradations; if using real renders materially improves refinement quality on unseen renders compared with the augmentation-only recipe, the proxy assumption is the bottleneck, while equal performance would confirm the degradations are faithful.

Watch

Extended reading notes

Core claim

The paper's central claim is that the visual instability of character art animation under camera motion is an inpainting problem rather than a rendering problem, and it can be solved by chaining two inpainting stages. Camera-aware Scene Inpainting reconstructs a stable large-scale 3D Gaussian field from the reference art: it removes the character, describes and completes the background, warps the completed background along a spiral camera trajectory with depth-guided inpainting to produce multi-view RGB-D images, trains the Gaussian field from them, and then extends the field along the user's custom trajectory before splatting a coarse background video. Pose-aware Video Inpainting then trains a gated video inpainting model on top of a multimodal diffusion transformer; the gate reweights character-appearance and pose latents by denoising step so that identity and motion are injected early while later steps focus on refining the coarse background. Because the video-inpainting model is trained without 3D reconstruction, using real videos degraded with black blocks, noise, blur, and random perspective transformations as proxies for Gaussian renders, the pipeline avoids the cost of reconstructing fields for every training sample. The paper reports that this yields coherent scene dynamics and character fidelity that prior 3D-agnostic, UNet-based methods do not achieve.

Load-bearing premise

The whole pipeline's reliability rests on a single untested equivalence: that real videos deliberately corrupted with black blocks, noise, blur, and random perspective warps look enough like the coarse videos rendered from reconstructed 3D Gaussian scenes that a model trained on the fake versions will clean up the real ones at inference.

Editorial extensions

If this is right

  • One character illustration can be turned into a video with large, coherent camera movement, including zooming and scene rotation, without manual scene modeling or rigging.
  • The pipeline also acts as a character-replacement tool: a reference character can be placed into an existing driving video while preserving the scene.
  • Because the video-inpainting stage is trained without 3D reconstruction, it can be retrained or upgraded on more video data and stronger base video models without rebuilding the scene stage.
  • Non-experts could animate stylized art in diverse visual styles from just one reference image and one driving video, since no professional animation software is required.
  • On the paper's benchmark, the full system reports the best scores among the compared image-animation and character-replacement methods on most image and video quality metrics, and the ablation shows the gating strategy and scene reconstruction each contribute independent gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The step-dependent gating principle plausibly generalizes beyond character art: any diffusion model that must preserve identity while harmonizing a scene could adopt early-injection, late-relaxation gates; the paper only demonstrates this for characters.
  • If the degraded-real-video proxy holds, the video-inpainting stage is renderer-agnostic and could refine coarse output from any scene representation, not only Gaussian fields; that transfer is left unstated in the paper.
  • A natural stress test the paper does not run is to push the requested camera trajectory far outside the spiral-reconstruction coverage and measure when background or character consistency breaks; the paper shows high-dynamic examples but does not quantify this boundary.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes DreamDance, a two-stage framework for animating character art with camera control. Stage 1, Camera-aware Scene Inpainting, reconstructs a 3D Gaussian scene from the reference image via multi-view inpainting and renders a coarse background video along a user-defined or estimated custom camera trajectory. Stage 2, Pose-aware Video Inpainting, trains a gated MM-DiT video inpainting model (based on CogVideoX) to inject the reference character and driving poses into the coarse background while refining its visual quality. The gating strategy weights character and pose conditioning as a function of denoising step. The paper reports qualitative comparisons against image animation and character replacement baselines, ablations, a user study, and quantitative metrics (FID, SSIM, PSNR, LPIPS, FVD) on a private 100-video split, claiming state-of-the-art performance.

Significance. If the results hold, the two-step inpainting decomposition is a sensible advance over 3D-agnostic animation methods, and the gated MM-DiT inpainting design is a plausible mechanism for injecting dynamic character content into a coarse scene video without losing camera motion. The ablation study does provide support for the roles of the 3D Gaussian scene, the DiT backbone, and the gates, and the gate-value visualization is a useful sanity check. The paper also includes comparisons with multiple strong baselines and a user study, which is commendable. However, the central training-inference mismatch—the unvalidated 3D Gaussian-free training proxy—and the absence of error bars on a private test set temper confidence in the central claims. The lack of released code/data limits reproducibility, although the authors state code will be released in the final version.

major comments (3)
  1. [§3.3 (3D Gaussian-free Training Approach)] The central training-inference mismatch is unvalidated: the pose-aware video inpainting model is trained exclusively on real videos degraded with black blocks, noise, blur, and random perspective transformations, but at inference it must refine coarse videos rendered from the reconstructed 3D Gaussian field. The paper itself concedes in §3.2 that these renders 'may contain blurring, distortions, and black voids,' and such artifacts are spatially structured by the scene geometry and temporally coherent with the camera path, unlike the i.i.d. augmentations used during training. No experiment in the paper measures the distribution gap between augmented real videos and actual 3DGS renders, and no metric isolates whether the final background preserves the input render's camera trajectory (e.g., via optical-flow alignment or point tracking). If the proxy is not faithful, the model may overwrite the render's parallax with its own prior, and the claimed precise camera control would fail, reducing DreamDance to a 3D-agnostic inpaint-and-generate baseline. Please add a validation experiment: feed real 3DGS-rendered coarse videos into the trained model and compare against a variant fine-tuned on a small set of real renders, or quantify the artifact distribution of the renders and show it is covered by the augmentation.
  2. [§4.2, Table 1] The quantitative evaluation is conducted on a private 100-video split with no error bars, confidence intervals, or significance tests. Several reported differences are modest (e.g., FID 29.659 vs 30.794 for DreamDance vs w/o G; SSIM 0.699 vs 0.626), and without variance information it is not possible to tell whether the improvements are reliable. Because the test set is drawn from the same MMD distribution as the training set (all from the collected MMD dataset), the claim of 'generalizability' is also under-supported. Please report standard deviations or bootstrap intervals and, if possible, evaluate on a public benchmark or release the test split.
  3. [§4.2, User study] The user study as reported lacks statistical rigor: the text states that 'abnormal responses' were filtered but does not describe the filtering criterion, and the average ranks in Figure 11 are presented without error bars or significance tests. The claims that 'over 77.27% of users' preferred DreamDance for image animation and 'more than 59.09%' for character replacement need a statistical analysis (e.g., Wilcoxon signed-rank test) to be meaningful. Please add this information or temper the claims.
minor comments (6)
  1. [Table 1 and §4.2] The reported metric 'LISPIS' should be 'LPIPS' (referencing Zhang et al., 2018).
  2. [§4, Evaluation metrics] The phrase 'PSNR [12], and and L1' contains a duplicated 'and'.
  3. [§3.2, Eq. (5)] In the spiral trajectory definition, the z-component of P(t) is '-sin(2πt)' without the radius r, which appears inconsistent with the x- and y-components; please check whether this is intended.
  4. [§4, Experiments] The in-text citation 'Following MikuDance [10]' points to reference [10], which is the MMD software website, rather than to the MikuDance paper (reference [51]); please correct the citation.
  5. [§3.1, Heading] The heading '3D Gaussain Splatting' contains a typo; it should be '3D Gaussian Splatting'.
  6. [§3, Method] In the description of the first stage, 'a coarse background video is rendered by splitting the 3D Gaussian field' should read 'splatting' rather than 'splitting'.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity: central claims rest on external baselines and ablations; one mild self-confirmation in the gate-value analysis and an in-group MikuDance comparison noted.

  1. other [Section 3.3 (gating strategy) and Section 4.1 / Figure 10 (ablation and gate visualization)]
    "To achieve this, we propose two denoising step-based gates that adaptively inject the latent character and pose into the base latent noise according to the denoising step t. Each gate consists of a Linear layer followed by a tanh activation function. ... As the denoising steps progress from 0 to 50, both the character gate and the pose gate decrease. This supports our conjecture that the model requires more information about the character's appearance and pose during the early denoising steps, whereas in the later stages, it prioritizes optimizing the quality of the existing latent videos."

    The gate-value plot is presented as evidence for the design conjecture that character/pose information should be prioritized early in denoising, but the gates are parameterized as tanh of a linear function of the denoising step t, so gate activity is an explicit, architected function of t. The observed decay is a readout of the trained weights of the same time-dependent injection the design already imposes, not an independent falsifiable test of the 'requires more information early' claim; the visualization re-states the construction rather than validating it.

full rationale

DreamDance's derivation chain is largely self-contained. Stage one (Camera-aware Scene Inpainting) combines external components (LLaVA description, Fooocus inpainting, DepthPro depth, VistaDream-style warp-and-inpaint, 3DGS optimization); stage two fine-tunes CogVideoX-5B with a gated MM-DiT architecture and its own '3D Gaussian-free' supervised training loss. The central superiority claim is measured against external baselines (AniAny, UniAnimate, MimicMotion, DisCo, SVD, MIMO, MotionShop-2) on 100 held-out MMD videos plus a 50-person user study, and the ablations are internal controls, not fitted inputs renamed as predictions. No equation reduces to its inputs: loss (2), gates (6), and splatting (4) are standard constructions. The weakest internal assumption — that degrading real videos with black blocks, noise, blur, and random perspective transforms (Sec. 3.3) faithfully simulates coarse 3DGS renders, which the paper itself admits 'may contain blurring, distortions, and black voids' (Sec. 3.2) — is a training-inference distribution gap and a genuinely unvalidated proxy, but it is not circular: the training input is not identical to the test input by construction, so refinement success is an empirical bet, not a tautology. Missing support is flagged: Table 1 and the user study never isolate whether the final background preserves the rendered camera trajectory, and no metric measures the proxy-vs-render gap. Self-citation: MikuDance [51] shares six of seven authors with this paper, is the design ancestor ('Derived from MikuDance'), and the evaluation protocol is attributed as 'Following MikuDance [10]', where reference [10] resolves to the MikuMikuDance software page rather than the MikuDance paper — a citation gap. Per hard rule 4, this alone is not circular, because the comparison rests on measured numbers and on independent external baselines. The one mild self-confirmation found is the gate-value visualization of Figure 10 (see step). Overall score 2: no load-bearing circularity, with a minor in-group comparison and a minor by-construction confirmation.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The framework imports many pretrained models and heuristics, and none of these are validated on this target domain. The most consequential assumption is that random video degradation simulates the coarse Gaussian renders. No free parameters are fit to the central claim in a circular way; only the spiral radius and training hyperparameters are hand-chosen.

free parameters (1)
  • Spiral trajectory radius r = not specified
    Introduced in Eq. (5) as the radius of the spiral camera path; hand-chosen and controls which parts of the scene are observed and reconstructed.
assumptions (5)
  • domain assumption Pretrained Fooocus, DepthPro, DPVO, Xpose, BiRefNet, CogVideoX, CLIP, and SDXL-Neta perform as expected on stylized character art.
    Every pipeline stage delegates key perception tasks to external models without verifying failure rates on MMD-style art.
  • domain assumption Warp-and-inpaint along a spiral trajectory yields multi-view consistent RGBD images sufficient to train a stable 3D Gaussian field.
    Stated in Section 3.2; the spiral formula in Eq. (5) is assumed to cover missing regions without contradicting geometry assignments.
  • domain assumption Random degradation of real videos approximates the coarse background rendering artifacts of the reconstructed Gaussian field.
    Section 3.3; the training-inference domain gap is assumed to be small.
  • domain assumption The 100 MMD test videos are representative and disjoint from training, and image and video metrics reflect animation quality.
    Section 4; no overlap analysis or metric validation is reported.
  • domain assumption DPVO camera poses from the driving videos are accurate enough to define the custom trajectory after standardization.
    Section 3; the limitations admit manual adjustments are often needed, which complicates evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DreamDance: Animating Character Art via Inpainting Stable Gaussian Worlds." pith.science (2026). https://pith.science/paper/EPNU2SLN

@misc{pith2026250524733,
  author       = {Pith},
  title        = {Pith review of: DreamDance: Animating Character Art via Inpainting Stable Gaussian Worlds},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EPNU2SLN}},
  note         = {Machine review of arXiv:2505.24733}
}
read the original abstract

This paper presents DreamDance, a novel character art animation framework capable of producing stable, consistent character and scene motion conditioned on precise camera trajectories. To achieve this, we re-formulate the animation task as two inpainting-based steps: Camera-aware Scene Inpainting and Pose-aware Video Inpainting. The first step leverages a pre-trained image inpainting model to generate multi-view scene images from the reference art and optimizes a stable large-scale Gaussian field, which enables coarse background video rendering with camera trajectories. However, the rendered video is rough and only conveys scene motion. To resolve this, the second step trains a pose-aware video inpainting model that injects the dynamic character into the scene video while enhancing background quality. Specifically, this model is a DiT-based video generation model with a gating strategy that adaptively integrates the character's appearance and pose information into the base background video. Through extensive experiments, we demonstrate the effectiveness and generalizability of DreamDance, producing high-quality and consistent character animations with remarkable camera dynamics.

Figures

Figures reproduced from arXiv: 2505.24733 by the authors.

Figure 1
Figure 1. We propose DreamDance, a novel paradigm that re￾formulates the character art animation task into two inpainting￾based steps: Camera-aware Scene Inpainting for stable scene re￾construction and Pose-aware Video Inpainting for dynamic char￾acter animation. image sequences to drive characters and 2D scene flow to guide backgrounds. Similar to MikuDance, existing meth￾ods incorporate other motion guidance, such as optica… view at source ↗
Figure 2
Figure 2. Illustration of our DreamDance. The reference character art is decomposed into foreground and background layers. The background image is used to reconstruct a stable 3D Gaussian scene through a wrap-and-inpaint scheme, enabling coarse background video rendering based on custom camera trajectories. The gated MM-DiT model then inpaints the background video based on the foreground character and the driving poses, gener… view at source ↗
Figure 4
Figure 4. The gating strategy in our MM-DiT model and its com￾parison with the mainstream condition incorporation methods. are standardized based on the first camera frame to ensure consistency with the spiral trajectory. Finally, based on the custom camera trajectory, a background video is rendered through volume splatting at each camera step. The reason we do not directly use the custom camera tra￾jectory to reconstruct the… view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Comparison with image animation baselines. The red arrows represent the approximate direction of camera movement, while the circles highlight significant correspondences. The black boxes contain the reconstructed 3D Gaussian scenes of our DreamDance. Reference Driving …
Figure 6
Figure 6. Figure 6: Comparison with character replacement baselines. MIMO and MotionShop-2 only support full-body reference images. Comparison with character replacement baselines. One valuable application of DreamDance is its ability to directly replace humans in driving videos with refe…
Figure 7
Figure 7. Figure 7: High-dynamic and precise camera control of our DreamDance. MikuDance exhibits inconsistencies due to its 3D-agnostic motion guidance, while AniAny produces static backgrounds. In contrast, DreamDance generates coherent and vivid animations. DreamDance w/o 3DGS w/ Unet …
Figure 8
Figure 8. Figure 8: Ablation experiments. “w/o 3DGS”, “w/ UNet”, and “w/o G” are defined in Section 4.1 [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Generalizability on various scenes (left) and characters (right). Please see our demo video for a clearer understanding. MM-DiT model, bypassing the 3D Gaussian reconstruction process. The results in [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Ablations on spiral trajectory (left), Scene enhance￾ment (middle), and visualization of the gate values (right). loid, antiquity, and line sketch, as demonstrated in the right part of [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 28 canonical work pages

  1. [1]

    Conditional gan with discrimi- native filter generation for text-to-video synthesis

    Yogesh Balaji, Martin Renqiang Min, Bing Bai, Rama Chel- lappa, and Hans Peter Graf. Conditional gan with discrimi- native filter generation for text-to-video synthesis. In IJCAI, page 2, 2019. 5

    work page 2019

  2. [2]

    Person image synthesis via de- noising diffusion model

    Ankan Kumar Bhunia, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Jorma Laaksonen, Mubarak Shah, and Fahad Shahbaz Khan. Person image synthesis via de- noising diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5968–5976, 2023. 1, 2

    work page 2023

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 1, 2

    arXiv 2023

  4. [4]

    Depth pro: Sharp monocular metric depth in less than a second

    Aleksei Bochkovskii, Ama ¨el Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than a second. arXiv preprint arXiv:2410.02073, 2024. 4

    arXiv 2024

  5. [5]

    X-dyna: Expressive dynamic hu- man image animation

    Di Chang, Hongyi Xu, You Xie, Yipeng Gao, Zhengfei Kuang, Shengqu Cai, Chenxu Zhang, Guoxian Song, Chao Wang, Yichun Shi, et al. X-dyna: Expressive dynamic hu- man image animation. arXiv preprint arXiv:2501.10021 ,

    arXiv

  6. [6]

    Luciddreamer: Domain-free gen- eration of 3d gaussian splatting scenes

    Jaeyoung Chung, Suyoung Lee, Hyeongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Luciddreamer: Domain-free gen- eration of 3d gaussian splatting scenes. arXiv preprint arXiv:2311.13384, 2023. 2, 3

    arXiv 2023

  7. [7]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. In The Twelfth International Conference on Learning Representa- tions, 2024. 2

    work page 2024

  8. [8]

    Textoon: Generat- ing vivid 2d cartoon characters from text descriptions

    Chao He, Jianqiang Ren, and Liefeng Bo. Textoon: Generat- ing vivid 2d cartoon characters from text descriptions. arXiv preprint arXiv:2501.10020, 2025. 2

    arXiv 2025

Show all 63 references
  1. [9]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 5

    2017

  2. [10]

    Mikumikudance

    Yu Higuchi. Mikumikudance. https : / / sites . google.com/view/evpvp. 1, 4, 5

  3. [11]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  4. [12]

    Image quality metrics: Psnr vs

    Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pages 2366–2369. IEEE, 2010. 5

    2010

  5. [13]

    Animate anyone: Consistent and controllable image- to-video synthesis for character animation

    Li Hu. Animate anyone: Consistent and controllable image- to-video synthesis for character animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8153–8163, 2024. 1, 2, 5, 8

    2024

  6. [14]

    Animate anyone 2: High-fidelity character image ani- mation with environment affordance, 2025

    Li Hu, Guangyuan Wang, Zhen Shen, Xin Gao, Dechao Meng, Lian Zhuo, Peng Zhang, Bang Zhang, and Liefeng Bo. Animate anyone 2: High-fidelity character image ani- mation with environment affordance, 2025. 2

    2025

  7. [15]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139–1,

  8. [16]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. 4

    2024 arXiv

  9. [17]

    neta-art-xl-1.0

    Neta.art Lab. neta-art-xl-1.0. https://huggingface. co/neta-art/neta-art-xl-1.0 . 5

  10. [18]

    Towards an end-to-end framework for flow-guided video inpainting

    Zhen Li, Cheng-Ze Lu, Jianhua Qin, Chun-Le Guo, and Ming-Ming Cheng. Towards an end-to-end framework for flow-guided video inpainting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17562–17571, 2022. 2

    2022

  11. [19]

    Wonderland: Nav- igating 3d scenes from a single image

    Hanwen Liang, Junli Cao, Vidit Goel, Guocheng Qian, Sergei Korolev, Demetri Terzopoulos, Konstantinos N Pla- taniotis, Sergey Tulyakov, and Jian Ren. Wonderland: Nav- igating 3d scenes from a single image. arXiv preprint arXiv:2412.12091, 2024. 3

    2024 arXiv

  12. [20]

    Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models

    Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, and Chao Liang. Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models. arXiv preprint arXiv:2502.01061, 2025. 2, 4

    2025 arXiv

  13. [21]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 4

    2024

  14. [22]

    Mimo: Controllable character video synthesis with spatial decomposed modeling

    Yifang Men, Yuan Yao, Miaomiao Cui, and Liefeng Bo. Mimo: Controllable character video synthesis with spatial decomposed modeling. arXiv preprint arXiv:2409.16160 ,

    arXiv

  15. [23]

    Tetsuya Nakajo. Live2d. https://www.live2d.com/. 1

  16. [24]

    Mofa-video: Controllable image animation via generative motion field adaptions in frozen image-to-video diffusion model

    Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. Mofa-video: Controllable image animation via generative motion field adaptions in frozen image-to-video diffusion model. In European Con- ference on Computer Vision, pages 111–128. Springer, 2024. 1

    2024

  17. [25]

    Finetrainers

    Sayak Paul. Finetrainers. https://github.com/a- r-r-o-w/finetrainers . 5

  18. [26]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  19. [27]

    Anigs: Animatable gaussian avatar from a single image with inconsistent gaussian reconstruction

    Lingteng Qiu, Shenhao Zhu, Qi Zuo, Xiaodong Gu, Yuan Dong, Junfei Zhang, Chao Xu, Zhe Li, Weihao Yuan, Liefeng Bo, et al. Anigs: Animatable gaussian avatar from a single image with inconsistent gaussian reconstruction. arXiv preprint arXiv:2412.02684, 2024. 2

    2024 arXiv

  20. [28]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, 9 Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning...

    2021

  21. [29]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 2, 3

    2022

  22. [30]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  23. [31]

    Human4dit: Free-view human video generation with 4d diffusion transformer

    Ruizhi Shao, Youxin Pang, Zerong Zheng, Jingxiang Sun, and Yebin Liu. Human4dit: Free-view human video generation with 4d diffusion transformer. arXiv preprint arXiv:2405.17405, 2024. 1, 2

    2024 arXiv

  24. [32]

    Realmdreamer: Text-driven 3d scene gener- ation with inpainting and depth diffusion

    Jaidev Shriram, Alex Trevithick, Lingjie Liu, and Ravi Ra- mamoorthi. Realmdreamer: Text-driven 3d scene gener- ation with inpainting and depth diffusion. arXiv preprint arXiv:2404.07199, 2024. 4

    2024 arXiv

  25. [33]

    A method for animating chil- dren’s drawings of the human figure

    Harrison Jesse Smith, Qingyuan Zheng, Yifei Li, Somya Jain, and Jessica K Hodgins. A method for animating chil- dren’s drawings of the human figure. ACM Transactions on Graphics, 42(3):1–15, 2023. 2

    2023

  26. [34]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265. PMLR, 2015. 3

    2015

  27. [35]

    Animate-x: Universal character image ani- mation with enhanced motion representation

    Shuai Tan, Biao Gong, Xiang Wang, Shiwei Zhang, Dandan Zheng, Ruobing Zheng, Kecheng Zheng, Jingdong Chen, and Ming Yang. Animate-x: Universal character image ani- mation with enhanced motion representation. arXiv preprint arXiv:2410.10306, 2024. 2

    arXiv

  28. [36]

    Deep patch vi- sual odometry

    Zachary Teed, Lahav Lipson, and Jia Deng. Deep patch vi- sual odometry. Advances in Neural Information Processing Systems, 36, 2024. 3

    2024

  29. [37]

    Edge: Editable dance generation from music

    Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 448–458, 2023. 5

    2023

  30. [38]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 5

    2018 arXiv

  31. [39]

    Vistadream: Sampling multi- view consistent images for single-view scene reconstruction

    Haiping Wang, Yuan Liu, Ziwei Liu, Wenping Wang, Zhen Dong, and Bisheng Yang. Vistadream: Sampling multi- view consistent images for single-view scene reconstruction. arXiv preprint arXiv:2410.16892, 2024. 2, 3, 4, 5

    2024 arXiv

  32. [40]

    Disco: Disentangled control for realistic human dance generation

    Tan Wang, Linjie Li, Kevin Lin, Yuanhao Zhai, Chung- Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for realistic human dance generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,...

    2024

  33. [41]

    Unianimate: Taming unified video diffusion mod- els for consistent human image animation

    Xiang Wang, Shiwei Zhang, Changxin Gao, Jiayu Wang, Xiaoqiang Zhou, Yingya Zhang, Luxin Yan, and Nong Sang. Unianimate: Taming unified video diffusion mod- els for consistent human image animation. arXiv preprint arXiv:2406.01188, 2024. 5, 8

    2024 arXiv

  34. [42]

    Leo: Generative latent image animator for human video synthesis

    Yaohui Wang, Xin Ma, Xinyuan Chen, Cunjian Chen, An- titza Dantcheva, Bo Dai, and Yu Qiao. Leo: Generative latent image animator for human video synthesis. Interna- tional Journal of Computer Vision, pages 1–13, 2024. 1

    2024

  35. [43]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si- moncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 5

    2004

  36. [44]

    Humanvid: Demystifying training data for camera-controllable human image animation

    Zhenzhi Wang, Yixuan Li, Yanhong Zeng, Youqing Fang, Yuwei Guo, Wenran Liu, Jing Tan, Kai Chen, Tianfan Xue, Bo Dai, et al. Humanvid: Demystifying training data for camera-controllable human image animation. arXiv preprint arXiv:2407.17438, 2024. 1, 2

    2024 arXiv

  37. [45]

    Ani- clipart: Clipart animation with text-to-video priors

    Ronghuan Wu, Wanchao Su, Kede Ma, and Jing Liao. Ani- clipart: Clipart animation with text-to-video priors. Interna- tional Journal of Computer Vision, pages 1–17, 2024. 2

    2024

  38. [46]

    Xpose: Detecting any keypoints

    Jie Yang, Ailing Zeng, Ruimao Zhang, and Lei Zhang. Xpose: Detecting any keypoints. arXiv preprint arXiv:2310.08530, 2023. 3

    2023 arXiv

  39. [47]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 2, 3, 5

    2024 arXiv

  40. [48]

    Wonderworld: Interactive 3d scene generation from a single image

    Hong-Xing Yu, Haoyi Duan, Charles Herrmann, William T Freeman, and Jiajun Wu. Wonderworld: Interactive 3d scene generation from a single image. arXiv preprint arXiv:2406.09394, 2024. 2, 3

    2024 arXiv

  41. [49]

    Motionshop-

    Junfei Zhang, Xiaodan Ye, Chao Xu, Feng Wang, Qing Ran, Kejie Qiu, Guangyuan Wang, Jianfeng Luo, Junyao Wu, Gang Cheng, Zilong Dong, and Liefeng Bo. Motionshop-

  42. [50]

    https://aigc3d.github.io/motionshop/. 6

  43. [51]

    Tapmo: Shape- aware motion generation of skeleton-free characters

    Jiaxu Zhang, Shaoli Huang, Zhigang Tu, Xin Chen, Xiao- hang Zhan, YU Gang, and Ying Shan. Tapmo: Shape- aware motion generation of skeleton-free characters. In The Twelfth International Conference on Learning Representa- tions, 2024. 2

    2024

  44. [52]

    Mikudance: Animating character art with mixed motion dynamics

    Jiaxu Zhang, Xianfang Zeng, Xin Chen, Wei Zuo, Gang Yu, and Zhigang Tu. Mikudance: Animating character art with mixed motion dynamics. arXiv preprint arXiv:2411.08656,

    arXiv

  45. [53]

    Flow-guided transformer for video inpainting

    Kaidong Zhang, Jingjing Fu, and Dong Liu. Flow-guided transformer for video inpainting. In European Conference on Computer Vision, pages 74–90. Springer, 2022. 2

    2022

  46. [54]

    Lvming Zhang. Fooocus. https://github.com/ lllyasviel/Fooocus. 2, 4

  47. [55]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 5 10

    2018

  48. [56]

    Mimicmo- tion: High-quality human motion video generation with confidence-aware pose guidance, 2024

    Yuang Zhang, Jiaxi Gu, Li-Wen Wang, Han Wang, Junqi Cheng, Yuefeng Zhu, and Fangyuan Zou. Mimicmo- tion: High-quality human motion video generation with confidence-aware pose guidance, 2024. 1, 2, 5, 8

    2024

  49. [57]

    Avid: Any-length video inpainting with dif- fusion model

    Zhixing Zhang, Bichen Wu, Xiaoyan Wang, Yaqiao Luo, Luxin Zhang, Yinan Zhao, Peter Vajda, Dimitris Metaxas, and Licheng Yu. Avid: Any-length video inpainting with dif- fusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 71...

    2024

  50. [58]

    Bilateral refer- ence for high-resolution dichotomous image segmentation

    Peng Zheng, Dehong Gao, Deng-Ping Fan, Li Liu, Jorma Laaksonen, Wanli Ouyang, and Nicu Sebe. Bilateral refer- ence for high-resolution dichotomous image segmentation. arXiv preprint arXiv:2401.03407, 2024. 3

    2024 arXiv

  51. [59]

    Drawingspinup: 3d animation from single character draw- ings

    Jie Zhou, Chufeng Xiao, Miu-Ling Lam, and Hongbo Fu. Drawingspinup: 3d animation from single character draw- ings. In SIGGRAPH Asia 2024 Conference Papers , pages 1–10, 2024. 2

    2024

  52. [60]

    Champ: Controllable and consistent human image animation with 3d parametric guidance

    Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu. Champ: Controllable and consistent human image animation with 3d parametric guidance. arXiv preprint arXiv:2403.14781 ,

    arXiv

  53. [61]

    Cococo: Improving text-guided video inpainting for better consistency, controllability and compatibility.arXiv preprint arXiv:2403.12035, 2024

    Bojia Zi, Shihao Zhao, Xianbiao Qi, Jianan Wang, Yukai Shi, Qianyu Chen, Bin Liang, Kam-Fai Wong, and Lei Zhang. Cococo: Improving text-guided video inpainting for better consistency, controllability and compatibility.arXiv preprint arXiv:2403.12035, 2024. 3

    2024 arXiv

  54. [62]

    Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers

    Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2024

  55. [63]

    Ewa volume splatting

    Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa volume splatting. In Proceedings Visu- alization, 2001. VIS’01., pages 29–538. IEEE, 2001. 3 11

    2001

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.