Pith. sign in

REVIEW 4 major objections 5 minor 54 references

RAIN: Real-time Animation of Infinite Video Stream

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RAIN claims a diffusion pipeline can animate infinite video streams in real time on a single RTX 4090 by grouping frames into shared-noise-level batches.

desk verdict A plausible StreamDiffusion variant whose own numbers undercut its headline claims; the 'infinite stable' framing is not supported even by the paper's admitted error accumulation. read the letter →

arxiv 2412.19489 v1 pith:S6NBD3UV submitted 2024-12-27 cs.CV

Zhilei Shu , Ruili Feng , Yang Cao , Zheng-Jun Zha This is my paper · ORCID
classification cs.CV
keywords real-timeanimationvideodiffusionstreaminggenerationStablefine-tuningLCMaccelerationtemporalattentioncharacterlow-latencyinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RAIN tries to establish that a diffusion-based character animation pipeline can process an infinite video stream in real time on a single consumer GPU, with continuity close to offline methods. The key claim is that expanding the StreamBatch by a factor of $p=4$, putting every four consecutive frames into a group that shares one noise level, and letting temporal attention operate across those groups, lets a fine-tuned Stable Diffusion model animate at 18 FPS on an RTX 4090. If true, live animation, virtual avatars, and streaming style transfer become practical on hardware that creators already own. The paper reports quality close to the offline AnimateAnyone baseline while producing frames with much lower latency.

What carries the argument

The central object is the grouped StreamBatch: $K=16$ frame tokens divided into $p=4$ groups, each group sharing one noise level, with noise levels spaced 250 timesteps apart. Under 4-step LCM sampling, this yields $K/p=4$ clean frames per cycle, which are popped out while four pure-noise frames are pushed in. The temporal attention module is fine-tuned to be 'temporal adaptive', meaning it learns to relate frames at different noise levels; removing the causal mask and training over the entire batch is what makes long-range continuity possible. This machinery lets one denoising pass do the work of a much larger temporal context than the four denoising steps alone would suggest.

What would settle it

Run the released model on a long fixed-camera stream, e.g., 10,000 frames, and plot a temporal-continuity metric such as FVD or identity-similarity over sliding windows as stream length grows. If visible artifacts, like the abnormal color blocks shown in Section 6.1, appear within that period without the character leaving the camera, the 'infinite, stable' claim is bounded in practice at that duration. Quantifying how often and how fast errors accumulate would settle the central claim.

Watch

Extended reading notes

Core claim

The paper claims that the bottleneck in stream-based video diffusion is not denoising capacity but the coupling between StreamBatch size and the number of sampling steps. RAIN decouples them by choosing $K=16$ frames and $p=4$ groups, so each group's four frames share a noise level ($t_0$, $t_0+250$, $t_0+500$, $t_0+750$), and by allowing the 1D temporal attention module to attend across noise levels and long time intervals. After a few epochs of fine-tuning the motion module with this grouped noise schedule, and after consistency distillation following the 3D inflation strategy, the model sustains 18 FPS on a single RTX 4090. On UBC-Fashion, SSIM is 0.921, LPIPS 0.063, and FVD 85.2, close to the offline method's 0.931, 0.044, and 81.6, respectively.

Load-bearing premise

The load-bearing premise is that the motion module, trained on fixed groups of four noise levels, remains stable when run in the sliding pop-and-push loop for arbitrarily long streams, even though the paper itself concedes in Section 6.1 that small errors accumulate and can require the character to leave and re-enter the camera to reset.

Editorial extensions

If this is right

  • Real-time character animation, including whole-body movement, face morphing, and style transfer, becomes feasible on a single RTX 4090 at 512x512 resolution.
  • Long-range temporal continuity can be achieved with only four denoising steps, contradicting the assumption that StreamBatch size must equal the number of steps.
  • The quality gap between streaming and offline animation narrows: RAIN reports SSIM 0.921 versus 0.931 and FVD 85.2 versus 81.6 against the offline baseline.
  • Because RAIN adds only 1D attention blocks and a few epochs of fine-tuning, existing Stable Diffusion adapters can likely be converted to streaming with modest cost.
  • For live-streaming tasks, RAIN reports 18.11 FPS, between StreamDiffusion at 37.13 and Live2Diff at 16.43, indicating a different point on the quality-speed trade-off.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the grouped-noise schedule is understood as parallel-in-time sampling, the error accumulation reported in Section 6.1 suggests that 'infinite' should be read as 'long but finite' unless an explicit error-bound analysis is supplied.
  • The cross-noise-level temporal attention result is a design choice that could transfer to other streaming diffusion tasks, such as live video editing or long-horizon video prediction, wherever the stream outlasts any fixed batch.
  • The reported 18 FPS bundles TensorRT acceleration and TinyVAE decoding; on other hardware or toolchains the achievable rate will differ, so comparisons should hold those components fixed.
  • A testable extension the paper does not run is a joint sweep of $p$ and $K$: its own ablation shows $K=8$ worsens FVD from 85.2 to 145.8, suggesting larger $K$ may buy further continuity at some FPS cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. RAIN proposes a streaming pipeline for real-time character animation with diffusion models on a single RTX 4090. It extends AnimateAnyone by dividing a K=16-frame temporal batch into p=4 groups with different noise levels (Eq. 7), enabling 4-step LCM sampling while retaining long-range temporal attention. Cross-noise-level attention is used across these groups. The paper reports 18.11 FPS on 512x512 video, with quantitative results on UBC-Fashion (Table 1), a qualitative face-morphing application, and a style-transfer task. It also includes an ablation on temporal batch size and causal masking (Table 3).

Significance. If the claims hold, RAIN would be a practically valuable system: it demonstrates that consumer-grade GPUs can animate characters in real time with a fine-tuned diffusion model, and the grouped StreamBatch idea (multiple frames per noise level) is an interesting way to increase GPU utilization beyond the number of denoising steps. The inclusion of specific FPS numbers, the causal-mask ablation, and the acknowledgment of limitations are constructive. However, the quantitative evidence in the paper directly undercuts the abstract's claim of "much better quality, accuracy, and consistency than competitors while costing less latency": RAIN is worse than AnimateAnyone on SSIM, LPIPS, and FVD, and slower than StreamDiffusion in FPS. The "infinite stable" claim is also contradicted by the paper's own Section 6.1, which concedes error accumulation. The core mechanism is plausible, but the paper's central claims need to be scaled back or supported by additional evidence.

major comments (4)
  1. [Section 5.1, Table 1] The quantitative quality comparison does not support the abstract's claim of "much better quality, accuracy, and consistency than competitors." Table 1 shows RAIN is worse than AnimateAnyone on SSIM (0.921 vs. 0.931), LPIPS (0.063 vs. 0.044), and FVD (85.2 vs. 81.6). The PSNR comparison is also invalid because the reported values use inconsistent overflow handling: AnimateAnyone is listed as 38.49 (overflow) while RAIN is 23.99 (non-overflow). The note that RAIN would be 37.20 under the overflow convention is still lower than 38.49, so the table's headline numbers are not directly comparable. This is a load-bearing evidence gap for the paper's central quality claim.
  2. [Section 5.3, Table 2] The abstract states RAIN costs "less latency" than competitors, but Table 2 reports only FPS, and RAIN's 18.11 FPS is substantially lower than StreamDiffusion's 37.13 FPS. FPS is not the same as latency; however, the paper provides no latency measurements at all. Thus the "less latency" claim is unsupported, and the throughput advantage over the primary stream-based competitor is absent. The paper should either provide actual latency comparisons or revise the claim to reflect that RAIN trades throughput for quality/continuity relative to StreamDiffusion.
  3. [Section 6.1] The "infinite, stable" claim in the abstract and introduction is contradicted by the paper's own limitations section. Section 6.1 states: "The streaming structure of RAIN causes the error to accumulated with time" and that abnormal color persists until the character exits and re-enters the camera. No error bound, fixed-point analysis, or long-horizon evaluation is provided; Figure 7 shows only a single 600-frame anomaly. A live stream does not guarantee the reset condition, so the claim of stable operation over unbounded streams is not established. The paper should either supply evidence of stability over much longer sequences (e.g., 10^4 or 10^5 frames) for typical inputs, or explicitly limit the claim to practical durations with occasional resets.
  4. [Algorithm 1] Algorithm 1 is titled "streaming video processing" and the caption claims it can handle "streaming video input with infinite length," but the algorithm as written is finite: it takes a video of fixed length L, loops w+a-1 times, and returns a processed video. There is no pop/push mechanism or infinite-loop structure, despite the text in Section 4.2 and Figure 3 describing such a mechanism. This makes the core algorithmic contribution under-specified and hampers reproducibility. The algorithm should be rewritten to explicitly show the streaming loop with popping clean frames and pushing new noisy frames.
minor comments (5)
  1. [Section 5] The sentence "The temporal adaptive arguments p, K are specified to 4" is ambiguous and appears to contain a typo: earlier in the paper K=16 and p=4, with K/p=4. Please clarify which parameters are set to 4.
  2. [Abstract and Introduction] The phrase "up to infinite long" should be corrected to "up to infinite length" or "for infinitely long streams."
  3. [Table 1] The footnote explaining the PSNR overflow issue is important and should be part of the table's main description, not a footnote, so that readers do not misinterpret the raw numbers.
  4. [Section 5.3 and Table 2] The paper compares only FPS with stream-based methods and provides no qualitative or quantitative quality comparison with StreamDiffusion, Live2Diff, or StreamV2V. Adding such a comparison would strengthen the claim that RAIN offers a favorable quality-speed trade-off.
  5. [Section 4.2] The text "If first p frames is already clean" has a subject-verb agreement error; it should read "If the first p frames are already clean."

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RAIN's claims are supported by held-out benchmark evaluation and ablations, not by a definitional or self-citational reduction.

full rationale

RAIN's derivation chain is empirical and self-contained against external tests. The training schedule in Eq. (7) assigns grouped noise levels to K=16 frame tokens; this is a design choice, not a fitted prediction. Inference (Algorithm 1) pops clean frames and pushes noise frames using the same grouped schedule, but the paper's headline numbers (Table 1: SSIM 0.921, FVD 85.2, LPIPS 0.063) are measured on the UBC-Fashion test set against external baselines such as AnimateAnyone, DreamPose, and BDMM. The causal-mask ablation (Table 3) is a genuine alternative architecture and does not reduce to the method by construction. No load-bearing step is justified solely by a self-citation; the cited components (AnimateAnyone, StreamDiffusion, AnimateLCM, AnimateDiff) are external priors. The one substantive weakness is stated in Section 6.1: 'The streaming structure of RAIN causes the error to accumulated with time' and 'Character needs to appear in and out of the camera so that the synthesized results can return to normal.' That is an evidence gap for the abstract's 'up to infinite long' and 'stable' wording, but it is a correctness/stability concern, not circularity. For those reasons the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No invented entities. The pipeline is assembled from existing components: AnimateAnyone reference mechanism, AnimateDiff motion module, LCM consistency distillation, StreamDiffusion batch denoising, and TinyVAE. The central claim depends on hand-chosen hyperparameters (p=4, K=16, t0 range, guidance strength) and on the unproven stability of the streaming pop/push loop; the paper itself documents error accumulation and static-scene restrictions.

free parameters (4)
  • p (group size) = 4
    Chosen so K/p = 4 matches the 4-step LCM sampling; the paper gives no search or sensitivity analysis beyond p=4 versus the K=8 variant.
  • K (temporal batch size) = 16
    Inherited from the AnimateDiff motion module and chosen as the batch that fits memory at 512x512. Only K=8 is tested as an ablation.
  • t0 (starting noise level) = Random in [1,250] during training; schedule T/Np per inference step
    The grouping schedule in Eq. 7 depends on t0; the paper states t0 is sampled evenly but does not analyze sensitivity.
  • guidance strength omega = 2.0 (UBC-Fashion), 2.5 (face morphing), 3.5 (style transfer)
    Picked per dataset to give the best results; no selection criterion or error bars are reported.
assumptions (4)
  • domain assumption LCM/consistency distillation in 4 steps preserves the video quality needed for the grouped equal-noise batch.
    The paper adopts AnimateLCM initialization and LCM distillation without analyzing whether the consistency property holds when frames in one batch have different noise levels; Section 4.3.
  • domain assumption Unmasked bidirectional temporal attention across all K frames is safe and beneficial for streaming.
    The causal-mask ablation in Table 3 shows masked attention hurts FVD a lot, but no mechanism explains why future frames can inform earlier frames in live streaming; Sections 4.1 and 5.4.
  • ad hoc to paper The soft-start streaming loop reaches a stable periodic state that can run indefinitely.
    Algorithm 1 assumes popping clean frames and pushing new noise frames every p*N steps works forever, yet Section 6.1 says errors accumulate and require a reset; no stability analysis is provided.
  • domain assumption Static or near-static scenes are sufficient for the claimed applications.
    Section 5.3 says the method does not apply to highly dynamic scenes, and Section 6.1 says occluded or new objects degrade; the 'infinite stream' claim is therefore restricted to fixed backgrounds.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAIN: Real-time Animation of Infinite Video Stream." pith.science (2026). https://pith.science/paper/S6NBD3UV

@misc{pith2026241219489,
  author       = {Pith},
  title        = {Pith review of: RAIN: Real-time Animation of Infinite Video Stream},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S6NBD3UV}},
  note         = {Machine review of arXiv:2412.19489}
}
read the original abstract

Live animation has gained immense popularity for enhancing online engagement, yet achieving high-quality, real-time, and stable animation with diffusion models remains challenging, especially on consumer-grade GPUs. Existing methods struggle with generating long, consistent video streams efficiently, often being limited by latency issues and degraded visual quality over extended periods. In this paper, we introduce RAIN, a pipeline solution capable of animating infinite video streams in real-time with low latency using a single RTX 4090 GPU. The core idea of RAIN is to efficiently compute frame-token attention across different noise levels and long time-intervals while simultaneously denoising a significantly larger number of frame-tokens than previous stream-based methods. This design allows RAIN to generate video frames with much shorter latency and faster speed, while maintaining long-range attention over extended video streams, resulting in enhanced continuity and consistency. Consequently, a Stable Diffusion model fine-tuned with RAIN in just a few epochs can produce video streams in real-time and low latency without much compromise in quality or consistency, up to infinite long. Despite its advanced capabilities, the RAIN only introduces a few additional 1D attention blocks, imposing minimal additional burden. Experiments in benchmark datasets and generating super-long videos demonstrating that RAIN can animate characters in real-time with much better quality, accuracy, and consistency than competitors while costing less latency. All code and models will be made publicly available.

Figures

Figures reproduced from arXiv: 2412.19489 by the authors.

Figure 1
Figure 1. Cross domain face morphing generation results. We achieve real-time animation of anime characters. Expression of real human [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Animation clips for crossdomain face morphing. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The overview pipeline of RAIN. We first feed the reference image into Reference UNet and CLIP Text Encoder, the spatial [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Generation results from UBC-Fashion test dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Results of cross domain face morphing: the two leftmost columns are the original DWPose sequence and the transformed [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Results of style transfer: Dynamic scenes lead to gradual loss of detail and synthesis failure, while stable scenes can be synthesized [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: An error accumulation case: The abnormal blush that initially occurs causes exceptional color blocks in the following generation [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 9
Figure 9. Figure 9: Real Human Facial Landmarks with 68 points [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 8
Figure 8. Figure 8: Anime Facial Landmarks with 26 points Keypoint Transformations In order to match the result of DWPose ( [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 10
Figure 10. Figure 10: Selected and Merged Facial Landmarks 1 [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Transformations of Landmarks: from left top to right bottom, landmarks from humans are mapped to landmarks of anime [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 36 canonical work pages

  1. [1]

    Online adaptive personalization for face anti-spoofing, 2022

    Davide Belli, Debasmit Das, Bence Major, and Fatih Porikli. Online adaptive personalization for face anti-spoofing, 2022. 8

    work page 2022

  2. [2]

    Ollin Boer Bohan. Tinyvae. https://github.com/ madebyollin/taesd, 2023. 3, 8

    work page 2023

  3. [3]

    Ollin Boer Bohan. Taesdv. https://github.com/ madebyollin/taesdv, 2024. 8

    work page 2024

  4. [4]

    Magicdance: Realistic human dance video gen- eration with motions & facial expressions transfer

    Di Chang, Yichun Shi, Quankai Gao, Jessica Fu, Hongyi Xu, Guoxian Song, Qing Yan, Xiao Yang, and Mohammad So- leymani. Magicdance: Realistic human dance video gen- eration with motions & facial expressions transfer. arXiv preprint arXiv:2311.12052, 2023. 3

    arXiv 2023

  5. [5]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 6

    work page 2024

  6. [6]

    Bermano, Gal Chechik, and Daniel Cohen-Or

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image gen- eration using textual inversion, 2022. 3

    2022

  7. [7]

    Gatys, Alexander S

    Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. A neural algorithm of artistic style, 2015. 3

    work page 2015

  8. [8]

    Open-animateanyone

    Qin Guo. Open-animateanyone. https://github. com/guoqincode/Open-AnimateAnyone, 2024. 8

    work page 2024

Show all 54 references
  1. [9]

    Animatediff: Animate your personalized text-to- image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to- image diffusion models without specific tuning. Interna- tional Conference on Learning Representations, 2024. 4, 5

    2024

  2. [10]

    Classifier-free diffusion guidance, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. 4

    2022

  3. [11]

    Image quality metrics: Psnr vs

    Alain Hor ´e and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th International Conference on Pattern Recognition, pages 2366–2369, 2010. 6

    2010

  4. [12]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685, 2021. 3

    2021 arXiv

  5. [13]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation

    Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:2311.17117, 2023. 3, 4, 8

    2023 arXiv

  6. [14]

    Arbitrary style transfer in real-time with adaptive instance normalization

    Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 1510–1519, 2017. 3 2https://civit.ai 3https://www.pixiv.net/users/101922785 4jwmeng@mail.ustc.edu.cn

    2017

  7. [15]

    Anime face detector

    hysts. Anime face detector. https://github.com/ hysts/anime-face-detector, 2021. 6, 8

    2021

  8. [16]

    Dreampose: Fashion image-to-video synthesis via stable diffusion, 2023

    Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Kemelmacher-Shlizerman. Dreampose: Fashion image-to-video synthesis via stable diffusion, 2023. 6

    2023

  9. [17]

    Streamdiffu- sion: A pipeline-level solution for real-time interactive gen- eration, 2023

    Akio Kodaira, Chenfeng Xu, Toshiki Hazama, Takanori Yoshimoto, Kohei Ohno, Shogo Mitsuhori, Soichi Sugano, Hanying Cho, Zhijian Liu, and Kurt Keutzer. Streamdiffu- sion: A pipeline-level solution for real-time interactive gen- eration, 2023. 3

    2023

  10. [18]

    Stable diffusion image variations

    Lambda Labs. Stable diffusion image variations. https: / / huggingface . co / lambdalabs / sd - image - variations-diffusers, 2023. 5

    2023

  11. [19]

    MetaPix: Few-Shot Video Retargeting

    Jessica Lee, Deva Ramanan, and Rohit Girdhar. MetaPix: Few-Shot Video Retargeting. ICLR, 2020. 2

    2020

  12. [20]

    Y . Li, N. Wang, J. Liu, and X. Hou. Demystifying neu- ral style transfer. Proceedings of the Twenty-Sixth Inter- national Joint Conference on Artificial Intelligence , pages 2230–2236, 2017. 3

    2017

  13. [21]

    Looking backward: Streaming video-to-video translation with feature banks

    Feng Liang, Akio Kodaira, Chenfeng Xu, Masayoshi Tomizuka, Kurt Keutzer, and Diana Marculescu. Looking backward: Streaming video-to-video translation with feature banks. arXiv preprint arXiv:2405.15757, 2024. 3

    2024 arXiv

  14. [22]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2019. 5

    2019

  15. [23]

    Latent consistency models: Synthesizing high- resolution images with few-step inference, 2023

    Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high- resolution images with few-step inference, 2023. 3, 4

    2023

  16. [24]

    Lcm-lora: A universal stable-diffusion acceleration module, 2023

    Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolin´ario Passos, Longbo Huang, Jian Li, and Hang Zhao. Lcm-lora: A universal stable-diffusion acceleration module, 2023. 3

    2023

  17. [25]

    SDEdit: Guided image synthesis and editing with stochastic differential equa- tions

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jia- jun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equa- tions. In International Conference on Learning Representa- tions, 2022. 3, 6

    2022

  18. [26]

    Moore-animateanyone

    Moore-Thread. Moore-animateanyone. https : / / github . com / MooreThreads / Moore - AnimateAnyone, 2024. 8

    2024

  19. [27]

    The 2017 davis challenge on video object segmentation

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Ar- bel´aez, Alexander Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv:1704.00675, 2017. 7

    2017 arXiv

  20. [28]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Con...

    2021

  21. [29]

    Vi- sion transformers for dense prediction

    Ren ´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vi- sion transformers for dense prediction. ICCV, 2021. 6

    2021

  22. [30]

    Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer

    Ren ´e Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 44(3), 2022. 6 9

    2022

  23. [31]

    Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer

    Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674–10685, 2021. 5

    2022

  24. [32]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. CoRR, abs/2202.00512,

    arXiv

  25. [33]

    De-fake: Detection and attribution of fake images generated by text- to-image generation models, 2023

    Zeyang Sha, Zheng Li, Ning Yu, and Yang Zhang. De-fake: Detection and attribution of fake images generated by text- to-image generation models, 2023. 8

    2023

  26. [34]

    Animating arbitrary objects via deep motion transfer

    Aliaksandr Siarohin, St ´ephane Lathuili`ere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2

    2019

  27. [35]

    First order motion model for image animation

    Aliaksandr Siarohin, St ´ephane Lathuili`ere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In Conference on Neural Information Pro- cessing Systems (NeurIPS), 2019. 2

    2019

  28. [36]

    Motion representations for ar- ticulated animation

    Aliaksandr Siarohin, Oliver Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for ar- ticulated animation. In CVPR, 2021. 6

    2021

  29. [37]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In International Conference on Learning Representations, 2021. 3

    2021

  30. [38]

    Score-based generative modeling through stochastic differential equa- tions

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions. In International Conference on Learning Represen- tations, 2021. 3

    2021

  31. [39]

    Consistency models

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, 2023. 3, 4

    2023

  32. [40]

    MoCoGAN: Decomposing motion and content for video generation

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1526–1535, 2018. 2

    2018

  33. [41]

    To- wards accurate generative models of video: A new metric & challenges, 2019

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges, 2019. 6

    2019

  34. [42]

    Animatelcm: Accelerating the animation of personalized diffusion mod- els and adapters with decoupled consistency learning

    Fu-Yun Wang, Zhaoyang Huang, Xiaoyu Shi, Weikang Bian, Guanglu Song, Yu Liu, and Hongsheng Li. Animatelcm: Accelerating the animation of personalized diffusion mod- els and adapters with decoupled consistency learning. arXiv preprint arXiv:2402.00769, 2024. 4

    2024 arXiv

  35. [43]

    Disco: Disentangled control for realistic human dance generation

    Tan Wang, Linjie Li, Kevin Lin, Yuanhao Zhai, Chung- Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for realistic human dance generation. arXiv preprint arXiv:2307.00040,

    arXiv

  36. [44]

    Bovik, H.R

    Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing , 13(4): 600–612, 2004. 6

    2004

  37. [45]

    Dire for diffusion-generated image detection

    Zhendong Wang, Jianmin Bao, Wengang Zhou, Weilun Wang, Hezhen Hu, Hong Chen, and Houqiang Li. Dire for diffusion-generated image detection. arXiv preprint arXiv:2303.09295, 2023. 8

    2023 arXiv

  38. [46]

    Live2diff: Live stream translation via uni-directional atten- tion in video diffusion models, 2024

    Zhening Xing, Gereon Fox, Yanhong Zeng, Xingang Pan, Mohamed Elgharib, Christian Theobalt, and Kai Chen. Live2diff: Live stream translation via uni-directional atten- tion in video diffusion models, 2024. 3

    2024

  39. [47]

    Magicanimate: Temporally consistent human image animation using diffusion model

    Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In arXiv, 2023. 3

    2023

  40. [48]

    Effec- tive whole-body pose estimation with two-stages distillation

    Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effec- tive whole-body pose estimation with two-stages distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4210–4220, 2023. 5, 6, 8

    2023

  41. [49]

    Cheung, Yuzhi Zhao, Yu Xue, and Kun Li

    Wing-Yin Yu, Lai-Man Po, Ray C.C. Cheung, Yuzhi Zhao, Yu Xue, and Kun Li. Bidirectionally deformable motion modulation for video-based human pose transfer. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7468–7478, 2023. 6

    2023

  42. [50]

    Dwnet: Dense warp-based network for pose- guided human video generation

    Polina Zablotskaia, Aliaksandr Siarohin, Bo Zhao, and Leonid Sigal. Dwnet: Dense warp-based network for pose- guided human video generation. CoRR, abs/1910.09139,

    1910 arXiv

  43. [51]

    Adding conditional control to text-to-image diffusion models, 2023

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023. 3, 6

    2023

  44. [52]

    Efros, Eli Shecht- man, and Oliver Wang

    Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 586–595, 2018. 6

    2018

  45. [53]

    Thin-plate spline motion model for image animation

    Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. 2022 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 3647– 3656, 2022. 6 10 RAIN: Real-time Animation of Infinite Video Stream Supplementary Material

    2022

  46. [54]

    The basic setting is K = 16, p= 4, H= W = 512, N = 1

    Implementation Details Inference We show the inference pipeline of our RAIN in Algorithm 1. The basic setting is K = 16, p= 4, H= W = 512, N = 1. The algorithm can inference with streaming video input with infinite length. LCM scheduler is used for sampling. The temporal batch...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.