REVIEW 3 major objections 5 minor 1 cited by
On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devices
T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read The paper claims that three training-free techniques — Linear Proportional Leap, Temporal Dimension Token Merging, and Concurrent Inference with Dynamic Loading — let a pre-trained text-to-video diffusion model run on an iPhone 15 Pro and…
desk verdict A real first demonstration of training-free on-device diffusion text-to-video, but the headline quality/latency claim is not backed by a single joint measurement. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the rectified-flow drift $v(P_k, t_k)$ and the leap identity $z_k = z_n + v(P_{n+1}, t_{n+1}) t_{n+1}$, which says that once the trajectory is straight, the accumulated update of all remaining denoising steps equals one scaled Euler step taken with the next drift. LPL uses the cosine similarity between $v(P_n, t_n)$ and $v(P_{n-1}, t_{n-1})$ as a runtime straightness probe, stopping the denoising loop when similarity stops improving. TDTM's machinery is a merge/unmerge pair on the temporal axis: input tokens of shape $[B, ST, C]$ become $[B, ST/2, C]$ by averaging consecutive frames, and the attention output is replicated back; this exploits the fact that adjacent frames in a video are near-duplicates. CI-DL's machinery is a block pipeline: the transformer is cut into segments, the CPU loads segment $i+1$ while the GPU executes segment $i$, and a dynamic number of segments stay resident in memory across denoising iterations, so the same blocks are not reloaded each step.
What would settle it
Apply Linear Proportional Leap to a flow-matching video model whose consecutive-drift cosine similarity stays below 0.95 through the second half of its schedule (CogVideoX, which the paper itself names as non-straight, is a direct candidate), and compare VBench frame-wise quality against full-step generation: a drop in imaging quality or aesthetic quality larger than the roughly 0.03 the paper reports on Open-Sora would show that LPL depends on the straightness premise, not on the leap formula alone. A sharper probe is to print the cosine-similarity curve for a 50-step rectified-flow generation and check whether it is monotone non-decreasing past step 25; if it dips, the dynamic stopping rule's 10⁻⁴ improvement tolerance can halt too early or too late, and the reported 1.94× speedup is not generally reproducible.
Extended reading notes
Core claim
On-device Sora's central claim is that the three bottlenecks of mobile video diffusion — too many denoising steps, quadratic attention cost, and model weights that exceed device memory — can each be bypassed without changing any weights. In the rectified-flow formulation used by Open-Sora, the denoising trajectory becomes nearly straight after the early steps; LPL detects this straightness from the cosine similarity of consecutive drifts and replaces all remaining steps with the single update $z_k = z_n + v(P_{n+1}, t_{n+1}) t_{n+1}$, cutting steps from 30 to as few as 16 with VBench scores essentially unchanged. TDTM merges pairs of adjacent tokens along the temporal dimension, computes attention on half the tokens, and replicates the output to restore the original shape, reducing self-attention by a factor of four and cross-attention by half. CI-DL partitions T5 and STDiT into blocks that are loaded by the CPU while the GPU executes the previous block, retaining a memory-fit subset across denoising iterations. The experimental section reports that with all three applied, Open-Sora on an iPhone 15 Pro produces 256×256 videos whose temporal-quality scores are 0.96–0.99 versus 0.97–0.99 for the same model on an A6000, with an average frame-wise drop of 0.03 and a 0.06 improvement in dynamic degree.
Load-bearing premise
The load-bearing premise is that the later stages of the rectified-flow reverse process are approximately straight, so one scaled Euler step with the final drift can replace all remaining denoising steps without visible quality loss; if the trajectory curves, LPL's leap overshoots and quality degrades.
Editorial extensions
If this is right
- Any pre-trained rectified-flow text-to-video model becomes deployable on a phone without re-training, eliminating the GPU-heavy distillation and compression pipelines that earlier on-device video generators require.
- LPL's step reduction and TDTM's attention reduction are multiplicative, and both stack with CI-DL's memory pipelining, so the combined 6x denoising speedup is not the sum but the product of the individual gains.
- Because the methods touch only the sampling schedule and token layout, they transfer to more compact DiT-based video generators as those models appear, making on-device generation progressively cheaper without new optimization work.
- Deploying the full pre-trained model on-device means prompts and videos never need to leave the phone, which makes private, offline text-to-video possible on commodity hardware.
- TDTM's motion cost matters: merging all 30 denoising steps drops dynamic degree from 0.23 to 0.06 on VBench, so applications wanting lively videos should merge only the first half of steps and sacrifice part of the speedup.
Reading between the lines
- The leap identity in LPL is a zero-training shortcut that effectively distills the model's learned drift into a near-one-step update in the straight region; the same reasoning should apply to rectified-flow image diffusion, where even larger step reductions are plausible because the trajectory straightness assumption is not video-specific.
- CI-DL's block pipeline is a general technique for any transformer whose weights exceed device memory, not just T5 and STDiT; the paper's latency formula predicts that gains grow with the number of blocks, so other over-capacity models should see comparable speedups.
- A natural extension the authors do not explore is merging both spatial and temporal tokens: combining TDTM with spatial token merging would cut self-attention by up to 16×, but the dynamic-degree drop observed with full temporal merging suggests such aggressive merging would need selective scheduling to stay usable.
- The paper runs only on the GPU and leaves the device's Neural Engine idle; if the device vendor's software eventually supports the required diffusion operators, offloading T5 or the VAE to a dedicated neural processor could cut the remaining ~500 seconds of end-to-end latency substantially.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents On-device Sora, a training-free framework for running diffusion-based text-to-video generation on mobile devices. The system combines three techniques: Linear Proportional Leap (LPL), which halts denoising early and replaces remaining steps with a scaled Euler leap; Temporal Dimension Token Merging (TDTM), which merges consecutive temporal tokens in attention layers; and Concurrent Inference with Dynamic Loading (CI-DL), which partitions models for memory-constrained execution. The authors implement the system on an iPhone 15 Pro using Open-Sora as the backbone and evaluate quality with VBench, reporting that On-device Sora achieves video quality comparable to Open-Sora on an NVIDIA A6000 while cutting end-to-end latency to about 638 seconds at 256x256 resolution. The paper also includes an ablation study of Pyramidal Flow with LPL. The central claim is that these techniques together enable efficient, high-quality on-device text-to-video generation without retraining.
Significance. If the central claim is substantiated, this is a useful and timely contribution: it avoids the large GPU training budgets required by prior on-device video generation methods such as SnapGen-V, and it is accompanied by an open-source implementation and real-device latency measurements. The system is concretely implemented on commodity smartphone hardware, and the three techniques are clearly separable and tested in isolation. However, the paper's headline result—high quality and low latency from the complete pipeline—is not directly evidenced: no VBench evaluation is reported for the exact configuration used in the latency table, and the quality metrics are saturated around 0.99 without error bars or significance tests. The LPL straightness assumption is also only validated indirectly. These gaps currently prevent a full endorsement of the claimed result.
major comments (3)
- [§7.4, Table 5 vs. §7.3, Table 4] The central claim of "comparable quality at 638 s" is not established because no quality measurement is reported for the exact 'All' configuration that produces the 638-second latency. Section 7.4 states that 'All' uses LPL activated at the 15th denoising step and TDTM applied throughout all steps. However, Table 4 shows that applying TDTM for 30/30 steps on VBench reduces Dynamic Degree to 0.06 (and 15/30 to 0.12), while Table 2 reports On-device Sora's Dynamic Degree as 0.27; these cannot describe the same configuration. The quality table and the latency table therefore appear to have been generated under different LPL/TDTM settings. Please provide VBench scores for the exact 'All' configuration used in Table 5, ideally with per-prompt error bars, and clarify which settings were used for each row of Table 2.
- [§7.1, Table 2] The quality comparison relies on heavily saturated metrics: Temporal Flickering, Motion Smoothness, Subject Consistency, and Background Consistency are all around 0.97–0.99 for both methods, and the reported differences (e.g., a 0.03 average drop in frame-wise quality) are within the noise of a single VBench run. No error bars, per-category standard deviations, or significance tests are provided, so the claim that quality is "nearly equivalent" is statistically unsupported. Please report confidence intervals (e.g., bootstrap over the 800 prompts) or per-prompt distributions, and consider supplementing with a less saturated metric such as FVD or human evaluation.
- [§4.2, Fig. 4, App. C] The LPL method rests on the assumption that the reverse-process trajectory is approximately straight after a certain number of steps, but this is validated only by cosine similarities between adjacent drifts on the same models; the paper itself concedes in App. C that LPL is inapplicable to non-rectified-flow models such as CogVideoX. The runtime stopping rule also involves hand-tuned parameters (a 1e-4 tolerance and a 50% minimum step fraction). Please provide a sensitivity analysis of these thresholds and, more importantly, a direct downstream evaluation (e.g., VBench or FVD) comparing LPL's output at early stop against the full-trajectory output, to confirm that the straightness assumption preserves quality rather than merely correlating with drift similarity.
minor comments (5)
- [§1, §9] The paper contains two naming typos: "Conference Inference and Dynamic Loading" in Section 1 should be "Concurrent Inference and Dynamic Loading," and "Linear Professional Leap" in Section 9 should be "Linear Proportional Leap."
- [App. A.1] The text repeatedly uses "retrained state" (e.g., "loaded in a retrained state" and "with a retrained state"); this should be "retained state."
- [Table 5] The baseline latency of 1768.32 seconds for STDiT comes from Table 1, which reports the 50-step configuration, whereas all other rows in Table 5 presumably use a 30-step schedule; please state the number of denoising steps for each row to make the comparison fair and reproducible.
- [§7.2, §7.3] Tables 3 and 4 report speedups relative to a 30/30 baseline, but Table 2's "On-device Sora" row does not state which LPL/TDTM configuration was used; please specify this so the reader can map the headline quality numbers to the actual pipeline.
- [§1, §8] The introduction claims broad applicability to "pre-trained video generative models, e.g., Open-Sora, Pyramidal Flow," but only LPL is experimented on Pyramidal Flow; TDTM and CI-DL are only evaluated with Open-Sora. Please clarify the scope of the generality claim.
Circularity Check
No significant circularity: LPL, TDTM, and CI-DL are empirical approximations validated against the external VBench benchmark, with no self-citation chain and no fitted quantity renamed as a prediction.
full rationale
The derivation chain is not circular. LPL (Sec. 4.2) is an algebraic consequence of an explicitly stated straight-trajectory assumption: Eq. (5) shows that if the remaining drift is held constant, the remaining steps collapse to a single scaled step. The paper does not claim to derive straightness from this equation; instead it probes straightness independently via cosine similarity between adjacent drifts (Fig. 4) and validates quality with the external VBench benchmark (Tab. 3). The dynamic-stop thresholds (10^-4 tolerance and 50% minimum steps) are design hyperparameters, not parameters fitted to VBench and then reported as a prediction of VBench. TDTM (Sec. 5.2) is a simple averaging-and-unmerging operation whose quality cost and speedup are measured externally (Tab. 4), and CI-DL (App. A.1) is an engineering scheme whose latency formulas (Eqs. 10-11) are definitions of the overlap mechanism rather than fitted outputs. There are no self-citations that carry the argument, no uniqueness theorem imported from the authors' prior work, and no renamed known result presented as a derivation. The skeptical observation that Tab. 5's 'All' configuration is not matched by a VBench table with the same settings is an experimental reporting gap, not a circular reduction of the central claim; the reader's noted limitation that LPL requires rectified-flow straightness is explicitly acknowledged in App. C. The central claims remain independently testable against Open-Sora on VBench and against measured on-device latency, so the paper receives a 0 circularity score.
Assumptions & free parameters
free parameters (3)
- LPL cosine-similarity tolerance =
1e-4
- LPL minimum step fraction =
50%
- TDTM merging schedule =
first 15 of 30 steps in Tab. 4 recommendation; 30/30 in Tab. 5 'All'
assumptions (4)
- standard math Euler method for ODE integration
- domain assumption Rectified-flow drift direction is approximately constant in later denoising steps
- domain assumption Consecutive video frames are similar enough that averaging their tokens preserves semantics
- domain assumption Open-Sora (STDiT, T5, VAE) is a valid pre-trained video generation model
Cite this review
Pith. "Pith review of On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devices." pith.science (2026). https://pith.science/paper/LRW3GFUA
@misc{pith2026250204363,
author = {Pith},
title = {Pith review of: On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devices},
year = {2026},
howpublished = {\url{https://pith.science/paper/LRW3GFUA}},
note = {Machine review of arXiv:2502.04363}
}
read the original abstract
We present On-device Sora, the first model training-free solution for diffusion-based on-device text-to-video generation that operates efficiently on smartphone-grade devices. To address the challenges of diffusion-based text-to-video generation on computation- and memory-limited mobile devices, the proposed On-device Sora applies three novel techniques to pre-trained video generative models. First, Linear Proportional Leap (LPL) reduces the excessive denoising steps required in video diffusion through an efficient leap-based approach. Second, Temporal Dimension Token Merging (TDTM) minimizes intensive token-processing computation in attention layers by merging consecutive tokens along the temporal dimension. Third, Concurrent Inference with Dynamic Loading (CI-DL) dynamically partitions large models into smaller blocks and loads them into memory for concurrent model inference, effectively addressing the challenges of limited device memory. We implement On-device Sora on the iPhone 15 Pro, and the experimental evaluations show that it is capable of generating high-quality videos on the device, comparable to those produced by high-end GPUs. These results show that On-device Sora enables efficient and high-quality video generation on resource-constrained mobile devices. We envision the proposed On-device Sora as a significant first step toward democratizing state-of-the-art generative technologies, enabling video generation on commodity mobile and embedded devices without resource-intensive re-training for model optimization (compression). The code implementation is available at a GitHub repository(https://github.com/eai-lab/On-device-Sora).
Figures
Figures from the paper (9 more)
Forward citations
Cited by 1 Pith paper
-
MobileWan: Closing the Quality Gap for Mobile Video Diffusion
A 5B-parameter video diffusion transformer is made deployable on mobile hardware via recurrence distillation, learnable head pruning, step distillation, and decoder optimization, achieving 83.79 VBench at 20s latency.
Reference graph
Works this paper leans on
-
[1]
iphone 15 pro—technical specifications, 2023
Apple. iphone 15 pro—technical specifications, 2023. [On- line]. Available: https://support.apple.com/en- us/111829. 2, 3, 6, 14, 16
2023
-
[2]
Swift, 2024
Apple. Swift, 2024. https://developer.apple. com/swift/. 6
2024
-
[3]
A discussion on euler method: A review
BN Biswas, Somnath Chatterjee, SP Mukherjee, and Subhradeep Pal. A discussion on euler method: A review. Electronic Journal of Mathematical Analysis and Applica- tions, 1(2):2090–2792, 2013. 2
2013
-
[4]
Stable video diffusion: Scaling latent video diffusion models to large datasets
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 1, 8
arXiv 2023
-
[5]
Align your latents: High-resolution video synthesis with la- tent diffusion models
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023. 1
2023
-
[6]
To- ken merging: Your vit but faster
Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. To- ken merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022. 2, 3, 5, 9
arXiv 2022
-
[7]
Token merging: Your ViT but faster
Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations, 2023. 2, 3, 5, 9
2023
-
[8]
Edgefusion: On-device text-to-image generation
Thibault Castells, Hyoung-Kyu Song, Tairen Piao, Shinkook Choi, Bo-Kyeong Kim, Hanyoung Yim, Changgwun Lee, Jae Gon Kim, and Tae-Ho Kim. Edgefusion: On-device text-to-image generation. arXiv preprint arXiv:2404.11925,
Show all 86 references
-
[9]
Tempme: Towards the explain- ability of temporal graph neural networks via motif discov- ery
Jialin Chen and Rex Ying. Tempme: Towards the explain- ability of temporal graph neural networks via motif discov- ery. Advances in Neural Information Processing Systems, 36,
-
[10]
Neural ordinary differential equa- tions
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equa- tions. Advances in neural information processing systems , 31, 2018. 4
2018
-
[11]
Panda-70m: Captioning 70m videos with multiple cross-modality teachers
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. arXiv preprint arXiv:240...
2024 arXiv
-
[12]
Speed is all you need: On-device acceleration of large diffu- sion models via gpu-aware optimizations
Yu-Hui Chen, Raman Sarokin, Juhyun Lee, Jiuqiang Tang, Chuo-Ling Chang, Andrei Kulik, and Matthias Grundmann. Speed is all you need: On-device acceleration of large diffu- sion models via gpu-aware optimizations. In Proceedings of the IEEE/CVF Conference on Computer Vision and...
2023
-
[13]
Squeezing large-scale diffusion models for mobile
Jiwoong Choi, Minkyu Kim, Daehyun Ahn, Taesu Kim, Yul- hwa Kim, Dongwon Jo, Hyesung Jeon, Jae-Joon Kim, and Hyungjun Kim. Squeezing large-scale diffusion models for mobile. arXiv preprint arXiv:2307.01193, 2023. 1
2023 arXiv
-
[14]
Diffusion models beat gans on image synthesis
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informa- tion processing systems, 34:8780–8794, 2021. 1
2021
-
[15]
Tutorial on variational autoencoders
Carl Doersch. Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908, 2016. 2, 3, 6, 8, 16
2016 arXiv
-
[16]
Scaling recti- fied flow transformers for high-resolution image synthesis
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...
2024
-
[17]
Efficient vision trans- former via token merger
Zhanzhou Feng and Shiliang Zhang. Efficient vision trans- former via token merger. IEEE Transactions on Image Pro- cessing, 2023. 5, 9
2023
-
[18]
Efficient time series processing for transform- ers and state-space models through token merging
Leon G ¨otz, Marcel Kollovieh, Stephan G¨unnemann, and Leo Schwinn. Efficient time series processing for transform- ers and state-space models through token merging. arXiv preprint arXiv:2405.17951, 2024. 9
2024 arXiv
-
[19]
Knowledge distillation: A survey
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. Interna- tional Journal of Computer Vision, 129(6):1789–1819, 2021. 16
2021
-
[20]
Gray and David L
Robert M. Gray and David L. Neuhoff. Quantization. IEEE transactions on information theory, 44(6):2325–2383, 1998. 5
1998
-
[21]
Flexible diffusion modeling of long videos
William Harvey, Saeid Naderiparizi, Vaden Masrani, Chris- tian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. Advances in Neural Information Processing Systems, 35:27953–27965, 2022. 1
2022
-
[22]
Distilling the knowledge in a neural net- work
Geoffrey Hinton. Distilling the knowledge in a neural net- work. arXiv preprint arXiv:1503.02531, 2015. 5
2015 arXiv
-
[23]
Denoising dif- fusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1
2020
-
[24]
Imagen video: High definition video generation with diffusion mod- els
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion mod- els. arXiv preprint arXiv:2210.02303, 2022. 1
-
[25]
Cascaded diffu- sion models for high fidelity image generation
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffu- sion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1–33, 2022. 1
2022
-
[26]
Video dif- fusion models
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022. 1
2022
-
[27]
Toward controlled generation of text
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhut- dinov, and Eric P Xing. Toward controlled generation of text. In International conference on machine learning, pages 1587–1596. PMLR, 2017. 1
2017
-
[28]
Vbench: Comprehensive bench- mark suite for video generative models
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive bench- mark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...
2024
-
[29]
Pyramidal flow matching for efficient video generative modeling
Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:2410.05954 ,
-
[30]
Imagic: Text-based real image editing with diffusion models
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007–6017, 2023. 1
2023
-
[31]
Text2video-zero: Text- to-image diffusion models are zero-shot video generators
Levon Khachatryan, Andranik Movsisyan, Vahram Tade- vosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text- to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Comput...
2023
-
[32]
Black Forest Labs. Flux. https://github.com/ black-forest-labs/flux, 2024. 15
2024
-
[33]
xformers: A modular and hackable trans- former modelling library
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hackable trans- forme...
2022
-
[34]
Vidtome: Video token merging for zero-shot video editing
Xirui Li, Chao Ma, Xiaokang Yang, and Ming-Hsuan Yang. Vidtome: Video token merging for zero-shot video editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7486–7495, 2024. 5, 9
2024
-
[35]
Snap- fusion: Text-to-image diffusion model on mobile devices within two seconds
Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snap- fusion: Text-to-image diffusion model on mobile devices within two seconds. Advances in Neural Information Pro- cessing Systems, 36, 2024. 1
2024
-
[36]
Animatediff-lightning: Cross-model diffusion distillation
Shanchuan Lin and Xiao Yang. Animatediff-lightning: Cross-model diffusion distillation. arXiv preprint arXiv:2403.12706, 2024. 1
2024 arXiv
-
[37]
Generative adversarial networks for image and video synthesis: Algorithms and applications
Ming-Yu Liu, Xun Huang, Jiahui Yu, Ting-Chun Wang, and Arun Mallya. Generative adversarial networks for image and video synthesis: Algorithms and applications. Proceedings of the IEEE, 109(5):839–862, 2021. 1
2021
-
[38]
Flow straight and fast: Learning to generate and transfer data with rectified flow
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022. 4, 8, 15, 17
2022 arXiv
-
[39]
Sora: A review on background, technology, limitations, and opportunities of large vision models
Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jian- feng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177, 2024. 13
2024 arXiv
-
[40]
Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongx- uan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022. 4
2022 arXiv
-
[41]
Snap video: Scaled spatiotemporal transformers for text-to-video synthesis
Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, et al. Snap video: Scaled spatiotemporal transformers for text-to-video synthesis. In Proceedings of the IEEE/CVF Conference...
2024
-
[42]
Dreamix: Video diffusion models are general video editors
Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen. Dreamix: Video diffusion models are general video editors. arXiv preprint arXiv:2302.01329, 2023. 1
2023 arXiv
-
[43]
A review on the attention mechanism of deep learning
Zhaoyang Niu, Guoqiang Zhong, and Hui Yu. A review on the attention mechanism of deep learning. Neurocomputing, 452:48–62, 2021. 3
2021
-
[44]
Generative models for video analysis and 3D range data applications
Xavier Orriols. Generative models for video analysis and 3D range data applications . Universitat Aut `onoma de Barcelona,, 2004. 1
2004
-
[45]
Pytorch: An im- perative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Ad- vances in neural information processing systems ...
2019
-
[46]
Scalable diffusion models with transformers
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,
-
[47]
Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 1, 3
2023 arXiv
-
[48]
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 2, 3, 6, 8,...
2020
-
[49]
Pruning algorithms-a survey
Russell Reed. Pruning algorithms-a survey. IEEE transac- tions on Neural Networks, 4(5):740–747, 1993. 5, 16
1993
-
[50]
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 8
2022
-
[51]
U- net: Convolutional networks for biomedical image segmen- tation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...
2015
-
[52]
¨Uber die numerische aufl¨osung von differential- gleichungen
Carl Runge. ¨Uber die numerische aufl¨osung von differential- gleichungen. Mathematische Annalen, 46(2):167–178, 1895. 4
-
[53]
Palette: Image-to-image diffusion models
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pages 1–10,
2022
-
[54]
Introduction to apple ml tools
¨Ozg¨ur Sahin and ¨Ozg¨ur Sahin. Introduction to apple ml tools. Develop Intelligent iOS Apps with Swift: Understand Texts, Classify Sentiments, and Autodetect Answers in Text Using NLP, pages 17–39, 2021. 6
2021
-
[55]
Sin- gan: Learning a generative model from a single natural im- age
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Sin- gan: Learning a generative model from a single natural im- age. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pages 4570–4580, 2019. 1
2019
-
[56]
Make-a-video: Text-to-video generation without text-video data
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,
-
[57]
Video edit- ing via factorized diffusion distillation
Uriel Singer, Amit Zohar, Yuval Kirstain, Shelly Sheynin, Adam Polyak, Devi Parikh, and Yaniv Taigman. Video edit- ing via factorized diffusion distillation. In European Con- ference on Computer Vision, pages 450–466. Springer, 2024. 1
2024
-
[58]
Denoising diffusion implicit models
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 1, 3
2010 arXiv
-
[59]
Roformer: Enhanced transformer with rotary position embedding
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,
-
[60]
A survey of multi- modal deep generative models
Masahiro Suzuki and Yutaka Matsuo. A survey of multi- modal deep generative models. Advanced Robotics, 36(5-6): 261–278, 2022. 1
2022
-
[61]
Vidgen-1m: A large-scale dataset for text-to-video genera- tion
Zhiyu Tan, Xiaomeng Yang, Luozheng Qin, and Hao Li. Vidgen-1m: A large-scale dataset for text-to-video genera- tion. arXiv preprint arXiv:2408.02629, 2024. 6
2024 arXiv
-
[62]
Qvd: Post-training quantization for video diffusion models
Shilong Tian, Hong Chen, Chengtao Lv, Yu Liu, Jinyang Guo, Xianglong Liu, Shengxi Li, Hao Yang, and Tao Xie. Qvd: Post-training quantization for video diffusion models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 10572–10581, 2024. 1
2024
-
[63]
To- wards accurate generative models of video: A new metric & challenges
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 1
2018 arXiv
-
[64]
Mobileone: An im- proved one millisecond mobile backbone
Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Mobileone: An im- proved one millisecond mobile backbone. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7907–7917, 2023. 1
2023
-
[65]
Mcvd-masked conditional video diffusion for prediction, generation, and interpolation
Vikram V oleti, Alexia Jolicoeur-Martineau, and Chris Pal. Mcvd-masked conditional video diffusion for prediction, generation, and interpolation. Advances in neural informa- tion processing systems, 35:23371–23385, 2022. 1
2022
-
[66]
Animatelcm: Computation-efficient personalized style video generation without personalized video data
Fu-Yun Wang, Zhaoyang Huang, Weikang Bian, Xiaoyu Shi, Keqiang Sun, Guanglu Song, Yu Liu, and Hongsheng Li. Animatelcm: Computation-efficient personalized style video generation without personalized video data. In SIGGRAPH Asia 2024 Technical Communications, New York, NY , USA,
2024
-
[67]
Transformers: State-of-the-art natural language processing
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau- mond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R ´emi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceed- ings of the 2020 conference on empirical methods...
2020
-
[68]
Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...
2023
-
[69]
In- dividual content and motion dynamics preserved pruning for video diffusion models
Yiming Wu, Huan Wang, Zhenghao Chen, and Dong Xu. In- dividual content and motion dynamics preserved pruning for video diffusion models. arXiv preprint arXiv:2411.18375 ,
-
[70]
Snapgen-v: Generating a five-second video within five seconds on a mobile device
Yushu Wu, Zhixing Zhang, Yanyu Li, Yanwu Xu, Anil Kag, Yang Sui, Huseyin Coskun, Ke Ma, Aleksei Lebe- dev, Ju Hu, et al. Snapgen-v: Generating a five-second video within five seconds on a mobile device. arXiv preprint arXiv:2412.10494, 2024. 1, 8
2024 arXiv
-
[71]
Mobile video diffusion
Haitam Ben Yahia, Denis Korzhenkov, Ioannis Lelekas, Amir Ghodrati, and Amirhossein Habibian. Mobile video diffusion. arXiv preprint arXiv:2412.07583, 2024. 1, 8
2024 arXiv
-
[72]
Stat: Spatial-temporal attention mechanism for video cap- tioning
Chenggang Yan, Yunbin Tu, Xingzheng Wang, Yongbing Zhang, Xinhong Hao, Yongdong Zhang, and Qionghai Dai. Stat: Spatial-temporal attention mechanism for video cap- tioning. IEEE transactions on multimedia , 22(1):229–241,
-
[73]
Diffusion models: A comprehensive survey of methods and applications
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Run- sheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming- Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4): 1–39, 2023. 4
2023
-
[74]
Cogvideox: Text-to-video diffusion models with an expert transformer
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 17
2024 arXiv
-
[75]
Schedule on the fly: Diffusion time prediction for faster and better image generation
Zilyu Ye, Zhiyang Chen, Tiancheng Li, Zemin Huang, Wei- jian Luo, and Guo-Jun Qi. Schedule on the fly: Diffusion time prediction for faster and better image generation. arXiv preprint arXiv:2412.01243, 2024. 17
2024 arXiv
-
[76]
Language model beats diffusion–tokenizer is key to visual generation
Lijun Yu, Jos ´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023. 3
-
[77]
Text-to-image diffusion models in gener- ative ai: A survey
Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image diffusion models in gener- ative ai: A survey. arXiv preprint arXiv:2303.07909, 2023. 2
2023 arXiv
-
[78]
Adding conditional control to text-to-image diffusion models
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 1
2023
-
[79]
A survey on personalized content synthesis with diffusion models
Xulu Zhang, Xiao-Yong Wei, Wengyu Zhang, Jinlin Wu, Zhaoxiang Zhang, Zhen Lei, and Qing Li. A survey on personalized content synthesis with diffusion models. arXiv preprint arXiv:2405.05538, 2024. 1
2024
-
[80]
Vidit-q: Efficient and accurate quantization of diffusion transformers for im- age and video generation
Tianchen Zhao, Tongcheng Fang, Enshu Liu, Rui Wan, Widyadewi Soedarmadji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, Huazhong Yang, et al. Vidit-q: Efficient and accurate quantization of diffusion transformers for im- age and video generation. arXiv preprint arXiv:2406.02540,
-
[81]
Mobilediffusion: Subsecond text-to-image generation on mobile devices
Yang Zhao, Yanwu Xu, Zhisheng Xiao, and Tingbo Hou. Mobilediffusion: Subsecond text-to-image generation on mobile devices. arXiv preprint arXiv:2311.16567, 2023. 1
2023 arXiv
-
[82]
Dpm- solver-v3: Improved diffusion ode solver with empirical model statistics
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm- solver-v3: Improved diffusion ode solver with empirical model statistics. Advances in Neural Information Process- ing Systems, 36:55502–55542, 2023. 4, 17
2023
-
[83]
Open-sora: Democratizing efficient video production for all, 2024
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024. 1, 2, 3, 4, 5, 6, 7, 8, 13, 14, 15, 16
2024
-
[84]
Slimflow: Training smaller one-step diffusion models with rectified flow
Yuanzhi Zhu, Xingchao Liu, and Qiang Liu. Slimflow: Training smaller one-step diffusion models with rectified flow. In European Conference on Computer Vision , pages 342–359. Springer, 2025. 8 A. Appendix - Implementation A.1. Concurrent Inference with Dynamic Loading On-devic...
2025
-
[86]
and Temporal Dimension Token Merging (Sec. 5). Straightness Constraints. For video generation mod- els that do not exhibit straightness during their denois- ing procedures, such as CogVideoX [74], which employs DPM-Solver [82], Linear Proportional Leap (LPL) is cur- rently ina...
-
[2024]
Association for Computing Machinery. 1
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.