Pith. sign in

REVIEW 4 major objections 5 minor 51 references

SUGAR: Subject-Driven Video Customization in a Zero-Shot Manner

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SUGAR claims that zero-shot subject-driven video customization—generating a video of a subject from a single image and a text prompt—can match or beat per-instance fine-tuning baselines, with no test-time optimization.

desk verdict A credible zero-shot video customization system with a strong synthetic data pipeline, but the motion-alignment claim outruns the evidence. read the letter →

arxiv 2412.10533 v1 pith:WGPYUEVN submitted 2024-12-13 cs.CV

Yufan Zhou , Ruiyi Zhang , Jiuxiang Gu , Nanxuan Zhao , Jing Shi , Tong Sun This is my paper · ORCID
classification cs.CV
keywords subject-drivenvideocustomizationzero-shotgenerationtext-to-videodiffusionsyntheticdatasetconstructionidentitypreservationclassifier-freeguidancedynamicsvideo-textalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that customizing a video to a specific subject can be done zero-shot: from one input image and a text prompt, the model generates a video of that subject with the requested style, texture, and motion, without any per-subject fine-tuning. The difficulty is that previous zero-shot approaches either lose identity or fail to follow the text, while fine-tuning approaches are expensive. The paper's solution is to train a transformer-based video diffusion model on a large synthetic corpus of image-video-text triplets, supplemented with real-world videos, and to use attention, training, and sampling designs that preserve identity while allowing motion. The reported experiments support the conclusion that the combination works: SUGAR reports the best identity preservation, dynamic degree, and text alignment among the compared methods, with no extra test-time cost.

What carries the argument

The machinery is the synthetic triplet pipeline, the selective attention design, the two-stage freezing training strategy, and the refined sampling algorithm. The pipeline produces 2.5 million (input image, target video, text) samples by segmenting out the subject, generating a customized image of it with a subject-driven text-to-image model, and animating that image with a pre-trained image-to-video model. The selective attention design lets only the first video frame attend directly to the identity embeddings, so the remaining frames can move more freely. The two-stage strategy first trains on real-world videos, then fine-tunes on a mix of real and synthetic data while freezing the first half of the transformer layers, preserving motion learned from real data while absorbing style and texture changes. The sampling algorithm applies separate guidance scales for identity and text, and in early timesteps drops the fine-grained identity embedding while keeping the coarse one, which the paper reports improves motion without losing identity.

What would settle it

Generate videos with the released model on prompts whose motions the image-to-video teacher cannot demonstrate (for example, a robot dancing or a dog swimming), and measure the dynamic degree with optical flow while tracking identity with a self-supervised visual similarity score. If the generated videos are no more dynamic than the teacher's near-static outputs whenever identity is preserved, the claim that the synthetic-triplet training plus the proposed fixes overcomes the teacher's motion bias is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that subject-driven video customization does not require test-time fine-tuning: a single subject image plus a text prompt is enough to generate a video that preserves the identity while following the text's style, texture, and motion. To get there, the paper proposes training a transformer-based latent diffusion video model on 2.5 million synthetic image-video-text triplets together with real-world videos, and it reports that the resulting model outperforms previous customization methods, including sequential text-to-image plus image-to-video pipelines, on identity preservation, motion dynamics, and video-text alignment, at zero extra test-time cost.

Load-bearing premise

The load-bearing premise is that the synthetic video triplets, although most of them show subjects standing still because the image-to-video teacher fails to generate text-aligned motion, can still be combined with real-world videos and the paper's attention, training, and sampling fixes to teach text-aligned motion; if that teacher bias cannot be repaired, the motion gains claimed by the method do not follow.

Editorial extensions

If this is right

  • At inference time a single subject image and a text prompt are the only inputs, so customization costs the same as ordinary text-to-video generation.
  • Users can steer the identity-motion trade-off at inference by adjusting the identity and text guidance scales and by choosing whether to drop the fine-grained identity embedding early in sampling.
  • The 2.5-million-triplet synthetic customization dataset is load-bearing: ablations show it improves text alignment over training on real-world videos alone, and video triplets outperform the corresponding single-frame image data.
  • The two-stage training strategy with the first half of the transformer layers frozen is what the paper credits for keeping real-video motion while absorbing style and texture changes from the synthetic corpus.
  • Because the dataset pipeline can swap in better pre-trained image-to-video generators, the method's quality ceiling is tied to those teachers rather than to the model architecture alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Restricting direct identity conditioning to the first frame is a design principle that could transfer to other transformer-based video models, and testing it there would show whether global identity conditioning generally suppresses motion.
  • The same triplet-construction recipe could be reused for other customization axes beyond style and motion, such as viewpoint or lighting changes, by changing the prompts fed to the text-to-image teacher.
  • A motion-category benchmark would likely reveal the method's limits: motions the image-to-video teacher cannot produce are the ones most likely to fail, so grouping evaluation prompts by motion type would be a sharper test than the reported averages.
  • The strong dependence of the results on the identity guidance scale suggests an automatic, per-prompt choice of that scale could replace the manual tuning reported in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SUGAR, a zero-shot subject-driven video customization method that takes a single subject image and a text prompt and generates a customized video of that subject. The method extends the CogVideoX transformer with DINO/CLIP image embeddings, and is trained on a newly constructed 2.5M image-video-text synthetic dataset generated via Toffee and DynamiCraft, mixed with real-world videos. The authors propose several model improvements: selective attention designs, a two-stage training strategy with frozen layers (SUGAR-TSF), and a refined sampling algorithm with dual guidance and early-stage dropping of image embeddings. Quantitative results on a DreamBench-derived benchmark are reported, and the abstract and conclusion claim state-of-the-art performance in identity preservation, video dynamics, and video-text alignment without test-time fine-tuning.

Significance. If the claims are substantiated, SUGAR would be a practically valuable contribution: it avoids per-subject fine-tuning, introduces a large-scale synthetic data construction pipeline, and systematically ablates attention, training, and sampling choices. The paper's strengths include the explicit large-scale dataset construction, the careful ablation of attention designs and training strategies, and the use of a standard subject set with deliberately designed motion prompts. However, the current evaluation does not fully support the headline state-of-the-art claim, because the quantitative evidence has no error bars or significance tests, no single cited operating point dominates all metrics, and the reported metrics do not measure whether generated motions actually match the motion verbs in the text prompts.

major comments (4)
  1. [Table 1 and Abstract]
  2. [Section 4.1]
  3. [Sections 3.1 and 4.1]
  4. [Section 3.2]
minor comments (5)
  1. [Section 3.2]
  2. [Figure 4 caption]
  3. [Section 4.2 and Figure 9]
  4. [Section 1 and 5]
  5. [Section 4.2]

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: SUGAR's results are empirical and benchmarked externally; the DINO/CLIP data-curation overlap is a validity caveat, not a derivation-level circularity.

full rationale

The claimed derivation chain is a supervised training pipeline: synthetic triplets are generated by Toffee-based customization followed by DynamiCraft animation, and the model is trained to denoise latents conditioned on DINO, CLIP, and T5 embeddings. No equation in the paper reduces a predicted quantity to a fitted parameter or to the evaluation metric by construction. The main caveat is that Sections 3.1 and 4.1 use the same DINO and CLIP similarity functions for synthetic-data filtering and for identity/text-alignment evaluation, and the optical-flow static-video filter overlaps with the RAFT dynamic-degree metric; this can bias metric values favorably and is a legitimate validity concern. It is not, however, circular in the derivation sense: the model is not trained to optimize these metric scores directly, the comparisons are against external baselines on DreamBench prompts, and the paper's own admission that the I2V teacher often fails to generate text-aligned motions is a data-quality limitation rather than a circular step. The self-citation to Toffee [51] is a methodological dependency, not a load-bearing circular justification, because the synthetic-data contribution is tested through internal ablations and Toffee outputs are only an intermediate data source. Overall, no load-bearing step reduces to its own inputs, so the circularity score is low.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality of the synthetic triplets and on proxy metrics. The paper introduces no new physical entities; its new components are architectural and procedural. Numerical hyperparameters such as guidance scales, sampling probability p, dropout rates, and t bar are hand-chosen, and several are swept to produce the reported trade-off curves.

free parameters (6)
  • Identity guidance scale ωI = 7.5 in main setting; swept over 7.5, 5, 4, 3, 2.5
    Controls the trade-off between identity preservation and motion; tuned on the evaluation prompts rather than set by a theory.
  • Text guidance scale ωT = 7.5 to 2.5 in main table
    Higher values improve identity at the cost of text alignment and motion; reported as operating points rather than a fixed value.
  • Synthetic data sampling probability p = 0.5 for SUGAR-TS and SUGAR-TSF; 0.0, 1.0 tested
    Controls the balance between real-world motion and synthetic attribute variation; chosen to balance dynamic degree and text alignment.
  • Condition embedding dropout probabilities = DINO 0.5, CLIP 0.2, text 0.2
    Used for classifier-free guidance; values are hand-chosen and not justified by experiments.
  • Early sampling timestep threshold t bar = T = 1000 normally; 900 when dropping image embeddings
    Determines when image embeddings are dropped to improve motion; chosen by experiment.
  • Frozen layer fraction in SUGAR-TSF = First half of model layers
    Freezing rule assumed to preserve motion capability; no search over the fraction is reported.
assumptions (4)
  • domain assumption DINOv2, CLIP, and T5 embeddings are reliable proxies for identity preservation and text-video alignment.
    Used as the training conditions and the evaluation metrics (Sections 3, 4.1). If these embeddings miss style or identity cues, the reported improvements may not match human perception.
  • domain assumption Grounded SAM segmentation extracts clean identity images z containing only the subject.
    The identity image is obtained by applying Grounded SAM to the input image and to sampled real-video frames (Sections 3, 3.2); segmentation failures would inject background or wrong object into the identity condition.
  • domain assumption Synthetic triplets generated by Toffee and DynamiCraft are clean enough to teach text-aligned identity transfer.
    The dataset pipeline filters by CLIP and DINO similarity, but the paper does not quantify failures; it also notes most synthetic videos are static (Section 3.2), so the motion capability must come from elsewhere.
  • ad hoc to paper First-half layer freezing preserves real-world motion while later layers learn attribute changes.
    Stated as an assumption in Section 3.2 without measuring which layers encode motion; the choice of first half is not derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SUGAR: Subject-Driven Video Customization in a Zero-Shot Manner." pith.science (2026). https://pith.science/paper/WGPYUEVN

@misc{pith2026241210533,
  author       = {Pith},
  title        = {Pith review of: SUGAR: Subject-Driven Video Customization in a Zero-Shot Manner},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WGPYUEVN}},
  note         = {Machine review of arXiv:2412.10533}
}
read the original abstract

We present SUGAR, a zero-shot method for subject-driven video customization. Given an input image, SUGAR is capable of generating videos for the subject contained in the image and aligning the generation with arbitrary visual attributes such as style and motion specified by user-input text. Unlike previous methods, which require test-time fine-tuning or fail to generate text-aligned videos, SUGAR achieves superior results without the need for extra cost at test-time. To enable zero-shot capability, we introduce a scalable pipeline to construct synthetic dataset which is specifically designed for subject-driven customization, leading to 2.5 millions of image-video-text triplets. Additionally, we propose several methods to enhance our model, including special attention designs, improved training strategies, and a refined sampling algorithm. Extensive experiments are conducted. Compared to previous methods, SUGAR achieves state-of-the-art results in identity preservation, video dynamics, and video-text alignment for subject-driven video customization, demonstrating the effectiveness of our proposed method.

Figures

Figures reproduced from arXiv: 2412.10533 by the authors.

Figure 1
Figure 1. Generated examples from the proposed method, where the frames are randomly sampled from the generated video. Our proposed [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. The proposed pipeline for synthetic data generation. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 2
Figure 2. Illustration of our model, randomly sampled frames from [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Different attention designs of our proposed model. One [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Comparison of models with different attention designs. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Comparison of models trained with video customization [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Generated examples with different levels of guidance. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 12
Figure 12. Figure 12: Generated examples from SUGAR and baselines. [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 13
Figure 13. Figure 13: Generated examples from SUGAR and baselines. [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 14
Figure 14. Figure 14: Generated examples from SUGAR and baselines. [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]
Figure 15
Figure 15. Figure 15: Generated examples from SUGAR and baseline. [PITH_FULL_IMAGE:figures/full_fig_p013_15.png]
Figure 16
Figure 16. Figure 16: Generated examples from SUGAR and baseline. [PITH_FULL_IMAGE:figures/full_fig_p013_16.png]
Figure 19
Figure 19. Figure 19: Testing prompts designed for active objects. [PITH_FULL_IMAGE:figures/full_fig_p013_19.png]
Figure 20
Figure 20. Figure 20: Testing prompts designed for static objects. [PITH_FULL_IMAGE:figures/full_fig_p014_20.png]
Figure 21
Figure 21. Figure 21: Testing prompts designed for animals. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_21.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 20 canonical work pages

  1. [1]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 1728–1738,

  2. [2]

    Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models

    Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233, 2024. 6

    arXiv 2024

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 1

    arXiv 2023

  4. [4]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22563–22575, 2023. 1

    work page 2023

  5. [5]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. 5

    2023

  6. [6]

    Video generation models as world simulators,

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators,

  7. [7]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 4, 5

    work page 2021

  8. [8]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers, 2024

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. Panda-70m: Captioning 70m videos with multiple cross-modality teachers, 2024. 4

    work page 2024

Show all 51 references
  1. [9]

    Subject-driven text-to-image generation via apprenticeship learning

    Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Rui, Xuhui Jia, Ming-Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. arXiv preprint arXiv:2304.00186, 2023. 1, 3, 4

    2023 arXiv

  2. [10]

    Anydoor: Zero-shot object-level im- age customization

    Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level im- age customization. arXiv preprint arXiv:2307.09481, 2023. 3

    2023 arXiv

  3. [11]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  4. [12]

    An image is worth one word: Personalizing text-to-image gener- ation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image gener- ation using textual inversion. In The Eleventh International Conference on Learning Representations, 2022. 1, 4

    2022

  5. [13]

    Photorealistic video generation with diffusion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Fei-Fei Li, Irfan Essa, Lu Jiang, and Jos ´e Lezama. Photorealistic video generation with diffusion models. In European Conference on Computer Vision, pages 393–411. Springer, 2025. 1

    2025

  6. [14]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. 5

    2021

  7. [15]

    Imagen video: High definition video generation with diffusion mod- els

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion mod- els. arXiv preprint arXiv:2210.02303, 2022. 1

    arXiv

  8. [16]

    Vbench: Com- prehensive benchmark suite for video generative models,

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench: Com- prehensive benchmark suite for video generative models,

  9. [17]

    Videobooth: Diffusion-based video generation with image prompts

    Yuming Jiang, Tianxing Wu, Shuai Yang, Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, and Ziwei Liu. Videobooth: Diffusion-based video generation with image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6689– 6700, 2024. 1, 6

    2024

  10. [18]

    Auto-encoding varia- tional bayes

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. arXiv preprint arXiv:1312.6114, 2013. 3

    2013 arXiv

  11. [19]

    Videopoet: A large language model for zero-shot video gen- eration

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos ´e Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vigh- nesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video gen- eration. arXiv preprint arXiv:2312.14125, 2023. 1

    2023 arXiv

  12. [20]

    Multi-concept customization of text-to-image diffusion.arXiv preprint arXiv:2212.04488,

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion.arXiv preprint arXiv:2212.04488,

    arXiv

  13. [21]

    Multi-concept customization of text-to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1931–1941, 2023. 4

    1931

  14. [22]

    Flux, 2024

    Black Forest Labs. Flux, 2024. 1

    2024

  15. [23]

    Luma dream machine, 2024

    LumaLabs. Luma dream machine, 2024. 1, 6, 12

    2024

  16. [24]

    Subject-diffusion: Open domain personalized text-to-image generation without test-time fine-tuning

    Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject-diffusion: Open domain personalized text-to-image generation without test-time fine-tuning. arXiv preprint arXiv:2307.11410, 2023. 3, 4

    2023 arXiv

  17. [25]

    Dinov2: Learning robust visual features with- out supervision, 2024

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mah- moud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michae...

    2024

  18. [26]

    Kosmos-g: Generating images in context with multimodal large language models

    Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos-g: Generating images in context with multimodal large language models. arXiv preprint arXiv:2310.02992, 2023. 3

    2023 arXiv

  19. [27]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth Interna- tional Conference on Learning Representations, 2024. 4

    2024

  20. [28]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  21. [29]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  22. [30]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Proceedings of the 38th International Conference on Machine Learning , pages 8821–8831. PMLR, 2021. 1

    2021

  23. [31]

    Grounded sam: Assembling open-world models for diverse visual tasks,

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kun- chang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks,

  24. [32]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1

    2022

  25. [33]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500...

    2023

  26. [34]

    Gen-3 alpha, 2024

    RunwayML. Gen-3 alpha, 2024. 1, 6, 12

    2024

  27. [35]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information...

    2022

  28. [36]

    Instant- booth: Personalized text-to-image generation without test- time finetuning

    Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instant- booth: Personalized text-to-image generation without test- time finetuning. arXiv preprint arXiv:2304.03411, 2023. 3, 4

    2023 arXiv

  29. [37]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. In The Eleventh International Con- ference on Learning Representations. 1

  30. [38]

    Vidu-1.5, 2024

    Shengshu Technology. Vidu-1.5, 2024. 6, 12

    2024

  31. [39]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 402–419. Springer,

    2020

  32. [40]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 3

    2023 arXiv

  33. [41]

    Lavie: High-quality video gener- ation with cascaded latent diffusion models

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video gener- ation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023. 1

    2023 arXiv

  34. [42]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. InThe Twelfth In- ternational Conference on Learning Representations , 2023. 5

    2023

  35. [43]

    Elite: Encoding visual con- cepts into textual embeddings for customized text-to-image generation

    Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual con- cepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302.13848, 2023. 4

    2023 arXiv

  36. [44]

    Dreamvideo: Composing your dream videos with customized subject and motion

    Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhi- heng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hong- ming Shan. Dreamvideo: Composing your dream videos with customized subject and motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2024

  37. [45]

    Dreamvideo-2: Zero-shot subject- driven video customization with precise motion control

    Yujie Wei, Shiwei Zhang, Hangjie Yuan, Xiang Wang, Hao- nan Qiu, Rui Zhao, Yutong Feng, Feng Liu, Zhizhong Huang, Jiaxin Ye, et al. Dreamvideo-2: Zero-shot subject- driven video customization with precise motion control. arXiv preprint arXiv:2410.13830, 2024. 3, 6

    arXiv

  38. [46]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...

    2023

  39. [47]

    Dynamicrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xin- tao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310.12190, 2023. 4, 6

    2023 arXiv

  40. [48]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 3, 6 10

    2024 arXiv

  41. [49]

    Enhanc- ing detail preservation for customized text-to-image gen- eration: A regularization-free approach

    Yufan Zhou, Ruiyi Zhang, Tong Sun, and Jinhui Xu. Enhanc- ing detail preservation for customized text-to-image gen- eration: A regularization-free approach. arXiv preprint arXiv:2305.13579, 2023. 4

    2023 arXiv

  42. [50]

    Cus- tomization assistant for text-to-image generation

    Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, and Tong Sun. Cus- tomization assistant for text-to-image generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9182–9191, 2024. 1, 3

    2024

  43. [51]

    pencil drawing drawn by a hand

    Yufan Zhou, Ruiyi Zhang, Kaizhi Zheng, Nanxuan Zhao, Jiuxiang Gu, Zichao Wang, Xin Eric Wang, and Tong Sun. Toffee: Efficient million-scale dataset construction for subject-driven text-to-image generation, 2024. 1, 3, 4, 6, 12 11 Figure 11. Comparison of models with different ...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.