Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Video Diffusion Transformers are In-Context Learners

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that pretrained video diffusion transformers harbor latent in-context learning, and that a simple concatenation-plus-LoRA recipe activates it for consistent multi-scene video generation.

desk verdict A useful empirical recipe for multi-scene video generation via concatenation and task-specific LoRA, but the title's claim about inherent in-context capability is untested because every result uses a trained LoRA. read the letter →

arxiv 2412.10783 v3 pith:3RE22VVZ submitted 2024-12-14 cs.CV

Zhengcong Fei , Di Qiu , Debang Li , Changqian Yu , Mingyuan Fan This is my paper · ORCID
classification cs.CV
keywords videodiffusiontransformersin-contextlearningmulti-scenegenerationlong-durationLoRAfine-tuningcontrollablestoryboardtext-to-video
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that modern text-to-video diffusion transformers already contain in-context learning abilities, and that a light fine-tuning step can switch them on. The recipe is to concatenate several video clips into one large clip, replace the separate captions with a single prompt that describes the whole set, and train a task-specific LoRA on roughly 100 to 200 curated examples. With that, one diffusion pass generates several related scenes at once, preserving a subject's identity and style across clips. The authors report that this yields consistent multi-scene videos longer than 30 seconds with no extra computation and no change to the underlying model architecture. A sympathetic reader would take the claim as: the bottleneck is not model capacity but the formatting of inputs and a small activation signal.

What carries the argument

The carrying mechanism is the concatenated video set: multiple clips are stacked spatially or temporally into one latent tensor, and their captions are merged into a single prompt template that opens with an overarching description and then lists per-clip prompts. The attention layers of the transformer then see all clips in one pass, so cross-clip consistency can be established by the same attention that establishes consistency inside a clip. A low-rank adapter (LoRA) of rank 32, trained on 100 to 200 curated task examples, amplifies this latent ability, and a training-free masking operation lets the model inpaint or outpaint a target clip from the others.

What would settle it

Run the same concatenated-input, joint-prompt pipeline on the same base model with no LoRA, or with a LoRA trained on unrelated videos. If the model cannot keep identities, styles, or scene relationships consistent in that condition, the claim that the pretrained model is inherently an in-context learner is not supported.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that a pretrained video diffusion transformer is already an in-context learner: when several clips from one source are concatenated and captioned together, the model can generate a coherent set of videos whose relationship is steered by the prompt, and a task-specific LoRA of rank 32 makes this reliable. The authors demonstrate this for multi-scene storyboards, portrait sets with a single identity, style transfer, and masked inpainting and outpainting. They use the CogVideoX-5b text-to-video model as the fixed backbone and train only the LoRA, so the same architecture and weights do ordinary single-video generation and multi-scene generation depending on how the input is arranged.

Load-bearing premise

The load-bearing premise is that a pretrained video diffusion transformer already has latent in-context ability; the paper never tests the pipeline without the task-specific LoRA, so the observed consistency could be learned from the 100 to 200 curated examples rather than unlocked from pretraining.

Editorial extensions

If this is right

  • Multi-scene videos longer than 30 seconds can be generated in a single diffusion pass, with no extra computational cost beyond generating one long clip.
  • Different controllable tasks, from portrait identity preservation to style transfer to inpainting and outpainting, can be handled by the same pipeline with a task-specific LoRA.
  • No modification to the video diffusion transformer's architecture or pretrained weights is needed; only a small adapter and reformatted inputs.
  • A unified prompt that describes all panels jointly leads to outputs that match the prompt and keep roles and styles consistent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the latent in-context claim is right, the same concatenation trick may transfer to other video backbones, so future models could be evaluated for multi-scene ability before any task-specific tuning.
  • The approach suggests that long-video consistency can be reframed as a set-consistency problem, which could connect to storyboard and cinematic generation systems.
  • A testable extension would remove the LoRA entirely and vary only the prompt template, or swap the LoRA for a cheap adapter trained on unrelated data, to see how much of the consistency comes from pretraining versus from the curated task data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a pipeline for enabling "in-context" generation in video diffusion transformers. Videos from one source are concatenated along spatial or temporal dimensions, captioned jointly, and a task-specific LoRA is fine-tuned on 100-200 curated examples. The authors claim that pretrained text-to-video diffusion transformers possess latent in-context capability that this procedure activates, yielding consistent multi-scene videos of over 30 seconds, identity/style preservation, and masked inpainting/outpainting, without modifying the base architecture or adding inference cost. Experiments are qualitative demonstrations with CogVideoX-5b (LoRA rank 32, 5000 steps, batch size 128, learning rate 1e-5) on multi-scene generation, portrait photography, style transfer, inpainting, and outpainting. Code, data, and weights are released.

Significance. If the central premise is established, the pipeline offers a simple, practical way to extend frozen text-to-video models to multi-scene conditional generation with modest per-task data. The public release of data, code, and weights and the use of LoRA rather than full fine-tuning are concrete strengths. However, the paper's current evidence is entirely qualitative and does not isolate whether the capability comes from pretraining or from the task-specific LoRA; the abstract's comparative claims ("better align with prompt specifications", "without additional computational overhead") are not measured. The significance is therefore conditional on the missing controls and quantitative evaluation.

major comments (3)
  1. [Section 3.2] The central premise, that pretrained video diffusion transformers already possess in-context generation capability, is never tested. Section 4.1 states that every result is produced by CogVideoX-5b plus a task-specific LoRA trained for 5000 steps on 100-200 curated examples, and all figures show outputs of this tuned model. The paper itself concedes in Section 1 that "small-scale, task-specific tuning data" is required, which undercuts the title's claim. This evidence is compatible with the LoRA learning the task from curated data rather than activating a latent capability of the base model. Please report a zero-shot baseline without any LoRA and a control LoRA trained on unrelated tasks; unless those fail, the attribution of the observed consistency to pretraining is unsupported.
  2. [Section 4.2] The results section is qualitative only. There are no metrics for prompt alignment, identity/style consistency, temporal coherence, or user preference, and no comparison against the base model without LoRA or against prior in-context or control methods such as [47] and [109]. Consequently, abstract claims that outputs "better align with prompt specifications" and that the method works "without additional computational overhead" are unsubstantiated. Please add quantitative evaluations and at least one comparison baseline; otherwise the main comparative claims cannot be assessed.
  3. [Section 4.1] The method relies on several unablated choices: LoRA rank 32, 5000 steps, learning rate 1e-5, batch size 128, and curated dataset sizes of 100-200. Since Section 3.2 argues that only "meticulously curated, high-quality text-video sets" are needed, the paper should at least ablate dataset size and LoRA rank to show the method is not brittle. Without such ablations, the claim of "minimal tuning required for activation" is not demonstrated.
minor comments (5)
  1. [Abstract and Section 3.2] There are typos: "spacial" should be "spatial", and "tempera" should be "temporal" (or "spatial" as intended).
  2. [Section 4.1] The statement "jointly caption multi-scene video clips from one source" is not formalized; specify how Qwen2-VL captions are merged into the prompt template and provide one full example of the template.
  3. [Figures 1-3] The figure captions do not identify which task-specific LoRA is used for each panel; label each result with its LoRA and training set.
  4. [Section 2.3] The related work section mentions [47] as inspiration but does not compare with it; given how close the pipelines are, at least a paragraph discussing the differences is needed.
  5. [Abstract and Section 5] The abstract promises that "data, code, and model weights are publicly available" while the main text says only "code and model weights"; align the statements.

Circularity Check

1 steps flagged · score 6.0 of 10

The paper's central attribution of in-context capability to the pretrained video DiT is untested: every demonstration uses a task-specific LoRA fit to 100–200 curated examples, so the observed multi-scene consistency is partly a fitted result, not an emergent property of the base model.

  1. fitted input called prediction [Section 3.2 (In-Context Generation), with Section 4.1 (Implementation Details) and Figure 1 caption]
    "We begin with the premise that through large-scale video pretraining, video diffusion transformer inherently understand the cross-frame video representation [48], i.e., existing text-to-video models are able to possess some level of in-context generation capability across a range of tasks, albeit with varying quality."

    Section 4.1 fits a task-specific LoRA on roughly 100 to 200 curated examples ('For each task, we curate a dataset of approximately 100 to 200 high-quality video sequences... with 5000 optimization steps... a LoRA rank of 32'), and every qualitative result in Figures 1–3 is produced with this tuned model. Figure 1's caption itself states the four sub-videos are 'concurrently generated within a single diffusion process that are tuned specifically.' Thus the demonstrated multi-scene consistency, identity preservation, and style matching are, by construction, outputs of a model trained on exactly that task family; they do not test the Section 3.2 premise that the pretrained video DiT already possesses the capability.

full rationale

The paper's method is simple: concatenate videos, jointly caption them, and LoRA-tune a video DiT on 100–200 curated examples. The circularity is not in the pipeline itself but in the attribution of the observed behavior to the pretrained model's latent in-context capability. The premise in Section 3.2 is asserted, never tested on the unmodified base model or with a control LoRA, and all figures show outputs of a task-specifically tuned model. This fits the 'fitted input called prediction' pattern: the model is trained to produce consistent multi-scene outputs, and those same outputs are then offered as evidence that the base model 'is an in-context learner.' There is no load-bearing self-citation chain; references to prior in-context LoRA work are external and not used to force a conclusion. However, because the central claim reduces, in part, to the effect of the task-specific fine-tuning, the paper receives a moderate-to-high circularity score of 6.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper's contributions rest on a small set of design choices, LoRA hyperparameters, curated data sizes, and the unverified premise that pretraining already gives the model in-context ability. No new physical or formal entities are introduced.

free parameters (6)
  • LoRA rank = 32
    Chosen adapter capacity; no sweep or sensitivity analysis is reported.
  • Learning rate = 1e-5
    Fixed for all tasks in Section 4.1; no ablation.
  • Training steps = 5000
    Fixed number of optimization steps in Section 4.1.
  • Batch size = 128
    Fixed batch size in Section 4.1; no ablation.
  • Curated dataset size per task = 100 to 200 video sequences
    Selection and size of training data are manual choices; the curation is the main task-specific input.
  • Number of concatenated sub-videos = four in all shown examples
    The prompt template and concatenation dimension are design choices; no ablation across clip counts or spatial versus temporal concatenation.
assumptions (4)
  • domain assumption Pre-trained video diffusion transformers have latent in-context generation capability for related scenes.
    Stated as the starting premise in Section 3.2 and supported only by a citation to [48]; if false, the title claim collapses.
  • domain assumption Concatenating clips and using one joint prompt is sufficient to expose cross-clip attention without architecture changes.
    Core recipe from Section 3.2; no experiment isolates whether concatenation or the joint prompt does the work.
  • domain assumption Task-specific LoRA tuning on 100 to 200 curated examples preserves pretrained knowledge while activating in-context behavior.
    Assumed throughout Section 4.1; no evaluation of forgetting or capacity trade-offs.
  • domain assumption Qwen2-VL generated captions are accurate enough to serve as training supervision.
    Used for all training captions in Section 4.1 without verification of caption correctness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Video Diffusion Transformers are In-Context Learners." pith.science (2026). https://pith.science/paper/3RE22VVZ

@misc{pith2026241210783,
  author       = {Pith},
  title        = {Pith review of: Video Diffusion Transformers are In-Context Learners},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3RE22VVZ}},
  note         = {Machine review of arXiv:2412.10783}
}
abstract

This paper investigates a solution for enabling in-context capabilities of video diffusion transformers, with minimal tuning required for activation. Specifically, we propose a simple pipeline to leverage in-context generation: ($\textbf{i}$) concatenate videos along spacial or time dimension, ($\textbf{ii}$) jointly caption multi-scene video clips from one source, and ($\textbf{iii}$) apply task-specific fine-tuning using carefully curated small datasets. Through a series of diverse controllable tasks, we demonstrate qualitatively that existing advanced text-to-video models can effectively perform in-context generation. Notably, it allows for the creation of consistent multi-scene videos exceeding 30 seconds in duration, without additional computational overhead. Importantly, this method requires no modifications to the original models, results in high-fidelity video outputs that better align with prompt specifications and maintain role consistency. Our framework presents a valuable tool for the research community and offers critical insights for advancing product-level controllable video generation systems. The data, code, and model weights are publicly available at: https://github.com/feizc/Video-In-Context.

Figures

Figures reproduced from arXiv: 2412.10783 by the authors.

Figure 1
Figure 1. Examples of in-context generalist for multi-scene video tasks. Four sub-videos are concurrently generated within a single diffusion process that are tuned specifically. A carefully designed prompt template, incorporating distinct scenes, is employed to ensure consistent portrayal and seamless integration of scenes in the generated video sets. 2 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Examples of in-context generalist for portrait photograph and style transfer tasks. Four sub-videos are generated simultaneously within a single diffusion process being specifically tuned for the desired outcome. Consistent subject identities are preserved across all sub-videos within each set, as demonstrated in the accompanying figure. 2.3 Controllable Video Generation As video generation techniques advanced, earl… view at source ↗
Figure 3
Figure 3. Examples of in-context generalist applied to inpainting and outpainting tasks. A sub-video, enclosed within a red box, is generated based on remaining video clips using a masking operation. This process ensures a consistent style and subjective maintained across all videos within each set. 5 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer

    cs.CV 2025-11 conditional novelty 6.0 of 10

    Expressing all RoPE positions on the query's grid ('one attention, one scale') plus a small boundary content-exchange step restores mixed-resolution diffusion generation that naive position interpolation destroys.

  2. Ingredients: Blending Custom Photos with Video Diffusion Transformers

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Ingredients adds a mask-supervised identity router to a video diffusion transformer, enabling multi-person videos from a few reference photos without per-identity fine-tuning.

Reference graph

Works this paper leans on

111 extracted references · 3 canonical work pages · cited by 2 Pith papers

  1. [47]

    In-context lora for diffusion transformers

    Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Huanzhang Dou, Chen Liang, Yutong Feng, Yu Liu, and Jingren Zhou. In-context lora for diffusion transformers. arXiv preprint arXiv:2410.23775, 2024

    arXiv 2024

  2. [109]

    Video in-context learning

    Wentao Zhang, Junliang Guo, Tianyu He, Li Zhao, Linli Xu, and Jiang Bian. Video in-context learning. arXiv preprint arXiv:2407.07356, 2024

    arXiv 2024

  3. [1]

    All are worth words: A vit backbone for diffusion models

    Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22669–22679, 2023

    2023

  4. [2]

    Latentwarp: Consistent diffusion latents for zero-shot video-to-video translation

    Yuxiang Bao, Di Qiu, Guoliang Kang, Baochang Zhang, Bo Jin, Kaiye Wang, and Pengfei Yan. Latentwarp: Consistent diffusion latents for zero-shot video-to-video translation. arXiv preprint arXiv:2311.00353, 2023

    arXiv 2023

  5. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  6. [4]

    Align your latents: High-resolution video synthesis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023

    2023

  7. [5]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  8. [6]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020

    2020

Show all 111 references
  1. [7]

    Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023

    arXiv

  2. [8]

    Seine: Short-to-long video diffusion model for generative transition and prediction

    Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. In The Twelfth International Conference on Learning Representations, 2023

    2023

  3. [9]

    Adversarial video generation on complex datasets

    Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets. arXiv preprint arXiv:1907.06571, 2019

    1907 arXiv

  4. [10]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022

    2022 arXiv

  5. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey DOSOVITSKIY . An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 7

    2010 arXiv

  6. [12]

    Scaling rectified flow transform- ers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In Forty-first International Conference on Machin...

    2024

  7. [13]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021

    2021

  8. [14]

    Stable audio open

    Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. arXiv preprint arXiv:2407.14358, 2024

    2024 arXiv

  9. [15]

    Motioncharacter: Identity- preserving and motion controllable human video generation

    Haopeng Fang, Di Qiu, Binjie Mao, Pengfei Yan, and He Tang. Motioncharacter: Identity- preserving and motion controllable human video generation. arXiv preprint arXiv:2411.18281, 2024

    2024

  10. [16]

    Fast image caption generation with position alignment

    Zheng-cong Fei. Fast image caption generation with position alignment. arXiv preprint arXiv:1912.06365, 2019

    1912 arXiv

  11. [17]

    Partially non-autoregressive image captioning

    Zhengcong Fei. Partially non-autoregressive image captioning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1309–1316, 2021

    2021

  12. [18]

    A-jepa: Joint-embedding predictive architecture can listen

    Zhengcong Fei, Mingyuan Fan, and Junshi Huang. A-jepa: Joint-embedding predictive architecture can listen. arXiv preprint arXiv:2311.15830, 2023

    2023 arXiv

  13. [19]

    Gradient-free textual inversion

    Zhengcong Fei, Mingyuan Fan, and Junshi Huang. Gradient-free textual inversion. In Proceedings of the 31st ACM International Conference on Multimedia , pages 1364–1373, 2023

    2023

  14. [20]

    Music consistency models

    Zhengcong Fei, Mingyuan Fan, and Junshi Huang. Music consistency models. arXiv preprint arXiv:2404.13358, 2024

    2024 arXiv

  15. [21]

    Flux that plays music

    Zhengcong Fei, Mingyuan Fan, Changqian Yu, and Junshi Huang. Flux that plays music. arXiv preprint arXiv:2409.00587, 2024

    2024 arXiv

  16. [22]

    Scalable diffusion models with state space backbone

    Zhengcong Fei, Mingyuan Fan, Changqian Yu, and Junshi Huang. Scalable diffusion models with state space backbone. arXiv preprint arXiv:2402.05608, 2024

    2024 arXiv

  17. [23]

    Diffusion-rwkv: Scaling rwkv-like architectures for diffusion models

    Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, and Junshi Huang. Diffusion-rwkv: Scaling rwkv-like architectures for diffusion models. arXiv preprint arXiv:2404.04478, 2024

    2024 arXiv

  18. [24]

    Scaling diffusion transformers to 16 billion parameters

    Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, and Junshi Huang. Scaling diffusion transformers to 16 billion parameters. arXiv preprint arXiv:2407.11633, 2024

    2024 arXiv

  19. [25]

    Dimba: Transformer-mamba diffusion models

    Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, Youqiang Zhang, and Junshi Huang. Dimba: Transformer-mamba diffusion models. arXiv preprint arXiv:2406.01159, 2024

    2024 arXiv

  20. [26]

    Progressive text-to-image genera- tion

    Zhengcong Fei, Mingyuan Fan, Li Zhu, and Junshi Huang. Progressive text-to-image genera- tion. arXiv preprint arXiv:2210.02291, 2022

    2022 arXiv

  21. [27]

    Masked auto-encoders meet generative adversarial networks and beyond

    Zhengcong Fei, Mingyuan Fan, Li Zhu, Junshi Huang, Xiaoming Wei, and Xiaolin Wei. Masked auto-encoders meet generative adversarial networks and beyond. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24449–24459, 2023

    2023

  22. [28]

    Ingredients: Blending custom photos with video diffusion transformers

    Zhengcong Fei, Debang Li, Di Qiu, Changqian Yu, and Mingyuan Fan. Ingredients: Blending custom photos with video diffusion transformers. arXiv preprint arXiv:2501.01790, 2025

    2025 arXiv

  23. [29]

    Deecap: Dynamic early exiting for efficient image captioning

    Zhengcong Fei, Xu Yan, Shuhui Wang, and Qi Tian. Deecap: Dynamic early exiting for efficient image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12216–12226, 2022

    2022

  24. [30]

    I2vcontrol-camera: Precise video camera control with adjustable motion strength

    Wanquan Feng, Jiawei Liu, Pengqi Tu, Tianhao Qi, Mingzhen Sun, Tianxiang Ma, Songtao Zhao, Siyu Zhou, and Qian He. I2vcontrol-camera: Precise video camera control with adjustable motion strength. arXiv preprint arXiv:2411.06525, 2024

    2024 arXiv

  25. [31]

    An image is worth one word: Personalizing text-to-image generation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022

    2022 arXiv

  26. [32]

    Lumina-t2x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers

    Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.05945, 2024. 8

    2024 arXiv

  27. [33]

    The unreasonable effectiveness of few-shot learning for machine translation

    Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Melvin Johnson, and Orhan Firat. The unreasonable effectiveness of few-shot learning for machine translation. In International Conference on Machine Learning, pages 10867–10878. PMLR, 2023

    2023

  28. [35]

    Emu video: Factorizing text-to-video generation by explicit image conditioning

    Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709, 2023

    2023 arXiv

  29. [36]

    Vector quantized diffusion model for text-to-image synthesis

    Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10696–10706, 2022

    2022

  30. [37]

    Cameractrl: Enabling camera control for text-to-video generation

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024

    2024 arXiv

  31. [38]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022

    2022

  32. [39]

    Latent video diffusion models for high-fidelity video generation with arbitrary lengths

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221, 2022

    2022 arXiv

  33. [40]

    Streamingt2v: Consistent, dynamic, and extendable long video generation from text

    Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tade- vosyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. arXiv preprint arXiv:2403.14773, 2024

    2024 arXiv

  34. [41]

    Imagen video: High definition video generation with diffusion models

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022

    arXiv

  35. [42]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  36. [43]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022

    2022 arXiv

  37. [44]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  38. [45]

    Depthcrafter: Generating consistent long depth sequences for open-world videos

    Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-world videos. arXiv preprint arXiv:2409.02095, 2024

    2024 arXiv

  39. [46]

    Videocontrolnet: A motion-guided video-to-video translation framework by using diffusion model with controlnet

    Zhihao Hu and Dong Xu. Videocontrolnet: A motion-guided video-to-video translation framework by using diffusion model with controlnet. arXiv preprint arXiv:2307.14073, 2023

    2023 arXiv

  40. [48]

    The platonic representation hypothesis

    Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024

    2024 arXiv

  41. [49]

    Panoptic segmentation

    Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9404–9413, 2019. 9

    2019

  42. [50]

    Videopoet: A large language model for zero-shot video generation

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023

    2023 arXiv

  43. [51]

    Controlnet ++ : Improving conditional controls with efficient consistency feedback

    Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, and Chen Chen. Controlnet ++ : Improving conditional controls with efficient consistency feedback. In European Conference on Computer Vision, pages 129–147. Springer, 2025

    2025

  44. [52]

    Few-shot in-context learning for knowledge base question answering

    Tianle Li, Xueguang Ma, Alex Zhuang, Yu Gu, Yu Su, and Wenhu Chen. Few-shot in-context learning for knowledge base question answering. arXiv preprint arXiv:2305.01750, 2023

    2023 arXiv

  45. [53]

    Mardini: Masked autoregressive diffusion for video generation at scale

    Haozhe Liu, Shikun Liu, Zijian Zhou, Mengmeng Xu, Yanping Xie, Xiao Han, Juan C Pérez, Ding Liu, Kumara Kahatapitiya, Menglin Jia, et al. Mardini: Masked autoregressive diffusion for video generation at scale. arXiv preprint arXiv:2410.20280, 2024

    arXiv

  46. [54]

    Vit-tts: visual text-to-speech with scalable diffusion transformer

    Huadai Liu, Rongjie Huang, Xuan Lin, Wenqiang Xu, Maozong Zheng, Hong Chen, Jinzheng He, and Zhou Zhao. Vit-tts: visual text-to-speech with scalable diffusion transformer. arXiv preprint arXiv:2305.12708, 2023

    2023 arXiv

  47. [55]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021

    2021

  48. [56]

    Video swin transformer

    Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022

    2022

  49. [57]

    Vdt: General-purpose video diffusion transformers via mask modeling

    Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding. Vdt: General-purpose video diffusion transformers via mask modeling. arXiv preprint arXiv:2305.13311, 2023

    2023 arXiv

  50. [58]

    Latte: Latent diffusion transformer for video generation

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024

    2024 arXiv

  51. [59]

    Adaptive machine translation with large language models

    Yasmin Moslem, Rejwanul Haque, John D Kelleher, and Andy Way. Adaptive machine translation with large language models. arXiv preprint arXiv:2301.13294, 2023

    2023 arXiv

  52. [60]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 4296–4304, 2024

    2024

  53. [61]

    Mofa-video: Controllable image animation via generative motion field adaptions in frozen image-to-video diffusion model

    Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. Mofa-video: Controllable image animation via generative motion field adaptions in frozen image-to-video diffusion model. arXiv preprint arXiv:2405.20222, 2024

    2024 arXiv

  54. [63]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023

    2023

  55. [64]

    Con- trolnext: Powerful and efficient control for image and video generation

    Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Ming-Chang Yang, and Jiaya Jia. Con- trolnext: Powerful and efficient control for image and video generation. arXiv preprint arXiv:2408.06070, 2024

    2024 arXiv

  56. [65]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720, 2024

    2024 arXiv

  57. [66]

    In-context learning with iterative demonstration selection

    Chengwei Qin, Aston Zhang, Chen Chen, Anirudh Dagar, and Wenming Ye. In-context learning with iterative demonstration selection. arXiv preprint arXiv:2310.09881, 2023

    2023 arXiv

  58. [67]

    Skyreels-a1: Expressive portrait animation in video diffusion transformers

    Di Qiu, Zhengcong Fei, Rui Wang, Jialin Bai, Changqian Yu, Mingyuan Fan, Guibin Chen, and Xiang Wen. Skyreels-a1: Expressive portrait animation in video diffusion transformers. arXiv preprint arXiv:2502.10841, 2025. 10

    2025 arXiv

  59. [68]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021

    2021

  60. [69]

    Improving language understanding with unsupervised learning

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding with unsupervised learning. Technical report, OpenAI, 2018

    2018

  61. [70]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  62. [71]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019

    2019

  63. [72]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  64. [73]

    Hierarchical text-conditional image generation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022

    2022 arXiv

  65. [74]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022

    2022

  66. [75]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part ...

    2015

  67. [76]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 225...

    2023

  68. [77]

    Palette: Image-to-image diffusion models

    Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–10, 2022

    2022

  69. [78]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information...

    2022

  70. [79]

    Temporal generative adversarial nets with singular value clipping

    Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In Proceedings of the IEEE international conference on computer vision, pages 2830–2839, 2017

    2017

  71. [80]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  72. [81]

    Segmenter: Transformer for semantic segmentation

    Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, 2021

    2021

  73. [82]

    Video-infinity: Distributed long video generation

    Zhenxiong Tan, Xingyi Yang, Songhua Liu, and Xinchao Wang. Video-infinity: Distributed long video generation. arXiv preprint arXiv:2406.16260, 2024

    2024 arXiv

  74. [83]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021

    2021

  75. [84]

    Mocogan: Decomposing motion and content for video generation

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526–1535, 2018

    2018

  76. [85]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017

    2017

  77. [86]

    Phenaki: Variable length video generation from open domain textual descriptions

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Lear...

    2022

  78. [87]

    Generating videos with scene dynamics

    Carl V ondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. Advances in neural information processing systems, 29, 2016

    2016

  79. [88]

    Gen-l-video: Multi-text to long video generation via temporal co-denoising

    Fu-Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen-l-video: Multi-text to long video generation via temporal co-denoising. arXiv preprint arXiv:2305.18264, 2023

    2023 arXiv

  80. [89]

    Boximator: Generating rich and controllable motions for video synthesis

    Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis. arXiv preprint arXiv:2402.01566, 2024

    2024 arXiv

  81. [90]

    Label words are anchors: An information flow perspective for understanding in-context learning

    Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. arXiv preprint arXiv:2305.14160, 2023

    2023 arXiv

  82. [91]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021

    2021

  83. [92]

    Pvt v2: Improved baselines with pyramid vision transformer

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 2022

    2022

  84. [93]

    Motionctrl: A unified and flexible motion controller for video generation

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. arXiv preprint arXiv:2312.03641, 2023

    2023 arXiv

  85. [94]

    Draganything: Motion control for anything using entity representation

    Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for anything using entity representation. In European Conference on Computer Vision, pages 331–348. Springer, 2025

    2025

  86. [95]

    Segformer: Simple and efficient design for semantic segmentation with transformers.Advances in Neural Information Processing Systems, 34:12077–12090, 2021

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers.Advances in Neural Information Processing Systems, 34:12077–12090, 2021

    2021

  87. [96]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  88. [97]

    Camco: Camera-controllable 3d-consistent image-to-video generation

    Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vah- dat. Camco: Camera-controllable 3d-consistent image-to-video generation. arXiv preprint arXiv:2406.02509, 2024

    2024 arXiv

  89. [98]

    Videogpt: Video generation using vq-vae and transformers

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157, 2021

    2021 arXiv

  90. [99]

    Direct-a-video: Customized video generation with user-directed camera movement and object motion

    Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. arXiv preprint arXiv:2402.03162, 2024

    2024 arXiv

  91. [100]

    Rerender a video: Zero-shot text-guided video-to-video translation

    Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation. In SIGGRAPH Asia 2023 Conference Papers, pages 1–11, 2023

    2023

  92. [101]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  93. [102]

    Space-time diffusion features for zero-shot text-driven motion transfer

    Danah Yatim, Rafail Fridman, Omer Bar-Tal, Yoni Kasten, and Tali Dekel. Space-time diffusion features for zero-shot text-driven motion transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8466–8476, 2024

    2024

  94. [103]

    Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory

    Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.08089, 2023

    2023 arXiv

  95. [104]

    Nuwa-xl: Diffusion over diffusion for extremely long video generation

    Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, et al. Nuwa-xl: Diffusion over diffusion for extremely long video generation. arXiv preprint arXiv:2303.12346, 2023. 12

    2023 arXiv

  96. [105]

    Video probabilistic diffusion models in projected latent space

    Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18456–18466, 2023

    2023

  97. [106]

    Generating videos with dynamics-aware implicit generative adversarial networks

    Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. arXiv preprint arXiv:2202.10571, 2022

    2022 arXiv

  98. [107]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023

    2023 arXiv

  99. [108]

    Colorful image colorization

    Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pages 649–666. Springer, 2016

    2016

  100. [110]

    Controlvideo: Training-free controllable text-to-video generation

    Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305.13077, 2023

    2023 arXiv

  101. [111]

    Moviedreamer: Hierarchical generation for coherent long visual sequence

    Canyu Zhao, Mingyu Liu, Wen Wang, Jianlong Yuan, Hao Chen, Bo Zhang, and Chunhua Shen. Moviedreamer: Hierarchical generation for coherent long visual sequence. arXiv preprint arXiv:2407.16655, 2024

    2024

  102. [112]

    Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers

    Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In CVPR, 2021

    2021

  103. [113]

    Storydiffu- sion: Consistent self-attention for long-range image and video generation

    Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou. Storydiffu- sion: Consistent self-attention for long-range image and video generation. arXiv preprint arXiv:2405.01434, 2024. 13

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.