Pith. sign in

REVIEW 3 major objections 6 minor 57 references

ModelGrow: Continual Text-to-Video Pre-training with Model Expansion and Language Understanding Enhancement

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that continual pre-training can grow a text-to-video model by duplicating transformer blocks and adding an LLM-conditioned cross-attention block, beating vanilla fine-tuning and LoRA on quality and semantic alignment.

desk verdict A credible engineering recipe for growing T2V models, but the headline gains are inflated by inference-time recaptioning and CFG tuning, and the forgetting claim is untested. read the letter →

arxiv 2412.18966 v1 pith:L35R3C4E submitted 2024-12-25 cs.CV cs.AIcs.LG

Zhefan Rao , Liya Ji , Yazhou Xing , Runtao Liu , Zhaoyang Liu , Jiaxin Xie , Ziqiao Peng , Yingqing He
show 1 more author
Qifeng Chen
This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords text-to-videogenerationcontinualpre-trainingmodelexpansionblockduplicationlargelanguagemodelsdiffusiontransformersemanticalignmentpromptre-captioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a pre-trained text-to-video diffusion model can be improved through continual pre-training rather than retraining from scratch, and that the two essential ingredients are structural growth and better language conditioning. It shows that duplicating transformer blocks, with zero-initialized copies that act as identity mappings, lets the model absorb new data without catastrophic forgetting. It then adds a second cross-attention block conditioned on frozen large language model embeddings, giving the model a controllable way to follow long, detailed prompts. On its benchmarks, the grown 1.4B model with LLM conditioning outscores the base model, a same-size vanilla fine-tune, and a LoRA fine-tune on both overall video quality and semantic alignment. The claim is that these gains come from the continual pre-training recipe itself, not from extra data or larger compute.

What carries the argument

The load-bearing mechanism is the zero-initialized duplicated block. Expanding the transformer by copying adjacent blocks and zeroing the last linear layer of every residual sub-block makes each new block an identity at initialization, so the loss does not spike and old capabilities are not disrupted; training then lets the new blocks specialize to new data. The second mechanism is the LLM cross-attention block: a second cross-attention module in each transformer block, initialized from the T5 cross-attention weights, gated by tanh with a zero-initialized gate, and conditioned on embeddings from a frozen LLM. Together they let the model grow in capacity and in its ability to follow long prompts while keeping the original text-conditioning path intact.

What would settle it

Run the final Expansion-1.4B-LLM model on VBench standard prompts using only the short prompt and the base CFG scale, with no LLM condition, and compare it with the base model and LoRA baseline under identical conditions; if the semantic-score gap shrinks to near zero, the claimed advantage is largely an artifact of recaptioning and guidance tuning. Alternatively, evaluate a vanilla fine-tuned model with the same short-plus-long prompt and tuned CFG; if it matches the expansion model, block duplication is not the driver.

Watch

Extended reading notes

Core claim

The central discovery is that continual pre-training of a pre-trained text-to-video transformer can be decomposed into model expansion and language-understanding enhancement. Model expansion duplicates transformer blocks and interleaves the copies with the originals, zero-initializing the new blocks so they behave as identity mappings at the start of training, which preserves previously learned behavior while adding capacity. Language understanding is enhanced by appending an LLM-conditioned cross-attention block after the existing T5 cross-attention block in every transformer block, initializing it from the T5 cross-attention weights and gating it with a zero-initialized tanh gate, and by feeding the model a concatenation of the original short prompt and an LLM-expanded long prompt. With these two mechanisms, the resulting Expansion-1.4B-LLM model improves over the base model and over vanilla and LoRA continual fine-tuning on VBench and CompBench, with the largest gains in semantic alignment.

Load-bearing premise

The headline comparison assumes all methods are evaluated under the same inference protocol, but the reported best model uses the merged short-and-long prompt and a tuned LLM guidance scale while baselines use the short prompt and the original scale.

Editorial extensions

If this is right

  • Continual pre-training can serve as a cheaper alternative to retraining a text-to-video model from scratch, since the base model is reused and only the new and expanded parameters are trained.
  • Model expansion via zero-initialized block duplication mitigates catastrophic forgetting relative to full fine-tuning at the same parameter count, because the identity initialization preserves the pretrained trajectory.
  • Adding an LLM-conditioned cross-attention block gives a controllable second text condition, whose influence can be tuned at inference through a separate classifier-free guidance scale.
  • Concatenating the original prompt with a long recaption preserves key objects from the user prompt while adding detail, a simple inference change that improves semantic fidelity.
  • The recipe transfers to other transformer-based text-to-video models in principle, since it only requires duplicating blocks and adding a cross-attention module.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A reader should not take the headline VBench gap as purely due to the training method: the final model was evaluated with the merged short-plus-long prompt and a separately tuned LLM guidance scale of 12.5, while baselines used short prompts and the original guidance, so inference recipe differences are confounded with the method.
  • The method is demonstrated on a single base architecture and dataset budget; whether block expansion transfers to other text-to-video backbones or to higher resolutions and longer videos is untested, though nothing in the mechanism obviously prevents it.
  • The LLM-conditioned cross-attention block effectively lets a frozen LLM act as a second, more semantically rich text encoder, so one testable consequence is that stronger LLM text encoders should further improve semantic alignment without retraining the video model.
  • The zero-initialized identity blocks suggest an analogy to residual learning, where each new block learns a correction on top of the pretrained stack; this could make the method more stable than naive fine-tuning at larger expansion ratios.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies continual general pre-training of a text-to-video diffusion transformer. Starting from an Open-Sora-style DiT with 0.7B parameters, ModelGrow adds (i) a model-expansion step that duplicates transformer blocks with insert stacking and zero-initialized residual branches, and (ii) a language-understanding step that appends a gated LLM cross-attention block to every transformer block, initializes it from the T5 cross-attention weights, and feeds it template-wrapped embeddings of prompts produced by LLaVA-NeXT. At inference the short prompt and the LLM long prompt are concatenated (SL), and a two-condition classifier-free guidance (Eq. 3) is used with separate scales s_t5 and s_llm. The base model is pre-trained on a mixture of Panda-70M, Vript, WebVid, and JourneyDB; all compared methods then see the same 400k-video continual corpus with the same total step budget. Evaluation uses VBench standard prompts (946 short prompts), T2V-CompBench attribute prompts (100 prompts), and an MLLM (LLaVA-v1.6-34b) semantic score, with ablations over expansion size, stacking variant, prompt strategy, and LLM-branch components.

Significance. The paper is potentially useful to the T2V community: it gives a systematic comparison of three block-expansion placements (insert/prefix/suffix) and three expansion scales (1.1B/1.4B/2.1B), proposes a gated LLM cross-attention block initialized from the T5 block, and promises open-source code and models. The controlled comparison in Table 3 (Expansion-1.4B-SL vs Expansion-1.4B-LLM-SL under the same SL prompt) supports a real but modest benefit of the LLM branch (MLLM 0.6470 to 0.6737; Image Quality 65.85 to 67.28), and the SL prompt-merging idea is well motivated and validated by Figure 5 and Table 3. The two-condition CFG in Eq. (3) is a clear, easy-to-replicate formulation. The main limitation is evidential: the headline Table 1 comparisons are not apples-to-apples, and the per-dimension regressions in Appendix D are inconsistent with the paper's own broad claims, so the significance of the reported gains cannot be judged from the current evidence.

major comments (3)
  1. [Table 1 vs Table 3; Section 4.3.2] Table 1 mixes training and architecture changes with inference-time changes, so the reported advantage of Expansion-1.4B-LLM is not attributable to the continual-pretraining method alone. For the CompBench columns of Table 1, the final model is evaluated with the merged short-plus-long prompt and s_llm=12.5 (Section 4.3.2, Figure 6, Table 3), whereas no baseline is evaluated with the SL prompt or the LLM guidance term; the MLLM gain (0.6737 vs 0.6015 for the Base model and vs 0.6438 for Expansion-1.4B) therefore includes recaptioning and CFG benefits. The paper's own matched comparison (Table 3) bounds the architectural effect at 0.6470 to 0.6737 MLLM under the SL prompt, and shows that with the short prompt alone the LLM model is worse than Expansion-1.4B (MLLM 0.6315 vs 0.6438; Image Quality 66.93 vs 66.52). For the VBench standard columns, Section 4.1 prescribes the official 946 short prompts, so the remaining protocol difference is the extra s_llm=12.5 term in Eq. (3), whose contribution is not isolated because no s_llm ablation or baseline-with-matched-guidance row is reported on VBench. Please report every method under both S and SL prompting and under matched guidance, and state the protocol per row in the table caption.
  2. [Figure 6 and Section 4.3.2] The LLM guidance scale is selected on the evaluation benchmark itself. Figure 6 sweeps s_llm from 4 to 20 on the CompBench prompts, and the chosen value s_llm=12.5 is then used for the reported Expansion-1.4B-LLM results, so the headline CompBench numbers are obtained with a hyperparameter tuned on the test set. The accompanying statement that scores improve as s_llm increases is also inconsistent with stopping at 12.5 rather than 20, and no plateau or optimum is shown. Furthermore, because the final configuration combines s_t5=7 with s_llm=12.5 in Eq. (3), its effective conditioning strength is roughly three times that of the baselines' single s_t5=7 term, so part of the VBench quality gain may be a guidance-strength effect. Please move the CFG selection to a validation split (or report the full curve with a justified operating point), and add a matched-guidance ablation for the baselines so the CFG contribution can be separated from the method's contribution.
  3. [Appendix D, Tables 5 and 6; Section 4.2] The aggregate scores hide large per-dimension regressions of the final model relative to Expansion-1.4B on the standard VBench prompts, none of which are discussed in the main text: Dynamic Degree drops from 0.5833 to 0.3056 (Table 5); Spatial Relationship drops from 0.3330 to 0.2466, Color from 0.8360 to 0.8131, Multiple Objects from 0.3438 to 0.3229, and Overall Consistency from 0.2821 to 0.2720 (Table 6). These numbers directly qualify the main-text claims (Section 4.2) that the LLM enhancement improves quality and semantic alignment, and the Dynamic Degree collapse is severe enough to affect the usefulness of the model for motion-rich content. Because this regression may be a side effect of the large s_llm=12.5 over-guidance, the paper should include motion-aware ablations (e.g., Dynamic Degree as a function of s_llm) and explicitly discuss why the semantic gains concentrate in some dimensions while other dimensions regress.
minor comments (6)
  1. [Section 3.1, Eq. (1)] The sentence "where the w denotes the guidance scale" follows Eq. (1), but w does not appear in Eq. (1) or its surrounding text; it looks like a leftover from a removed classifier-free-guidance equation and should be deleted or completed.
  2. [Section 3.3 vs Figure 8] The gating factor of the LLM cross-attention block is called lambda in the text and gamma in Figure 8; the notation should be unified, and the figure should make explicit where the tanh gate is applied.
  3. [Table 1 and Table 3 captions] The tables do not state which prompt protocol (S, L, or SL) and which CFG values are used for each row; because the paper's central comparison depends on this, the captions (or a protocol table) must specify these settings per row.
  4. [Section 1 and Section 2.3] The novelty claims "the first comprehensive study of continual pre-training in text-to-video generation" and "we are the first work that systematically explores..." (Section 1) should be qualified with "to the best of our knowledge" and checked against concurrent work on LLM-conditioned diffusion and continual T2V training, since the Related Work does not cover LLM-embedding conditioning in text-to-image diffusion beyond the gating reference to Lumina-T2X.
  5. [Section 4.2, Table 1] The motivating claim that direct fine-tuning "would lead to the model performance drop" (Section 1) is not supported by Table 1, where Vanilla-0.7B has total score 76.27 percent versus 76.28 percent for the Base model and even improves several quality dimensions; a dedicated forgetting measurement (e.g., pre/post continual-training evaluation on held-out original-distribution prompts) would make the motivation concrete.
  6. [Appendix A and overall rendering] Minor presentation issues: the insert-stacking formula in Appendix A is typeset as "P M N = x" and should read P*M/N = x; the first-page figure caption leaks duplicated prompt strings and "Click me" placeholders into the text, and "hairriding" is missing a space; and no evaluation seeds or repeated samples are reported for the 100-prompt CompBench evaluation, whose MLLM scores are likely to be noisy.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor benchmark-selection issue: the LLM guidance scale s_llm is tuned on CompBench and then CompBench scores are reported, but the VBench standard-prompt comparisons remain independent.

  1. fitted input called prediction [Sec. 4.3.2 (Figure 6) and Sec. 4.1 / Table 1]
    "We fix the guidance scale of T5 st5 to 7 and vary the guidance scale of llm sllm from 4 to 20. Figure 6 shows the quality score and semantic score reasoned by MLLM on the CompBench prompt, respectively. The larger value sllm indicates the more significant impact of the llm condition on the generated results. We can see the score has improved with the increase of sllm, showing the effectiveness of incorporating LLMs embeddings. In the paper, we set CFG T5 to 7 and CFG LLMs to 12.5 to run our final model Expansion-1.4B-LLM."

    The inference hyperparameter s_llm is selected by sweeping on the CompBench evaluation prompts (Figure 6), and the final model's best CompBench MLLM score (0.6737) is then reported in Table 1 as evidence of the LLM-enhancement method. The CompBench result is therefore not an independent out-of-sample prediction; it reflects the choice of guidance scale on that same benchmark. This is a mild form of fitted-input-as-evidence rather than a full derivation-level circularity. The VBench standard-prompt comparison in the same table uses the official 946 short prompts and is a separate, not CFG-tuned-on-that-benchmark result, so the central architectural comparison retains independent content.

full rationale

ModelGrow is an empirical continual-pretraining paper, not a theorem derivation, so most circularity patterns do not apply. The model-expansion scheme is grounded in the external LLaMA Pro block-expansion idea, the LLM cross-attention conditioning is externally motivated by Lumina-T2X's gated zero-init mechanism, and the two-condition CFG formula follows InstructPix2Pix; none of these load-bearing components is justified by a self-citation from the present authors. The paper does not invoke a uniqueness theorem, does not smuggle an ansatz through a self-citation, and does not rename a known result as a new derivation. The main concern is that s_llm=12.5 is selected on the CompBench prompts and then the CompBench scores are reported as evidence of the LLM enhancement's effectiveness; this is benchmark tuning rather than a circular derivation. In addition, the headline Table 1 comparison is not fully matched in inference protocol, since the final LLM model benefits from the SL (short-plus-long) prompt and a tuned CFG while baselines are evaluated with standard short prompts, but that is an evaluation-control issue rather than a circularity of the claimed derivation. Overall, the central claims retain independent empirical content, so the circularity score is low.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. The additional LLM cross-attention block is an architectural component assembled from existing attention and gating mechanisms, so the invented-entities ledger is empty. The main ledger entries are the benchmark-tuned CFG scale and the hand-chosen block expansion count.

free parameters (3)
  • LLM guidance scale s_llm = 12.5
    Chosen by sweeping s_llm from 4 to 20 on the CompBench evaluation (Figure 6); all reported Expansion-1.4B-LLM results use this tuned value, so part of the benchmark gain is a hand-fitted inference hyperparameter.
  • Block expansion count P = P = N (doubling blocks, 1.4B total)
    The number of duplicated blocks is an architectural choice ablated as 1.1B, 1.4B, and 2.1B; no selection criterion is given, and the 2.1B model performs best in the ablation yet is not the main model.
  • T5 guidance scale s_t5 = 7.0
    Fixed CFG scale for the T5 condition, carried over from Open-Sora defaults; it is not swept, so it mainly affects the comparison only insofar as the LLM model also uses a separate s_llm.
assumptions (4)
  • standard math Diffusion denoising objective (Eq. 1) and two-condition classifier-free guidance (Eq. 3) correctly model the training and inference procedure.
    Straightforward extension of standard CFG to two text conditions, following InstructPix2Pix; no derivation is provided, but the setup matches established practice.
  • domain assumption Zero initialization of the last linear layer in each new residual block makes the duplicated blocks identity mappings, so expansion preserves the base model's output at step zero.
    Assumes the Open-Sora transformer block residual structure; plausible but not formally proven, and it is the mechanism that makes block expansion stable.
  • domain assumption Frozen LLM text embeddings from an unspecified large language model are a useful conditioning signal that the new cross-attention block can adapt to.
    The paper shows a Llama-3-style template but never states which LLM produces the embeddings used for conditioning; the central mechanism depends on this choice.
  • domain assumption The VBench and T2V-CompBench metrics measure the general T2V ability the paper claims to improve, and the new 400k re-captioned dataset is representative of general-domain video.
    No forgetting evaluation on the original pretraining distribution is provided; the only evidence is on these benchmarks with hand-selected prompts and CFG scales.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ModelGrow: Continual Text-to-Video Pre-training with Model Expansion and Language Understanding Enhancement." pith.science (2026). https://pith.science/paper/L35R3C4E

@misc{pith2026241218966,
  author       = {Pith},
  title        = {Pith review of: ModelGrow: Continual Text-to-Video Pre-training with Model Expansion and Language Understanding Enhancement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L35R3C4E}},
  note         = {Machine review of arXiv:2412.18966}
}
read the original abstract

Text-to-video (T2V) generation has gained significant attention recently. However, the costs of training a T2V model from scratch remain persistently high, and there is considerable room for improving the generation performance, especially under limited computation resources. This work explores the continual general pre-training of text-to-video models, enabling the model to "grow" its abilities based on a pre-trained foundation, analogous to how humans acquire new knowledge based on past experiences. There is a lack of extensive study of the continual pre-training techniques in T2V generation. In this work, we take the initial step toward exploring this task systematically and propose ModelGrow. Specifically, we break this task into two key aspects: increasing model capacity and improving semantic understanding. For model capacity, we introduce several novel techniques to expand the model size, enabling it to store new knowledge and improve generation performance. For semantic understanding, we propose a method that leverages large language models as advanced text encoders, integrating them into T2V models to enhance language comprehension and guide generation results according to detailed prompts. This approach enables the model to achieve better semantic alignment, particularly in response to complex user prompts. Extensive experiments demonstrate the effectiveness of our method across various metrics. The source code and the model of ModelGrow will be publicly available.

Figures

Figures reproduced from arXiv: 2412.18966 by the authors.

Figure 1
Figure 1. We continue the pre-training of a text-to-video diffusion model with ModelGrow, which includes model expansion and language [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Simplified forward pipeline and variants of block expansion methods. (a) The vanilla pipeline is the traditional stacking of [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the pipeline cooperating with the LLMs enhancement. We modify the architecture of the transformer block by [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Qualitative results of our results compared with baselines. Our model Expansion-1.4B-LLM generates videos with higher quality [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Examples of our model with different prompts. We use [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Ablation study on classifier-free guidance (CFG). We fix [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Details of Insert Stacking. We also show the three ex [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Overview of the transformer block of the text-to-video generation model. We modify the architecture of the transformer block [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Example of training prompts recaption. tures, and expressions. Describe the lighting, atmosphere, and any notable objects to establish the mood. Furthermore, it also specifies camera details, such as angle, movement, and shot type, to convey the visual presentation. Me…
Figure 10
Figure 10. Figure 10: Qualitative results of our results compared with baselines. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Qualitative results of our results compared with baselines. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Qualitative results of our results compared with baselines. [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Qualitative results of our results compared with baselines. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 23 canonical work pages

  1. [1]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021. 5

    2021

  2. [2]

    Lumiere: A space-time dif- fusion model for video generation

    Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Her- rmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, Oliver Wang, Deqing Sun, Tali Dekel, and Inbar Mosseri. Lumiere: A space-time dif- fusion model for video generation. 2024. 3

    work page 2024

  3. [3]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. 2, 3, 5

    work page 2023

  4. [4]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023. 3

    2023

  5. [5]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. 5

    2023

  6. [6]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  7. [7]

    Videocrafter1: Open diffusion models for high-quality video generation

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023. 3

    arXiv

  8. [8]

    Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7310– 7320, 2024. 3

    2024

Show all 57 references
  1. [9]

    Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image syn- thesis

    Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image syn- thesis. In The Twelfth International Conference on Learning...

    2024

  2. [10]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IE...

    2024

  3. [11]

    Contin- ual pre-training mitigates forgetting in language and vision

    Andrea Cossu, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco, Tinne Tuytelaars, and Davide Bacciu. Contin- ual pre-training mitigates forgetting in language and vision. Neural Networks, 179:106492, 2024. 2

    2024

  4. [12]

    A continual learning survey: Defying for- getting in classification tasks

    Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ale ˇs Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying for- getting in classification tasks. IEEE transactions on pattern analysis and machine intelligence , 44(7):336...

    2021

  5. [13]

    Irc- gan: Introspective recurrent convolutional gan for text-to- video generation

    Kangle Deng, Tianyi Fei, Xin Huang, and Yuxin Peng. Irc- gan: Introspective recurrent convolutional gan for text-to- video generation. In IJCAI, pages 2216–2222, 2019. 3

    2019

  6. [14]

    Catastrophic forgetting in connectionist networks

    Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128–135, 1999. 2

    1999

  7. [15]

    Videostu- dio: Generating consistent-content and multi-scene videos

    Ting Yao Fuchen Long, Zhaofan Qiu and Tao Mei. Videostu- dio: Generating consistent-content and multi-scene videos. In ECCV, 2024. 3

    2024

  8. [16]

    Lumina-t2x: Trans- forming text into any modality, resolution, and duration via flow-based large diffusion transformers

    Peng Gao, Le Zhuo, Chris Liu, , Ruoyi Du, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Trans- forming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.05945, 2024. 5

    2024 arXiv

  9. [17]

    Preserve your own correlation: A noise prior for video diffusion models

    Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Comp...

    2023

  10. [18]

    Seed-x: Mul- timodal models with unified multi-granularity comprehen- sion and generation

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Mul- timodal models with unified multi-granularity comprehen- sion and generation. arXiv preprint arXiv:2404.14396, 2024. 3

    2024 arXiv

  11. [19]

    Animatediff: Animate your personalized text-to- image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to- image diffusion models without specific tuning. Interna- tional Conference on Learning Representations, 2024. 3

    2024

  12. [20]

    Photorealistic video generation with diffusion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Fei-Fei Li, Irfan Essa, Lu Jiang, and Jos ´e Lezama. Photorealistic video generation with diffusion models. In European Conference on Computer Vision, pages 393–411. Springer, 2025. 3

    2025

  13. [21]

    Latent video diffusion models for high-fidelity video generation with arbitrary lengths

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221, 2(3):4, 2022. 3

    2022 arXiv

  14. [22]

    Animate-a-story: Storytelling with retrieval-augmented video generation

    Yingqing He, Menghan Xia, Haoxin Chen, Xiaodong Cun, Yuan Gong, Jinbo Xing, Yong Zhang, Xintao Wang, Chao Weng, Ying Shan, et al. Animate-a-story: Storytelling with retrieval-augmented video generation. arXiv preprint arXiv:2307.06940, 2023. 3

    2023 arXiv

  15. [23]

    Llms meet multimodal genera- tion and editing: A survey.arXiv preprint arXiv:2405.19334,

    Yingqing He, Zhaoyang Liu, Jingye Chen, Zeyue Tian, Hongyu Liu, Xiaowei Chi, Runtao Liu, Ruibin Yuan, Yazhou Xing, Wenhai Wang, et al. Llms meet multimodal genera- tion and editing: A survey.arXiv preprint arXiv:2405.19334,

    arXiv

  16. [24]

    Imagen video: High definition video generation with diffusion mod- els

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion mod- els. arXiv preprint arXiv:2210.02303, 2022. 3

    arXiv

  17. [25]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022. 3

    2022

  18. [26]

    Direct2v: Large language mod- els are frame-level directors for zero-shot text-to-video gen- eration

    Susung Hong, Junyoung Seo, Heeseong Shin, Sunghwan Hong, and Seungryong Kim. Direct2v: Large language mod- els are frame-level directors for zero-shot text-to-video gen- eration. arXiv preprint arXiv:2305.14330, 2023. 2, 3

    2023 arXiv

  19. [27]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019. 2

    2019

  20. [28]

    Lora: Low- rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low- rank adaptation of large language models. In International Conference on Learning Representations, 2021. 2, 6, 7, 11

    2021

  21. [29]

    VBench: Com- prehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Com- prehensive benchmark suite for video generative models....

    2024

  22. [30]

    Simple and scalable strategies to continually pre-train large language models

    Adam Ibrahim, Benjamin Th ´erien, Kshitij Gupta, Mats L Richter, Quentin Anthony, Timoth ´ee Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. arXiv preprint arXiv:2403.08763, 2024. 2

    2024 arXiv

  23. [31]

    Continual pre-training of language models

    Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. In Proceedings of The Eleventh Inter- national Conference on Learning Representations (ICLR- 2023), 2023. 1

    2023

  24. [32]

    Open-sora-plan, 2024

    PKU-Yuan Lab and Tuzhan AI etc. Open-sora-plan, 2024. 3

    2024

  25. [33]

    Llava-next: Tack- ling multi-image, video, and 3d in large multimodal models,

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next: Tack- ling multi-image, video, and 3d in large multimodal models,

  26. [34]

    Video generation from text

    Yitong Li, Martin Min, Dinghan Shen, David Carlson, and Lawrence Carin. Video generation from text. InProceedings of the AAAI conference on artificial intelligence, 2018. 3

    2018

  27. [35]

    Llm-grounded video diffusion models

    Long Lian, Baifeng Shi, Adam Yala, Trevor Darrell, and Boyi Li. Llm-grounded video diffusion models. In The Twelfth International Conference on Learning Representa- tions, 2023. 2, 3

    2023

  28. [36]

    Latte: Latent diffusion transformer for video generation

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Zi- wei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024. 3, 4, 9

    2024 arXiv

  29. [37]

    Snap video: Scaled spatiotemporal transformers for text-to-video synthesis

    Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, et al. Snap video: Scaled spatiotemporal transformers for text-to-video synthesis. In Proceedings of the IEEE/CVF Conference...

    2024

  30. [38]

    Sync-draw: Automatic video generation using deep recurrent attentive architectures

    Gaurav Mittal, Tanya Marwah, and Vineeth N Balasubrama- nian. Sync-draw: Automatic video generation using deep recurrent attentive architectures. In Proceedings of the 25th ACM international conference on Multimedia , pages 1096– 1104, 2017. 3

    2017

  31. [39]

    Jour- neydb: A benchmark for generative image understanding,

    Junting Pan, Keqiang Sun, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Jour- neydb: A benchmark for generative image understanding,

  32. [40]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  33. [41]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  34. [42]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 2, 5

    2020

  35. [43]

    T2v-compbench: A comprehen- sive benchmark for compositional text-to-video generation

    Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. T2v-compbench: A comprehen- sive benchmark for compositional text-to-video generation. arXiv preprint arXiv:2407.14505, 2024. 2, 6

    2024 arXiv

  36. [44]

    Generative pretraining in multi- modality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multi- modality. arXiv preprint arXiv:2307.05222, 2023. 3

    2023 arXiv

  37. [45]

    Generative multimodal mod- els are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiy- ing Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal mod- els are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , ...

    2024

  38. [46]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 3

    2017

  39. [47]

    Magicvideo-v2: Multi-stage high-aesthetic video generation, 2024

    Weimin Wang, Jiawei Liu, Zhijie Lin, Jiangqiao Yan, Shuo Chen, Chetwin Low, Tuyen Hoang, Jie Wu, Jun Hao Liew, Hanshu Yan, Daquan Zhou, and Jiashi Feng. Magicvideo-v2: Multi-stage high-aesthetic video generation, 2024. 3

    2024

  40. [48]

    Trace: A comprehensive benchmark for continual learning in large language models

    Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang, Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, et al. Trace: A comprehensive benchmark for continual learning in large language models. arXiv preprint arXiv:2310.06762, 2023. 2

    arXiv

  41. [49]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 3

    2024 arXiv

  42. [50]

    Lavie: High-quality video gener- ation with cascaded latent diffusion models

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video gener- ation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023. 3

    2023 arXiv

  43. [51]

    LLaMA pro: Progressive LLaMA with block expansion

    Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, and Ping Luo. LLaMA pro: Progressive LLaMA with block expansion. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers) , pages 6518– 653...

    2024

  44. [52]

    Vript: A video is worth thousands of words, 2024

    Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video is worth thousands of words, 2024. 5

    2024

  45. [53]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 3

    2024 arXiv

  46. [54]

    Language model beats diffusion-tokenizer is key to visual generation

    Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion-tokenizer is key to visual generation. In The Twelfth International Conference on Learn...

    2023

  47. [55]

    Show-1: Marrying pixel and latent dif- fusion models for text-to-video generation

    David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent dif- fusion models for text-to-video generation. arXiv preprint arXiv:2309.15818, 2023. 3

    2023 arXiv

  48. [56]

    Open-sora: Democratizing efficient video production for all, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024. 3, 4, 5, 9

    2024

  49. [57]

    Magicvideo: Efficient video generation with latent diffusion models

    Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022. 3

    2022 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.