Pith. sign in

REVIEW 5 major objections 8 minor 1 cited by

Frame-Level Captions for Long Video Generation with Complex Multi Scenes

T0 review · 5 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Per-frame captions stop long videos from drifting off-script

desk verdict A plausible frame-level conditioning pipeline whose central claim rests on a metric computed from the same prompts that condition the model, and comparisons only to the authors' own baselines. read the letter →

arxiv 2505.20827 v1 pith:ZKRT3YJN submitted 2025-05-27 cs.CV

Guangcong Zheng , Jianlong Yuan , Bo Wang , Haoyang Huang , Guoqing Ma , Nan Duan This is my paper
classification cs.CV
keywords longvideogenerationframe-levelcaptionsmulti-scenesemanticconfusionParallelMulti-WindowDenoisingDiffusionForcingcross-attentionVBench2.0
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the main obstacle to generating long, story-like videos is not motion quality but semantic confusion: when one global prompt describes many scenes, the model mixes up which content belongs where. The proposed remedy is to give every frame its own detailed caption, let each video segment attend only to its own caption, train with Diffusion Forcing so segments can be denoised at different rates, and at inference denoise all overlapping windows in parallel instead of sequentially. On the VBench 2.0 Complex Plots and Complex Landscapes benchmarks, the method reports lower Confusion Degree, higher frame-level text-video consistency, and better aesthetic and image quality for 30-second videos than video-level prompting followed by sequential inference. The authors read these results as showing that fine-grained per-frame text guidance plus parallel denoising avoids both error accumulation and scene-blending.

What carries the argument

The argument is carried by three coupled components. The first is Frame-Level Cross-Attention: instead of letting every latent token attend to one global caption, each latent token $z_f$ attends only to its own frame-level caption embedding $c_f$ via $\mathrm{CrossAttention}(q_f, c_f) = \mathrm{Softmax}(q_f W_q (c_f W_k)^T / \sqrt{d})(c_f W_v)$, so text-video alignment is local while self-attention still supplies temporal coherence. The second is Diffusion Forcing training, which assigns different noise levels to different segments of one training sequence so the model learns to denoise at different rates and can later accept flexible inference schedules. The third is Parallel Multi-Window Denoising (PMWD), in which the target long video is split into $K$ overlapping windows of training length that are all denoised simultaneously; overlapping latents are averaged after each step, turning the strictly causal dependency of sequential generation into bidirectional information flow, with every latent still guided by its own prompt.

What would settle it

Generate the same multi-scene prompt twice with the paper's pipeline, once with the frame-level captions in their original order and once with the captions permuted; if the resulting videos are not measurably different in scene order, or if Confusion Degree computed on the permuted captions does not rise, then per-frame text is not doing the causal work the paper claims. Complementary check: have human raters rank story coherence for videos with low versus high Confusion Degree; if low-CD videos are not systematically judged more coherent, the headline metric is not measuring narrative quality.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a single global text prompt is too coarse a control signal for multi-scene long videos, and that replacing it with one detailed caption per frame, plus letting each video token attend only to its own caption, removes the semantic confusion that makes complex narratives blend together. Trained with Diffusion Forcing and generated with Parallel Multi-Window Denoising, the WanX2.1-T2V-1.3B model produces 30-second Complex Plot videos with Confusion Degree 0.1385 versus 0.2962 for video-level prompting, and frame-level text-video consistency of 0.2054 versus 0.1518; the Complex Landscape results move in the same direction. The authors interpret these numbers as direct evidence that per-frame text guidance lets a model follow a story's order and scene changes, while parallel denoising prevents the error build-up that sequential methods suffer.

Load-bearing premise

The whole comparison rests on the premise that automatically turning one global prompt into per-frame prompts preserves the story's full meaning and order, and that the Confusion Degree metric built from those same per-frame prompts is a fair measure of narrative coherence rather than a measure of how well the model reproduces its own conditioning.

Editorial extensions

If this is right

  • Long videos generated with frame-level prompts follow multi-stage story instructions better than the same model with a single global prompt, as measured by lower Confusion Degree and higher frame-level text-video consistency.
  • Parallel Multi-Window Denoising avoids the cascading error accumulation of autoregressive sliding-window inference, which shows up as less degradation in aesthetic and image quality for 30-second videos.
  • Frame-level annotation can be built without explicit shot detection, because descriptions are assigned per sampled frame and shared only when adjacent frames are nearly identical.
  • Models trained on frame-level prompts can generate as many as six distinct scenes within a 5-second clip even though the training clips typically contain only one or two scene changes, indicating the model learns general scene-transition logic.
  • Diffusion Forcing plus per-frame prompts enables first-and-last-frame conditioning without auxiliary encoders, because boundary frames can simply be given clean timesteps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the authors leave implicit is that the same frame-level annotation and attention recipe could apply to storyboard-to-video and image-to-video generation, where each keyframe already has its own caption and per-frame cross-attention should reduce scene blending.
  • The Confusion Degree could be reused as a data-curation filter: any generated long video whose frames match neighboring prompts better than their own could be automatically flagged for regeneration, making the metric useful beyond evaluation.
  • A concrete prediction that would isolate the contribution of annotation density from that of parallel inference: degrading the frame captions (coarser sampling, shorter descriptions, or a noisier captioner) should monotonically worsen Confusion Degree and frame-level consistency.
  • Because PMWD denoises the whole sequence together, its memory cost grows with video length; a practical extension would be chunked parallel windows with boundary averaging, trading a small amount of bidirectionality for much lower memory.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper proposes a frame-level captioning pipeline for long, multi-scene video generation. It introduces a scalable annotation methodology that uses Gemini Pro 2.5 to convert global prompts into per-frame (per-latent) descriptions, a Frame-Level Cross-Attention mechanism that conditions each latent token on its own text prompt, Diffusion Forcing training to handle variable noise schedules, and a Parallel Multi-Window Denoising (PMWD) inference method that denoises overlapping windows simultaneously. Experiments are conducted on the VBench 2.0 Complex Plots and Complex Landscapes categories using a fine-tuned WanX2.1-T2V-1.3B model, and the paper introduces a new Confusion Degree (CD) metric alongside video-level and frame-level text-video consistency scores. The authors report that their frame-level approach reduces CD and improves frame-level consistency compared to their own diffusion-forcing baselines with video-level prompts.

Significance. If the evaluation were sound, the frame-level annotation methodology and PMWD would be practically useful contributions to long-video generation with complex narratives, and the dataset construction pipeline could support future fine-grained text-video alignment. The paper also promises to release annotation methods and trained models, which would aid reproducibility. However, the central claims ("better at following instructions," "State-of-the-Art Performance") are not supported by the current evidence: the main metric is entangled with the method's own conditioning, no external or prior-work baselines are compared, and no independent validation of the new metric is provided. With additional experiments and external metrics, the contributions could become convincing, but as presented the evidence is insufficient.

major comments (5)
  1. [Section 5.3, Eqs. (3)-(5)] The Confusion Degree (CD) is computed using the same frame-level prompts {P_i} that serve as the conditioning signal for the proposed method during training (§4.2) and inference (§4.4, Algorithm 2). For the video-level baseline, these frame-level prompts are not provided to the model, so CD measures self-consistency between the output frames and the method's own conditioning for the frame-level system, while for the baseline it measures agreement with a hidden Gemini-generated decomposition. Additionally, the conversion system prompt in Section 3 instructs that adjacent prompts differ "as small as possible" and to "maintain similar sentence structures," which inflates STT(P_i, P_j) for nearby i and j and lowers the threshold in Eq. (4). A model that simply copies its input captions can achieve low CD without realizing the global story. No human evaluation, no external captioner, and no standard VBench 2.0 faithfulness score are reported to break this circularity, so the CD advantage in Table 1 does not establish better instruction following.
  2. [Tables 1 and 2] All experimental comparisons are against the authors' own diffusion-forcing fine-tunes (video-level vs. frame-level prompts, and different inference modes). No prior multi-scene long-video systems (e.g., Gen-L-Video, MovieDreamer, StoryDiffusion, LCT, VideoStudio) and not even the original WanX2.1-T2V-1.3B model are included as baselines. The contributions list in the introduction claims "State-of-the-Art Performance on Complex Videos," and the abstract claims the method is "better at following instructions," but these claims are unsupported without any external baseline or prior-art comparison.
  3. [Section 5.3] The Confusion Degree is a newly proposed metric, but its validity is not established: no correlation with human judgment of narrative coherence, nor with existing text-video faithfulness metrics, is reported. Since CD is the primary evidence for the paper's central claim, it cannot carry that weight alone. Moreover, the paper reports only VBench motion smoothness, aesthetic quality, and image quality, omitting the VBench 2.0 faithfulness score that is the benchmark's principal metric for the Complex Plots and Complex Landscapes categories; that omission leaves prompt adherence unmeasured by an established standard.
  4. [Section 4.4, Algorithm 2] In PMWD, every window is denoised with the same timestep t at each step, so the algorithm does not exercise the variable-noise-rate capability that Diffusion Forcing is designed to provide. As written, PMWD reduces to synchronous full-sequence denoising with overlap averaging, which is closely related to Gen-L-Video's temporal co-denoising. The claim that Diffusion Forcing "enables" PMWD is therefore not demonstrated. If the intended inference schedule assigns different timesteps per window or uses the "step-size" parameter mentioned in §4.3, this must be specified in the algorithm and reflected in the experiments; otherwise, the distinction from existing parallel-window methods is not established.
  5. [Sections 3 and 5.1] The manuscript repeatedly refers to an appendix for the full frame-level annotation prompt, the Gemini Pro 2.5 conversion details, and training hyperparameters, but no appendix is present in the submitted text. Since the dataset annotation methodology is a central contribution and the authors state their intent to share these methods, the actual prompt templates, data-processing details, and fine-tuning hyperparameters must be included in the manuscript or supplementary material to allow reproduction and verification.
minor comments (8)
  1. [Eq. (3)] The normalization of STT and STF is described only in prose; please introduce distinct notation for the normalized versions (e.g., STT_norm and STF_norm) to avoid ambiguity in Eqs. (4)-(5).
  2. [Section 5.1] Training details such as learning rate, optimizer, warmup schedule, and number of GPUs are not reported; please include them.
  3. [Tables 1 and 2] The number of prompts per category and the number of random seeds used to compute the reported means and standard deviations are not stated; please specify these to allow statistical assessment.
  4. [Section 6.2] The statement that a 5s video corresponds to 21 latents should clarify the mapping between video seconds, raw frames, and latent tokens, since earlier in Section 3 the paper describes sampling one frame every 8 frames from 8-second clips.
  5. [Figure 1] The caption mentions "ten video key frames," but the selection criterion for these key frames is not explained; please clarify.
  6. [Section 2 and Section 4.4] Since Gen-L-Video already uses overlapping windows with temporal co-denoising and averaging, the paper should explicitly state the technical differences between Gen-L-Video and the proposed PMWD.
  7. [Global] There are several typos and inconsistent terms, e.g., "Denosing" in algorithm titles, "Setttings" in the Section 6 header, and "multi-prompting" used interchangeably with "frame-level." A careful proofread is needed.
  8. [Abstract and Conclusion] The abstract claims the method is "better at following instructions" based on only two VBench 2.0 categories and without comparisons to prior work; the claim should be tempered to match the scope of the experiments.

Circularity Check

1 steps flagged · score 4.0 of 10

Confusion Degree is computed from the same frame-level prompts used as conditioning, making the reported prompt-following advantage partly an output-to-input consistency artifact.

  1. self definitional [Section 5.3, Eqs. 3-5; Section 3, 'During inference...'; Section 4.2, Eq. 1; Algorithm 2, line 11]
    "Then ˜ST T(Pi, Pj) = ST T(Pi, Pj)/ST T(Pi, Pi) and ˜ST F(Pi, Vj) = ST F(Pi, Vj)/ST F(Pi, Vi) are applied as normalization function to ensure ˜ST T(Pi, Pi) = 1 and ˜ST F(Pi, Vi) = 1. The confusion degree of a text Pi in the generated video V is defined as: CD(Pi) = X j∈{1,...,F } max(0, ˜ST F(Pi, Vj) − ˜ST T(Pi, Pj)) ... During inference, we use gemini pro 2.5 to convert a user input from short/detailed caption to a frame-level detailed caption."

    The frame-level method is trained and inferred with exactly this {P_i}: Eq. 1 restricts cross-attention of latent zf to its paired caption cf, and Algorithm 2 denoises each window with ctext[s:e] taken from the same frame-level prompts. CD therefore checks, for the proposed method, how well each generated frame matches its own conditioning plus how the text embeddings of those same prompts overlap; if Vj ≈ f(Pj), then ~STF(Pi,Vj) ≈ ~STT(Pi,Pj) and CD≈0 by construction, regardless of whether the global story is correctly realized.

full rationale

The paper contains genuinely independent contributions: frame-level cross-attention, Diffusion Forcing training, and Parallel Multi-Window Denoising are architectural and inference changes, and the reported aesthetic/image quality and motion smoothness are external VBench-style quality metrics that do not reduce to the method's own conditioning. The self-citations in the references are related-work citations, not load-bearing uniqueness arguments. However, the headline claim of superior instruction following in Tables 1 and 2 rests primarily on Confusion Degree and frame-level text-video consistency, both of which are computed from the same Gemini-generated frame-level prompts that the proposed method receives as conditioning. For the frame-level model, a low CD can be achieved by faithfully reproducing its own per-frame prompt sequence, even if that sequence misrepresents the global story; for the video-level baseline, the same metric measures agreement with hidden per-frame annotations it never saw. This is a partial circularity in the central comparison, though not a complete one, because the quality metrics and the architectural results retain independent content.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim relies on the effectiveness of automatic frame-level captioning, the transfer of Diffusion Forcing to WanX2.1, and the validity of the Confusion Degree metric. The only fitted or selected numbers are inference hyperparameters; no physical constants are involved. No new particles, forces, or external entities are proposed.

free parameters (3)
  • Diffusion Forcing step-size = not specified
    Controls the allowed difference in noise schedules between adjacent segments (Section 4.3). No ablation or chosen value is reported, yet it determines the balance between consistency and dynamics.
  • Number of frame-level prompts (NUM_PROMPTS) = 21 at inference
    The system prompt converts a 21s global prompt into 21 frame-level prompts, while training clips contain about 24 frames; this mismatch is not analyzed.
  • PMWD window count K and overlap = not specified
    Algorithm 2 requires K, Fwindow, and Foverlapped values. The paper states a fixed 21-latent window consistent with training but does not report the K or overlap settings used in Tables 1 and 2.
assumptions (4)
  • domain assumption Diffusion Forcing training from prior work is compatible with WanX2.1 DiT and yields temporal flexibility.
    Section 4.3 adopts Diffusion Forcing without architectural analysis or ablation.
  • domain assumption MLLM-generated frame-level captions are semantically accurate and temporally consistent with the source video.
    Section 3 relies on automatic annotation; no quality evaluation of the annotations is provided.
  • domain assumption The curated subset of VBench 2.0 metrics plus the new Confusion Degree metric is a fair test of long multi-scene video generation.
    Section 5.3 selects a subset of VBench metrics and introduces CD; the full benchmark or a human study is not reported.
  • domain assumption Per-frame caption conditioning does not degrade motion smoothness.
    Section 4.2 changes cross-attention while relying on self-attention for temporal coherence; motion smoothness is only weakly tested (98.53 vs 98.63).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Frame-Level Captions for Long Video Generation with Complex Multi Scenes." pith.science (2026). https://pith.science/paper/ZKRT3YJN

@misc{pith2026250520827,
  author       = {Pith},
  title        = {Pith review of: Frame-Level Captions for Long Video Generation with Complex Multi Scenes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZKRT3YJN}},
  note         = {Machine review of arXiv:2505.20827}
}
read the original abstract

Generating long videos that can show complex stories, like movie scenes from scripts, has great promise and offers much more than short clips. However, current methods that use autoregression with diffusion models often struggle because their step-by-step process naturally leads to a serious error accumulation (drift). Also, many existing ways to make long videos focus on single, continuous scenes, making them less useful for stories with many events and changes. This paper introduces a new approach to solve these problems. First, we propose a novel way to annotate datasets at the frame-level, providing detailed text guidance needed for making complex, multi-scene long videos. This detailed guidance works with a Frame-Level Attention Mechanism to make sure text and video match precisely. A key feature is that each part (frame) within these windows can be guided by its own distinct text prompt. Our training uses Diffusion Forcing to provide the model with the ability to handle time flexibly. We tested our approach on difficult VBench 2.0 benchmarks ("Complex Plots" and "Complex Landscapes") based on the WanX2.1-T2V-1.3B model. The results show our method is better at following instructions in complex, changing scenes and creates high-quality long videos. We plan to share our dataset annotation methods and trained models with the research community. Project page: https://zgctroy.github.io/frame-level-captions .

Figures

Figures reproduced from arXiv: 2505.20827 by the authors.

Figure 1
Figure 1. Illustration of Semantic Confusion and Error Accumulation in Video Generation. This figure contrasts 30s videos generated by conventional video-level prompt (right) and our proposed frame-level prompt (ten video key frames below), demonstrating improved generation quality of our method in aspects of semantic confusion and error accumulation. risk losing information, and depend heavily on good shot detection and capt… view at source ↗
Figure 2
Figure 2. Overview of the proposed frame-level training method. Frame-Level Cross-Attention links the visual data of each video segment (latent token) directly to its own specific text description. a DiT then generates video from these latents. However, when generating long videos with detailed stories and dynamic action, these models face several basic problems: • Imprecise Content Control: Using a single text description (c… view at source ↗
Figure 3
Figure 3. Three inference modes. Multiple yellow tokens represent different frame-level prompts. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Complex Plot Generation. This figure illustrates the impact of different prompting strategies [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Comparative Video Generation Outputs. This figure showcases videos generated by [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Comparative Video Generation Outputs. This figure showcases videos generated by [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Comparative Video Generation Outputs. This figure showcases videos generated by [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Illustration of Key Frame Generation for Sequences with Multiple, Rapidly Changing [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: First-and-Last-Frame-to-Video Generation Enabled by Diffusion Forcing and Our Dynamic [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LoViC: Efficient Long Video Generation with Context Compression

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LoViC uses FlexFormer, a single-query-token Q-Former with interpolated rotary positional encoding, to compress long video-text context for efficient long-video generation.

Reference graph

Works this paper leans on

47 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Lumiere: A space-time diffusion model for video generation

    Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, et al. Lumiere: A space-time diffusion model for video generation. In SIGGRAPH, pages 1–11, 2024

    work page 2024

  2. [2]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  3. [3]

    Align your latents: High-resolution video synthesis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, pages 22563–22575, 2023

    2023

  4. [4]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators. OpenAI Blog, 1:8, 2024

    2024

  5. [5]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, pages 961–970, 2015

    work page 2015

  6. [6]

    Diffusion forcing: Next-token prediction meets full-sequence diffusion

    Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 37:24081–24125, 2024

    2024

  7. [7]

    Skyreels-v2: Infinite-length film generative model

    Guibin Chen, Dixuan Lin, Jiangping Yang, Chunze Lin, Juncheng Zhu, Mingyuan Fan, Hao Zhang, Sheng Chen, Zheng Chen, Chengchen Ma, et al. Skyreels-v2: Infinite-length film generative model. arXiv preprint arXiv:2504.13074, 2025

    arXiv 2025

  8. [8]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In CVPR, pages 13320–13331, 2024

    work page 2024

Show all 47 references
  1. [9]

    Learning temporal coherence via self-supervision for gan-based video generation

    Mengyu Chu, You Xie, Jonas Mayer, Laura Leal-Taixé, and Nils Thuerey. Learning temporal coherence via self-supervision for gan-based video generation. TOG, 39(4):75–1, 2020

    2020

  2. [10]

    Factorizing text-to-video generation by explicit image conditioning

    Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Factorizing text-to-video generation by explicit image conditioning. In ECCV, pages 205–224. Springer, 2024

    2024

  3. [11]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023

    2023 arXiv

  4. [12]

    Long context tuning for video generation

    Yuwei Guo, Ceyuan Yang, Ziyan Yang, Zhibei Ma, Zhijie Lin, Zhenheng Yang, Dahua Lin, and Lu Jiang. Long context tuning for video generation. arXiv preprint arXiv:2503.10589, 2025

    2025 arXiv

  5. [13]

    Streamingt2v: Consistent, dynamic, and extendable long video generation from text

    Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tade- vosyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. arXiv preprint arXiv:2403.14773, 2024

    2024 arXiv

  6. [14]

    Autoregressive diffusion models

    Emiel Hoogeboom, Alexey A Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In International Conference on Learning Representations, 2023

    2023

  7. [15]

    Step-video-ti2v technical report: A state-of-the-art text-driven image-to-video generation model

    Haoyang Huang, Guoqing Ma, Nan Duan, Xing Chen, Changyi Wan, Ranchen Ming, Tianyu Wang, Bo Wang, Zhiying Lu, Aojie Li, et al. Step-video-ti2v technical report: A state-of-the-art text-driven image-to-video generation model. arXiv preprint arXiv:2503.11251, 2025

    2025 arXiv

  8. [16]

    Fifo-diffusion: Generating infinite videos from text without training, 2024

    Jihwan Kim, Junoh Kang, Jinyoung Choi, and Bohyung Han. Fifo-diffusion: Generating infinite videos from text without training, 2024. URL https://arxiv.org/abs/2405.11473. 19

    2024 arXiv

  9. [17]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  10. [18]

    A survey on long video generation: Challenges, methods, and prospects, 2024

    Chengxuan Li, Di Huang, Zeyu Lu, Yang Xiao, Qingqi Pei, and Lei Bai. A survey on long video generation: Challenges, methods, and prospects, 2024. URL https://arxiv.org/ abs/2403.16407

    2024 arXiv

  11. [19]

    Unified video action model, 2025

    Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified video action model, 2025. URL https://arxiv.org/abs/2503.00200

    2025 arXiv

  12. [20]

    Open-sora plan: Open-source large video generation model

    Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:2412.00131, 2024

    2024 arXiv

  13. [21]

    Videostudio: Generating consistent-content and multi-scene videos

    Fuchen Long, Zhaofan Qiu, Ting Yao, and Tao Mei. Videostudio: Generating consistent-content and multi-scene videos. In ECCV, pages 468–485. Springer, 2024

    2024

  14. [23]

    Step-video-t2v technical report: The practice, challenges, and future of video foundation model

    Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, et al. Step-video-t2v technical report: The practice, challenges, and future of video foundation model. arXiv preprint arXiv:2502.10248, 2025

    2025 arXiv

  15. [24]

    Mevg: Multi-event video generation with text-to-video models

    Gyeongrok Oh, Jaehwan Jeong, Sieun Kim, Wonmin Byeon, Jinkyu Kim, Sungwoong Kim, and Sangpil Kim. Mevg: Multi-event video generation with text-to-video models. In ECCV, pages 401–418. Springer, 2024

    2024

  16. [25]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pages 4195–4205, 2023

    2023

  17. [26]

    Mask2dit: Dual mask-based diffusion transformer for multi-scene long video generation

    Tianhao Qi, Jianlong Yuan, Wanquan Feng, Shancheng Fang, Jiawei Liu, SiYu Zhou, Qian He, Hongtao Xie, and Yongdong Zhang. Mask2dit: Dual mask-based diffusion transformer for multi-scene long video generation. arXiv preprint arXiv:2503.19881, 2025

    2025 arXiv

  18. [27]

    Freenoise: Tuning-free longer video diffusion via noise rescheduling

    Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu. Freenoise: Tuning-free longer video diffusion via noise rescheduling. ICLR, 2023

    2023

  19. [28]

    Seaweed-7b: Cost-effective training of video generation foundation model

    Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, et al. Seaweed-7b: Cost-effective training of video generation foundation model. arXiv preprint arXiv:2504.08685, 2025

    2025 arXiv

  20. [29]

    Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2

    Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In CVPR, pages 3626–3636, 2022

    2022

  21. [30]

    Lightweight, pre-trained transformers for remote sensing timeseries

    Gabriel Tseng, Ruben Cartuyvels, Ivan Zvonkov, Mirali Purohit, David Rolnick, and Hannah Kerner. Lightweight, pre-trained transformers for remote sensing timeseries. arXiv preprint arXiv:2304.14065, 2023

    2023 arXiv

  22. [31]

    Mocogan: Decomposing motion and content for video generation

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In CVPR, pages 1526–1535, 2018

    2018

  23. [32]

    Wan: Open and advanced large-scale video generative models

    Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025

    2025 arXiv

  24. [33]

    Storyanchors: Generating consistent multi-scene story frames for long-form narratives, 2025

    Bo Wang, Haoyang Huang, Zhiyin Lu, Fengyuan Liu, Guoqing Ma, Jianlong Yuan, Yuan Zhang, and Nan Duan. Storyanchors: Generating consistent multi-scene story frames for long-form narratives, 2025. URL https://arxiv.org/abs/2505.08350. 20

    2025 arXiv

  25. [34]

    Gen-l-video: Multi-text to long video generation via temporal co-denoising

    Fu-Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen-l-video: Multi-text to long video generation via temporal co-denoising. arXiv preprint arXiv:2305.18264, 2023

    2023 arXiv

  26. [35]

    Koala-36m: A large-scale video dataset improving con- sistency between fine-grained conditions and video content

    Qiuheng Wang, Yukai Shi, Jiarong Ou, Rui Chen, Ke Lin, Jiahao Wang, Boyuan Jiang, Haotian Yang, Mingwu Zheng, Xin Tao, et al. Koala-36m: A large-scale video dataset improving con- sistency between fine-grained conditions and video content. arXiv preprint arXiv:2410.08260, 2024

    arXiv

  27. [36]

    Lvbench: An extreme long video understanding benchmark, 2024

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark, 2024

    2024

  28. [37]

    Videoagent: Long-form video understanding with large language model as agent, 2024

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent, 2024. URL https://arxiv.org/ abs/2403.10517

    2024 arXiv

  29. [38]

    Vatex: A large-scale, high-quality multilingual dataset for video-and-language research

    Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In ICCV, pages 4581–4591, 2019

    2019

  30. [39]

    Imaginator: Condi- tional spatio-temporal gan for video generation

    Yaohui Wang, Piotr Bilinski, Francois Bremond, and Antitza Dantcheva. Imaginator: Condi- tional spatio-temporal gan for video generation. In WACV, pages 1160–1169, 2020

    2020

  31. [40]

    Advancing high-resolution video-language representation with large-scale video transcriptions

    Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In CVPR, pages 5036–5045, 2022

    2022

  32. [41]

    Long video diffusion generation with segmented cross-attention and content-rich video data curation

    Xin Yan, Yuxuan Cai, Qiuyue Wang, Yuan Zhou, Wenhao Huang, and Huan Yang. Long video diffusion generation with segmented cross-attention and content-rich video data curation. arXiv preprint arXiv:2412.01316, 2024

    2024 arXiv

  33. [42]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  34. [43]

    Merlot: Multimodal neural script knowledge models

    Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Neurips, 34:23634–23651, 2021

    2021

  35. [44]

    Moviedreamer: Hierarchical generation for coherent long visual sequence

    Canyu Zhao, Mingyu Liu, Wen Wang, Jianlong Yuan, Hao Chen, Bo Zhang, and Chunhua Shen. Moviedreamer: Hierarchical generation for coherent long visual sequence. arXiv preprint arXiv:2407.16655, 2024

    2024

  36. [45]

    Vbench-2.0: Advancing video generation bench- mark suite for intrinsic faithfulness, 2025

    Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, and Ziwei Liu. Vbench-2.0: Advancing video generation bench- mark suite for intrinsic faithfulness, 2025. URL https://arxiv.org/abs/2503.21755

    2025 arXiv

  37. [46]

    Videogen-of-thought: A collaborative framework for multi-shot video generation

    Mingzhe Zheng, Yongqi Xu, Haojian Huang, Xuran Ma, Yexin Liu, Wenjie Shu, Yatian Pang, Feilong Tang, Qifeng Chen, Harry Yang, et al. Videogen-of-thought: A collaborative framework for multi-shot video generation. arXiv preprint arXiv:2412.02259, 2024

    2024

  38. [47]

    Towards automatic learning of procedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, volume 32, 2018

    2018

  39. [48]

    Storydiffusion: Consistent self-attention for long-range image and video generation

    Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou. Storydiffusion: Consistent self-attention for long-range image and video generation. Neurips, 37:110315– 110340, 2024. 21

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.