Pith. sign in

REVIEW 5 major objections 5 minor 8 cited by

Goku: Flow Based Video Generative Foundation Models

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Goku claims a single flow transformer can lead both image and video generation benchmarks.

desk verdict A serious industrial recipe for joint image-video generation with rectified flow, but the benchmark claims are under-specified and need protocol details or artifacts before they can be taken at face value. read the letter →

arxiv 2502.04896 v2 pith:VO5PWDYV submitted 2025-02-07 cs.CV

Shoufa Chen , Chongjian Ge , Yuqi Zhang , Yida Zhang , Fengda Zhu , Hao Yang , Hongxiang Hao , Hui Wu
show 14 more authors
Zhichao Lai Yifei Hu Ting-Che Lin Shilong Zhang Fu Li Chuan Li Xing Wang Yanghua Peng Peize Sun Ping Luo Yi Jiang Zehuan Yuan Bingyue Peng Xiaobing Liu
This is my paper · ORCID
classification cs.CV
keywords rectifiedflowtext-to-videogenerationtext-to-imagejointimage-videomodelvideoVAEtransformerVBenchGenEval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single family of rectified-flow transformer models, trained jointly on images and videos, can set leading benchmark scores in both text-to-image and text-to-video generation. The authors report 0.76 on GenEval, 83.65 on DPG-Bench, and 84.85 on VBench, the last placing their text-to-video model at the top of the leaderboard as of January 25, 2025. If these results hold under comparable evaluation protocols, Goku would be evidence that image and video generation can share one latent space, one architecture, and one training run rather than needing separate specialized models. The paper's contribution is largely an engineering recipe: a data curation pipeline, a joint 3D VAE, multi-stage training, and infrastructure for long-sequence training.

What carries the argument

The carrying mechanism is rectified flow over a shared image-video latent space. A 3D joint VAE compresses video by $8 \times 8 \times 4$ and images by $8 \times 8$; the transformer sees packed image and video tokens with full attention, positional information from 3D RoPE, and RMSNorm on query and key vectors to stabilize training. The flow objective is to predict the velocity $v_t = dx_t / dt$ along the linear interpolation, and multi-stage training lets the network first learn text-to-image semantics before adding temporal motion and then specializing per modality.

What would settle it

Run the released Goku-T2V checkpoint on VBench with the same video count, resolution, duration, sample steps, sampler, and classifier-free-guidance scale used for the strongest baselines, and rerun GenEval with every model using identical prompts (original and rewritten); if the margins disappear or reverse under a common protocol, the state-of-the-art claim fails. Checking whether any VBench or GenEval prompts appear in the described 160M image-text and 36M video-text training corpora would also settle contamination concerns.

Watch

Extended reading notes

Core claim

Goku is a joint image-and-video generative model built on rectified flow: training samples interpolate linearly between noise and data, $x_t = t x_1 + (1-t) x_0$, and the transformer predicts the velocity field. Images and videos are encoded by a shared 3D VAE into one latent space, packed together in a full-attention transformer with 3D RoPE and Q-K normalization, and trained in three stages: text-to-image pretraining, joint image-video learning at increasing resolution, then modality-specific finetuning. On benchmarks, Goku-T2I reports 0.76 on GenEval (with ChatGPT-4o rewritten prompts) and 83.65 on DPG-Bench, and Goku-T2V reports 84.85 on VBench, which the paper states was the top leaderboard position as of 2025-01-25. The paper also claims rectified flow converges faster than DDPM in a controlled ImageNet-1K pilot, and that joint training improves video frame quality compared with video-only finetuning.

Load-bearing premise

The load-bearing premise is that the reported benchmark scores were produced under evaluation protocols comparable to those of the baselines and without training-data overlap with benchmark prompts, since the paper does not specify the VBench generation configuration and its headline GenEval score uses ChatGPT-4o prompt rewriting that was not applied to most baselines.

Editorial extensions

If this is right

  • A single 8B transformer, trained on roughly 160M image-text and 36M video-text pairs, is claimed to outperform specialized commercial video models on VBench's overall score.
  • Joint image-video training is claimed to improve video frame photorealism relative to fine-tuning from the same image-pretrained weights on video only.
  • Rectified flow reaches a given FID-50K on ImageNet-256 with fewer training steps than DDPM in the paper's 1B pilot.
  • Captions that include a motion score give users a way to steer the amount of motion in generated videos.
  • Scaling from 2B to 8B parameters is claimed to reduce structural distortions such as malformed limbs and wheels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors' protocol for VBench is not stated; if their generation setup differs from that used for the commercial baselines, the leaderboard gap could shrink or reverse under a common protocol.
  • The GenEval 0.76 depends on rewriting prompts with ChatGPT-4o; applying rewriting uniformly to all baselines, or evaluating Goku without rewriting, would give a fairer comparison than the table's mixed protocol.
  • A testable extension suggested by the paper is training the same architecture with image and video data fully mixed from the start rather than staged, which would test whether the staged curriculum is load-bearing or merely convenient.
  • Beyond the paper, one could probe whether motion-score conditioning transfers to user-controlled motion strength in video editing tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents Goku, a family of rectified-flow Transformer models for joint text-to-image and text-to-video generation, with 2B and 8B parameter variants. It describes a multi-stage data curation pipeline (filtering, captioning, balancing), a joint image-video VAE, full-attention transformer architecture with 3D RoPE and QK normalization, and a large-scale training infrastructure. The central empirical claims are that Goku-T2I achieves 0.76 on GenEval and 83.65 on DPG-Bench, and that Goku-T2V achieves 84.85 on VBench, placing it first on the VBench leaderboard as of 2025-01-25. The paper also reports a zero-shot UCF-101 FVD of 217.24 at 128x128 resolution.

Significance. If the stated results are reproducible under comparable protocols, Goku would be a strong joint image-video generation system and a useful reference point for scaling rectified-flow transformers. The paper's strengths include a detailed account of the data curation pipeline, the joint VAE and full-attention design, the multi-stage training recipe, and the infrastructure engineering (sequence parallelism, fine-grained activation checkpointing, fault tolerance, ByteCheckpoint). These engineering contributions are potentially valuable to the community. However, the headline state-of-the-art claims rest on benchmark scores whose measurement protocols are incompletely specified, and some of the comparisons are not apples-to-apples (prompt rewriting, resolution differences, and possibly evaluation-prompt overlap with training data). The paper would be acceptable only after these evaluation issues are resolved.

major comments (5)
  1. [Appendix A, Table 7] The VBench evaluation protocol is not specified. Appendix A's VBench paragraph only describes the benchmark suite; it omits the generation resolution, duration, frame rate, number of samples per prompt, sampler, sampling steps, classifier-free guidance, random seeds, and whether prompts were rewritten before generation. Since Goku's margin over CausVid is only 0.58 points (84.85 vs. 84.27), the top-leaderboard claim is sensitive to protocol differences. Please provide the full VBench configuration, including prompt handling and the exact number of generated videos per dimension.
  2. [Table 5, Section 5.1] The GenEval comparison is not apples-to-apples. The headline 0.76 score is obtained after rewriting GenEval prompts with ChatGPT-4o, while the baseline scores in the same table are reported without this protocol (except for the †-marked DALL-E 3 and Emu3 entries). Prompt rewriting can systematically improve alignment scores, so the comparison is potentially biased. Please report Goku-T2I on the original GenEval prompts alongside baselines on the same prompts, and separately report rewritten-prompt scores for both Goku and baselines under an identical rewriting protocol.
  3. [Section 5.2, Table 6] The UCF-101 zero-shot evaluation uses captions generated by Tarsier-34B (Wang et al., 2024a), and the training captioning pipeline uses Tarsier2 (Yuan et al., 2025). This creates a risk that the evaluation prompts are drawn from the same model family used for training captions, which may favor Goku over baselines. Additionally, the state-of-the-art FVD claim is made at 128x128 resolution while most baselines are evaluated at 256x256 or higher; lower resolution can artificially improve FVD. Please state whether the Tarsier-generated UCF-101 prompts overlap with training data, and report Goku's results at the same resolution and prompt set as the key baselines.
  4. [Tables 5, 7, 8] All headline benchmark numbers are reported as single runs without error bars, variance, or the number of repeated generations. For a state-of-the-art claim that is decided by a 0.58-point margin on VBench, run-to-run variability could change the ranking. Please provide multiple seeds or at least a variance estimate for the core benchmark scores, or otherwise justify that the differences are statistically significant.
  5. [Section 4.1, Section 5] The paper does not state whether the training data was deduplicated against the public GenEval, DPG-Bench, or VBench prompt sets. The data pipeline draws from LAION, Panda-70M, InternVid, OpenVid, and internet sources, and the benchmarks use public prompts; overlap could inflate scores. Please describe any decontamination procedure, or report which benchmarks, if any, were used to filter training data.
minor comments (5)
  1. [Abstract and Table 5] The abstract states 'Goku achieves 0.76 on GenEval' without noting that this result uses ChatGPT-4o prompt rewriting; please add the qualifier for consistency with the table.
  2. [Section 4.1] There is a typo: 'collet' should be 'collect'.
  3. [Table 6] The table would be clearer if the best result at a resolution comparable to each baseline were separated from the best overall; otherwise the reader may conflate the 128x128 FVD with a like-for-like comparison.
  4. [Appendix A] The names 'T2I-Compbench' and 'T2I-CompBench' are used inconsistently; please unify the capitalization.
  5. [Section 5.2] The phrase 'an video-language model' should be 'a video-language model'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark claims are external empirical measurements, not derived from fitted inputs.

full rationale

This paper is an empirical systems report rather than a derivation chain. The rectified-flow formulation in Eq. (1) is adopted from external prior work (Albergo & Vanden-Eijnden 2023; Lipman et al. 2023; Liu et al. 2023), and the headline claims are measurements on external benchmark suites (GenEval, DPG-Bench, VBench, UCF-101) against fixed evaluation protocols. The benchmark scores are not re-derived from model weights, nor is any fitted parameter renamed as a prediction. The report includes some in-house tools (e.g., Tarsier/Tarsier2 for captioning, ByteCheckpoint, MegaScale-style fault tolerance), but those are engineering components, not results whose validity is imported from the present paper. The use of Tarsier-34B to caption UCF-101 test videos while Tarsier2 is used in the training-data captioning pipeline, and the use of ChatGPT-4o prompt rewriting for GenEval, are evaluation-protocol concerns rather than circularity: they do not make a claimed output equivalent to an input by construction. Similarly, the absence of a detailed VBench generation protocol is a reproducibility and correctness risk, not a circularity risk. No load-bearing step reduces to a self-citation or to a self-defined quantity, so the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The paper contributes a training recipe, so its ledger consists of hand-chosen data thresholds and unstated background assumptions about the tools and benchmarks it relies on, rather than new theoretical objects. No new entities are postulated. The absence of released artifacts makes the hand-chosen thresholds unverifiable externally.

free parameters (4)
  • Aesthetic score threshold = 4.3 (480p), 4.5 (720p/1080p)
    Hand-chosen filter in Table 4 that decides which video clips enter training; the threshold directly shapes data quality and therefore the reported benchmark scores.
  • DINOv2 similarity threshold = 0.85 (480p), 0.90 (720p/1080p)
    Hand-chosen shot-boundary cutoff in Table 4 that determines clip segmentation and data volume.
  • OCR text coverage threshold = 0.02 (480p), 0.01 (720p/1080p)
    Hand-chosen filter in Table 4 excluding clips with excessive text, which removes captions-heavy content and shifts the generation distribution.
  • Motion score range = 0.3-20 (480p), 0.5-15 (720p), 0.5-8 (1080p)
    Hand-chosen motion filters in Table 4; the motion score is also appended to captions (Section 4.3), directly modifying the conditioning signal.
assumptions (6)
  • standard math The rectified flow linear interpolation x_t = t*x_1 + (1-t)*x_0 defines a valid probability path from noise to data (Eq. 1).
    Invoked in Section 2.3 as the training objective; taken from the cited works of Lipman et al., Liu et al., and Albergo and Vanden-Eijnden.
  • domain assumption The 3D joint image-video VAE compresses pixels into a latent space that preserves the information needed for the reported generation quality.
    Section 2.1 adopts the VAE from prior work (Yang et al., 2024c) without reporting reconstruction or compression loss figures for the joint variant.
  • domain assumption Flan-T5 text embeddings carry sufficient conditioning information for the reported text alignment scores.
    Section 2.2 fixes Flan-T5 as the text encoder without ablating alternative encoders.
  • domain assumption The official benchmark implementations were followed, and the training corpus does not contain the GenEval, DPG-Bench, or VBench prompts.
    Section 5 reports leaderboard comparisons with no stated evaluation protocol for VBench and no contamination check; this premise is required for the SOTA claim.
  • domain assumption The internal OCR model, internal video classifier, aesthetic scoring models, and RAFT motion estimates measure what the paper claims.
    Section 4.2 relies on these tools for filtering; none is described with accuracy metrics.
  • ad hoc to paper RoPE converges faster than sinusoidal embeddings, and appending the motion score to captions improves motion control.
    Section 2.2 and Section 4.3 state these as empirical findings without showing the supporting experiments; the motion-score claim is load-bearing for the prompt-conditioned motion argument.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Goku: Flow Based Video Generative Foundation Models." pith.science (2026). https://pith.science/paper/VO5PWDYV

@misc{pith2026250204896,
  author       = {Pith},
  title        = {Pith review of: Goku: Flow Based Video Generative Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VO5PWDYV}},
  note         = {Machine review of arXiv:2502.04896}
}
read the original abstract

This paper introduces Goku, a state-of-the-art family of joint image-and-video generation models leveraging rectified flow Transformers to achieve industry-leading performance. We detail the foundational elements enabling high-quality visual generation, including the data curation pipeline, model architecture design, flow formulation, and advanced infrastructure for efficient and robust large-scale training. The Goku models demonstrate superior performance in both qualitative and quantitative evaluations, setting new benchmarks across major tasks. Specifically, Goku achieves 0.76 on GenEval and 83.65 on DPG-Bench for text-to-image generation, and 84.85 on VBench for text-to-video tasks. We believe that this work provides valuable insights and practical advancements for the research community in developing joint image-and-video generation models.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning

    cs.CV 2025-09 conditional novelty 6.0 of 10

    HuMo uses a two-stage training scheme and a face-focus trick to generate human videos that follow text, keep a reference person's identity, and sync speech to audio, beating several single-task systems on benchmarks.

  2. AnyAni: An Interactive System with Generative AI for Animation Effect Creation and Code Understanding in Web Development

    cs.HC 2025-06 conditional novelty 6.0 of 10

    AnyAni combines LLM generation, a version tree, and video-based checking to help front-end developers create and understand web animations; a nine-person study reports usability gains over a chatbot baseline.

  3. Phantom-Data : Towards a General Subject-Consistent Video Generation Dataset

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Phantom-Data provides around one million cross-context, identity-consistent reference-video pairs for subject-to-video generation, and training on it improves prompt following and visual quality.

  4. M4V: Multimodal Mamba for Efficient Text-to-Video Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    M4V shows a Mamba-based text-to-video model can roughly match attention-based PyramidFlow on VBench while cutting mixer-layer FLOPs by 45% at 768x1280.

  5. Improving Video Diffusion Transformer Training by Multi-Feature Fusion and Alignment from Self-Supervised Vision Encoders

    cs.CV 2025-09 conditional novelty 5.0 of 10

    Matching video diffusion transformer tokens to concatenated DINOv2 and SAM2 features improves FVD/FID and speeds convergence, e.g., 400K-step fusion beats 1M-step baseline on UCF-101.

  6. From Black Box to Transparency: Enhancing Automated Interpreting Assessment with Explainable AI in College Classrooms

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    An explainable model using BLEURT, CometKiwi, pause features, and Chinese phraseological diversity predicts human-rated quality dimensions in English-Chinese consecutive interpreting, with SHAP identifying the stronge...

  7. RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Control

    cs.CV 2025-02 conditional novelty 5.0 of 10

    Metric-scale depth alignment plus scene-constrained noise shaping improves camera controllability and video quality for image-to-video generation on RealEstate10K.

  8. Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation

    cs.CV 2025-08 conditional novelty 4.0 of 10

    A hierarchical direct preference optimization with four alignment levels plus automated data selection improves physical plausibility of text-to-video models.

Reference graph

Works this paper leans on

97 extracted references · 31 canonical work pages · cited by 8 Pith papers

  1. [1]

    Agarwal, N., Ali, A., Bala, M., Balaji, Y., Barker, E., Cai, T., Chattopadhyay, P., Chen, Y., Cui, Y., Ding, Y., et al. (2025). Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575

    arXiv 2025

  2. [2]

    Albergo, M. S. and Vanden-Eijnden, E. (2023). Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations

    2023

  3. [3]

    Bacher, I., Javidnia, H., Dev, S., Agrahari, R., Hossari, M., Nicholson, M., Conran, C., Tang, J., Song, P., Corrigan, D., et al. (2021). An advert creation system for 3d product placements. In Machine Learning and Knowledge Discovery in Databases: Applied Data Science Track: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14--18, 2020, Pro...

    2021

  4. [4]

    Bao, F., Xiang, C., Yue, G., He, G., Zhu, H., Zheng, K., Zhao, M., Liu, S., Wang, Y., and Zhu, J. (2024). Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233

    arXiv 2024

  5. [5]

    Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., et al. (2023). Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf , 2(3):8

    2023

  6. [6]

    Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. (2023a). Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127

    arXiv 2023

  7. [7]

    W., Fidler, S., and Kreis, K

    Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. (2023b). Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22563--22575

    2023

  8. [8]

    Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., and Ramesh, A. (2024). Video generation models as world simulators

    2024

Show all 97 references
  1. [9]

    and Zisserman, A

    Carreira, J. and Zisserman, A. (2017). Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6299--6308

    2017

  2. [10]

    Castellano, B. (2024). PySceneDetect

    2024

  3. [11]

    Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. (2023). Pixart-alphaalpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426

    2023 arXiv

  4. [12]

    Chen, S., Xu, M., Ren, J., Cong, Y., He, S., Xie, Y., Sinha, A., Luo, P., Xiang, T., and Perez-Rua, J.-M. (2024a). Gentron: Diffusion transformers for image and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6441--6451

    2024

  5. [13]

    Chen, T., Xu, B., Zhang, C., and Guestrin, C. (2016). Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174

    2016 arXiv

  6. [14]

    E., Fang, Y., Lee, H.-Y., Ren, J., Yang, M.-H., et al

    Chen, T.-S., Siarohin, A., Menapace, W., Deyneka, E., Chao, H.-w., Jeon, B. E., Fang, Y., Lee, H.-Y., Ren, J., Yang, M.-H., et al. (2024b). Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and ...

    2024

  7. [15]

    Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al. (2024c). How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821

    2024 arXiv

  8. [16]

    W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al

    Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2024). Scaling instruction-finetuned language models. Journal of Machine Learning Research , 25(70):1--53

    2024

  9. [17]

    Contributors, I. H. (2013). Image hash

    2013

  10. [18]

    Corporation, N. (2022). Nvidia h100 tensor core gpu architecture

    2022

  11. [19]

    Corporation, N. (2023). Nvidia announces dgx gh200 ai supercomputer

    2023

  12. [20]

    Corporation, N. (2024). Nvidia h200 nvl pcie gpu accelerates ai and hpc applications

    2024

  13. [21]

    Dao, T. (2024). Flash A ttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR)

    2024

  14. [22]

    P., Caron, M., Geirhos, R., Alabdulmohsin, I., et al

    Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., et al. (2023). Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning , pages 7480--7512. PMLR

    2023

  15. [23]

    M., et al

    Dehghani, M., Mustafa, B., Djolonga, J., Heek, J., Minderer, M., Caron, M., Steiner, A., Puigcerver, J., Geirhos, R., Alabdulmohsin, I. M., et al. (2024). Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing S...

    2024

  16. [24]

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). ImageNet : A large-scale hierarchical image database. In CVPR , pages 248--255

    2009

  17. [25]

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., M \"u ller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. (2024). Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning

    2024

  18. [26]

    Esser, P., Rombach, R., and Ommer, B. (2021). Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12873--12883

    2021

  19. [27]

    Ge, S., Nah, S., Liu, G., Poon, T., Tao, A., Catanzaro, B., Jacobs, D., Huang, J.-B., Liu, M.-Y., and Balaji, Y. (2023). Preserve your own correlation: A noise prior for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages ...

    2023

  20. [28]

    Ghosh, D., Hajishirzi, H., and Schmidt, L. (2024). Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems , 36

    2024

  21. [29]

    S., Shah, A., Yin, X., Parikh, D., and Misra, I

    Girdhar, R., Singh, M., Brown, A., Duval, Q., Azadi, S., Rambhatla, S. S., Shah, A., Yin, X., Parikh, D., and Misra, I. (2023). Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709

    2023 arXiv

  22. [30]

    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances in neural information processing systems , 27

    2014

  23. [31]

    and Schmidhuber, J

    Ha, D. and Schmidhuber, J. (2018). World models. arXiv preprint arXiv:1803.10122

    2018 arXiv

  24. [32]

    He, J., Xue, T., Liu, D., Lin, X., Gao, P., Lin, D., Qiao, Y., Ouyang, W., and Liu, Z. (2024). Venhancer: Generative space-time enhancement for video generation. arXiv preprint arXiv:2407.07667

    2024 arXiv

  25. [33]

    He, Y., Yang, T., Zhang, Y., Shan, Y., and Chen, Q. (2022). Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221 , 2(3):4

    2022 arXiv

  26. [34]

    P., Poole, B., Norouzi, M., Fleet, D

    Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. (2022a). Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303

    2022 arXiv

  27. [35]

    Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840--6851

    2020

  28. [36]

    Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. (2022b). Video diffusion models. Advances in Neural Information Processing Systems , 35:8633--8646

    2022

  29. [37]

    Hong, W., Ding, M., Zheng, W., Liu, X., and Tang, J. (2022). Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868

    2022 arXiv

  30. [38]

    Hu, X., Wang, R., Fang, Y., Fu, B., Cheng, P., and Yu, G. (2024). Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135

    2024 arXiv

  31. [39]

    Huang, K., Sun, K., Xie, E., Li, Z., and Liu, X. (2023). T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems , 36:78723--78747

    2023

  32. [40]

    Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al. (2024). Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2...

    2024

  33. [41]

    A., Tanaka, M., Zhang, C., Zhang, M., Song, S

    Jacobs, S. A., Tanaka, M., Zhang, C., Zhang, M., Song, S. L., Rajbhandari, S., and He, Y. (2023). Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309.14509

    2023 arXiv

  34. [42]

    Ji, Y., Zhang, J., Wu, J., Zhang, S., Chen, S., GE, C., Sun, P., Chen, W., Shao, W., Xiao, X., et al. (2024). Prompt-a-video: Prompt your video diffusion model via preference-aligned llm. arXiv preprint arXiv:2412.15156

    2024 arXiv

  35. [43]

    Jiang, Z., Lin, H., Zhong, Y., Huang, Q., Chen, Y., Zhang, Z., Peng, Y., Li, X., Xie, C., Nong, S., et al. (2024). Megascale: Scaling large language model training to more than 10,000 gpus. arXiv preprint arXiv:2402.15627

    2024 arXiv

  36. [44]

    Jin, Y., Sun, Z., Li, N., Xu, K., Jiang, H., Zhuang, N., Huang, Q., Song, Y., Mu, Y., and Lin, Z. (2024). Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:2410.05954

    2024

  37. [45]

    Ju, X., Gao, Y., Zhang, Z., Yuan, Z., Wang, X., Zeng, A., Xiong, Y., Xu, Q., and Shan, Y. (2024). Miradata: A large-scale video dataset with long durations and structured captions. arXiv preprint arXiv:2407.06358

    2024 arXiv

  38. [46]

    Kingma, D. P. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114

    2013 arXiv

  39. [47]

    Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., et al. (2024). Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603

    2024 arXiv

  40. [48]

    A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B

    Korthikanti, V. A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B. (2023). Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems , 5:341--353

    2023

  41. [49]

    Kling ai

    Kuaishou (2024). Kling ai. https://klingai.com/

    2024

  42. [50]

    and etc., T

    Lab, P.-Y. and etc., T. A. (2024). Open-sora-plan

    2024

  43. [51]

    Li, S., Xue, F., Baranwal, C., Li, Y., and You, Y. (2021). Sequence parallelism: Long sequence training from system perspective. arXiv preprint arXiv:2105.13120

    2021 arXiv

  44. [52]

    Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M. (2023). Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations

    2023

  45. [53]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Liu, X., Gong, C., and qiang liu (2023). Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations

    2023

  46. [54]

    N., Zhu, S.-C., and Gao, J

    Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W., Wu, Y. N., Zhu, S.-C., and Gao, J. (2024). Chameleon: Plug-and-play compositional reasoning with large language models. Advances in Neural Information Processing Systems , 36

    2024

  47. [55]

    Luma (2024). Luma ai. https://lumalabs.ai/dream-machine

    2024

  48. [56]

    Mishkin, P., Ahmad, L., Brundage, M., Krueger, G., and Sastry, G. (2022). Dall e 2 preview-risks and limitations. Noudettu , 28(2022):3

    2022

  49. [57]

    Nan, K., Xie, R., Zhou, P., Fan, T., Yang, Z., Chen, Z., Li, X., Yang, J., and Tai, Y. (2024). Openvid-1m: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.02371

    2024 arXiv

  50. [58]

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023). Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193

    2023 arXiv

  51. [59]

    and Xie, S

    Peebles, W. and Xie, S. (2023). Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195--4205

    2023

  52. [60]

    pika (2024). Pika ai. https://pika.art/try

    2024

  53. [61]

    Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., M \"u ller, J., Penna, J., and Rombach, R. (2023). Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952

    2023 arXiv

  54. [62]

    Polyak, A., Zohar, A., Brown, A., Tjandra, A., Sinha, A., Lee, A., Vyas, A., Shi, B., Ma, C.-Y., Chuang, C.-Y., et al. (2024). Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720

    2024 arXiv

  55. [63]

    Quevedo, J., McIntyre, Q., Campbell, S., and Wachen, R. (2024). Oasis: A universe in a transformer

    2024

  56. [64]

    Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. (2022). Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 1(2):3

    2022 arXiv

  57. [65]

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684--10695

    2022

  58. [66]

    Gen-2: Generate novel videos with text, images or video clips

    Runway (2023). Gen-2: Generate novel videos with text, images or video clips. https://runwayml.com/research/gen-2/

    2023

  59. [67]

    Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022). Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing System...

    2022

  60. [68]

    Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. (2024). Flashattention-3: Fast and accurate attention with asynchrony and low-precision. arXiv preprint arXiv:2407.08608

    2024 arXiv

  61. [69]

    Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., Parikh, D., Gupta, S., and Taigman, Y. (2023). Make-a-video: Text-to-video generation without text-video data. In The Eleventh International Conference on Learning Representations

    2023

  62. [70]

    Skorokhodov, I., Tulyakov, S., and Elhoseiny, M. (2022). Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3626--3636

    2022

  63. [71]

    R., and Shah, M

    Soomro, K., Zamir, A. R., and Shah, M. (2012). Ucf101: A dataset of 101 human actions classes from videos in the wild. Technical report, Center for Research in Computer Vision, Orlando, FL 32816, USA. CRCV-TR-12-01

    2012

  64. [72]

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. (2024). Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 568:127063

    2024

  65. [73]

    Sun, P., Jiang, Y., Chen, S., Zhang, S., Peng, B., Luo, P., and Yuan, Z. (2024). Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525

    2024 arXiv

  66. [74]

    Team, V. (2024). Vchitect-2.0: Parallel transformer for scaling up video diffusion models. https://github.com/Vchitect/Vchitect-2.0

    2024

  67. [75]

    and Deng, J

    Teed, Z. and Deng, J. (2020). Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part II 16 , pages 402--419. Springer

    2020

  68. [76]

    Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. (2018). Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717

    2018 arXiv

  69. [77]

    Valevski, D., Leviathan, Y., Arar, M., and Fruchter, S. (2024). Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837

    2024 arXiv

  70. [78]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems , 30

    2017

  71. [79]

    Wan, B., Han, M., Sheng, Y., Lai, Z., Zhang, M., Zhang, J., Peng, Y., Lin, H., Liu, X., and Wu, C. (2024). Bytecheckpoint: A unified checkpointing system for llm development. arXiv preprint arXiv:2407.20143

    2024 arXiv

  72. [80]

    Wang, J., Yuan, L., Zhang, Y., and Sun, H. (2024a). Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634

    2024 arXiv

  73. [81]

    Wang, X., Zhang, X., Luo, Z., Sun, Q., Cui, Y., Wang, J., Zhang, F., Wang, Y., Li, Z., Yu, Q., Zhao, Y., Ao, Y., Min, X., Li, T., Wu, B., Zhao, B., Zhang, B., Wang, L., Liu, G., He, Z., Yang, X., Liu, J., Lin, Y., Huang, T., and Wang, Z. (2024b). Emu3: Next-token prediction is...

    2024

  74. [82]

    Wang, Y., Chen, X., Ma, X., Zhou, S., Huang, Z., Wang, Y., Yang, C., He, Y., Yu, J., Yang, P., et al. (2023a). Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103

    2023 arXiv

  75. [83]

    Wang, Y., He, Y., Li, Y., Li, K., Yu, J., Ma, X., Li, X., Chen, G., Chen, X., Wang, Y., et al. (2023b). Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942

    2023 arXiv

  76. [84]

    J., Bjontegaard, G., and Luthra, A

    Wiegand, T., Sullivan, G. J., Bjontegaard, G., and Luthra, A. (2003). Overview of the h. 264/avc video coding standard. IEEE Transactions on circuits and systems for video technology , 13(7):560--576

    2003

  77. [85]

    Z., Ge, Y., Wang, X., Lei, S

    Wu, J. Z., Ge, Y., Wang, X., Lei, S. W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., and Shou, M. Z. (2023). Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , page...

    2023

  78. [86]

    J., Wang, W., Lin, K

    Xie, J., Mao, W., Bai, Z., Zhang, D. J., Wang, W., Lin, K. Q., Gu, Y., Chen, Z., Yang, Z., and Shou, M. Z. (2024). Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528

    2024 arXiv

  79. [87]

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. (2024a). Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  80. [88]

    Yang, M., Li, J., Fang, Z., Chen, S., Yu, Y., Fu, Q., Yang, W., and Ye, D. (2024b). Playable game generation. arXiv preprint arXiv:2412.00887

    2024 arXiv

  81. [89]

    Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G., et al. (2024c). Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072

    2024 arXiv

  82. [90]

    Yuan, L., Wang, J., Sun, H., Zhang, Y., and Lin, Y. (2025). Tarsier2: Advancing large vision-language models from detailed video description to comprehensive video understanding. arXiv preprint arXiv:2501.07888

    2025 arXiv

  83. [91]

    Zeng, Y., Wei, G., Zheng, J., Zou, J., Wei, Y., Zhang, Y., and Li, H. (2024). Make pixels dance: High-dynamic video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8850--8860

    2024

  84. [92]

    and Sennrich, R

    Zhang, B. and Sennrich, R. (2019). Root mean square layer normalization. Advances in Neural Information Processing Systems , 32

    2019

  85. [93]

    Zhang, J., Chen, J., Wang, C., Yu, Z., Qi, T., Liu, C., and Wu, D. (2024). Virbo: Multimodal multilingual avatar video generation in digital marketing. arXiv preprint arXiv:2403.11700

    2024 arXiv

  86. [94]

    Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., Desmaison, A., Balioglu, C., Damania, P., Nguyen, B., Chauhan, G., Hao, Y., Mathews, A., and Li, S. (2023). Pytorch fsdp: Experiences on scaling fully sharded data p...

    2023

  87. [95]

    Zheng, Z., Peng, X., Yang, T., Shen, C., Li, S., Liu, H., Zhou, Y., Li, T., and You, Y. (2024). Open-sora: Democratizing efficient video production for all

    2024

  88. [96]

    Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. (2024). Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039

    2024 arXiv

  89. [97]

    Zhou, D., Wang, W., Yan, H., Lv, W., Zhu, Y., and Feng, J. (2022). Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018

    2022 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.