Pith. sign in

REVIEW 3 major objections 4 minor 63 references

Towards Precise Scaling Laws for Video Diffusion Transformers

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that video diffusion transformers obey precise scaling laws for validation loss, batch size, and learning rate, with fitted power laws that predict held-out losses within 0.15 percent.

desk verdict Solid empirical scaling-law study for video DiTs, but Eq 17 drops the α_T exponent and the derived Nopt comparison and 40.1% savings claim are unsupported as printed. read the letter →

arxiv 2411.17470 v2 pith:GEUEKERH submitted 2024-11-25 cs.CV cs.AIcs.LG

Yuanyang Yin , Yaqi Zhao , Mingwu Zheng , Ke Lin , Jiarong Ou , Rui Chen , Victor Shea-Jay Huang , Jiahao Wang
show 6 more authors
Xin Tao Pengfei Wan Di Zhang Baoqun Yin Wentao Zhang Kun Gai
This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords scalinglawsvideodiffusiontransformersoptimalhyperparameterslearningratebatchsizevalidationlosscompute-optimalmodelpowerlaw
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video diffusion transformers obey scaling laws, but only when the batch size and learning rate are also scaled optimally rather than held fixed. The paper fits power laws $B_{\mathrm{opt}}=\alpha_B T^{\beta_B}N^{\gamma_B}$ and $\eta_{\mathrm{opt}}=\alpha_\eta T^{\beta_\eta}N^{\gamma_\eta}$ for those hyperparameters across model size $N$ and training tokens $T$, then fits the validation loss as $L(T,N)=(T_c/T)^{\alpha_T}+(N_c/N)^{\alpha_N}+L_\infty$. With these choices the loss law predicts held-out runs closely: a 1.07B-parameter model trained on 10B tokens comes within 0.03% of prediction and a 0.72B model trained on 140B tokens within 0.15%. If the laws hold beyond the fitted range, practitioners can pick the model size and hyperparameters before large training runs, and can deliberately choose a smaller model near the flat loss minimum, cutting inference cost by about 40.1% at a compute budget of $10^{10}$ TFlops with only minor loss degradation.

What carries the argument

The central mechanism is a stepwise-loss analysis of mini-batch SGD. The expected loss change at step $k$ is approximated as $\Delta L_k \approx -\eta\|\nabla L\|^2 + \tfrac12\eta^2(\nabla L^\top H \nabla L + \mathrm{tr}(H\Sigma_k)/B)$, where $\mathrm{tr}(H\Sigma_k)/B$ is the gradient-noise term that shrinks with batch size. Balancing per-step gain against the number of steps $T/B$ yields power-law forms $B_{\mathrm{opt}}=\alpha_B T^{\beta_B}N^{\gamma_B}$ and $\eta_{\mathrm{opt}}=\alpha_\eta T^{\beta_\eta}N^{\gamma_\eta}$; the loss law $L(T,N)=(T_c/T)^{\alpha_T}+(N_c/N)^{\alpha_N}+L_\infty$ then separates the data-limited error, the model-limited error, and an irreducible entropy floor $L_\infty$. These fitted laws are what carry the extrapolation to larger models and compute budgets.

What would settle it

Train a video diffusion transformer far outside the fitted region, for example roughly 1.5B parameters on roughly 200B tokens, using the batch size and learning rate from the fitted laws, and compare the measured validation loss against the $L(T,N)$ prediction; likewise, measure the optimal model size by IsoFLOP curves at $C=10^{22}$ FLOPs and check whether $N_{\mathrm{opt}}$ still follows the same exponent. A deviation beyond the roughly 0.15% seen in the paper would mean the power laws bend.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that the validation loss of a video diffusion transformer is a predictable power-law function of model size and training tokens, provided each run uses the optimal batch size and learning rate for that $(N,T)$. It derives these optima from a mini-batch SGD analysis in which the per-step loss improvement is $-\eta\|\nabla L\|^2 + \tfrac12\eta^2(\nabla L^\top H \nabla L + \mathrm{tr}(H\Sigma)/B)$; larger models have larger Lipschitz constants and need smaller learning rates and larger batches, while more tokens shift the balance toward larger batches. The fitted exponents are $\beta_B=0.8080$, $\gamma_B=0.1906$, $\beta_\eta=-0.0453$, $\gamma_\eta=-0.1619$, and the loss law has $T_c=0.0373$, $\alpha_T=0.2917$, $N_c=0.0082$, $\alpha_N=0.3188$, $L_\infty=0.4856$. The paper validates the loss law by training a 1.07B model on 10B tokens (0.03% deviation) and a 0.72B model on 140B tokens (0.15% deviation), and shows that fixed suboptimal hyperparameters overestimate the optimal model size, with a 30.26% slope deviation versus 3.57% when hyperparameters are optimal.

Load-bearing premise

The load-bearing premise is that the three power laws fitted on models up to 0.26B parameters and 12B tokens keep their shape all the way to 0.72B parameters with 140B tokens and to $10^{22}$ FLOPs; if the exponents bend outside the fitted region, the predicted losses, model sizes, and the 40.1% saving would not hold.

Editorial extensions

If this is right

  • Given a compute budget, the fitted law $N_{\mathrm{opt}}=0.8705\,C^{0.4294}$ selects the model size before training; following it at $10^{10}$ TFlops saves about 39.9% of parameters relative to the fixed-suboptimal rule while keeping validation loss nearly unchanged.
  • For any model size and token budget, validation loss is predictable through the $L(T,N)$ formula, so non-optimal sizes chosen for inference cost or hardware constraints can still have their loss anticipated.
  • The optimal batch size and learning rate for a target scale can be read off the two hyperparameter laws, removing the need for a grid search at large scale.
  • Because the loss curve is flat near the optimum, choosing a smaller model than the nominal optimum trades only a small, predictable loss increase for a large inference-cost reduction, which is the origin of the claimed 40.1% saving.
  • Fixed-suboptimal hyperparameters, by contrast, overestimate the optimal model size and make the loss fit substantially worse (MSE $4.31\times10^{-7}$ vs $2.35\times10^{-7}$), so hyperparameter choice is part of the scaling law, not an afterthought.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The flat-loss-near-optimum structure implies the fitted exponents only need to be approximately right for the practical decisions on model size and hyperparameters to remain near-optimal; a direct test would be to perturb $B$ and $\eta$ by the same relative amount at a new scale and measure the loss difference.
  • The same methodology should be re-fit, not reused, when resolution, frame count, architecture, or optimizer and scheduler change; the paper itself notes its laws were fitted at 256x256 resolution, 17 frames, and constant learning rate, so the exponents are likely setup-specific.
  • A natural extension is to treat the learning-rate schedule as another hyperparameter axis; the convergence-bound argument suggests an optimal schedule could be folded into the $\eta_{\mathrm{opt}}$ law rather than requiring a separate constant-rate assumption.
  • If the approach transfers to other architectures or modalities, the 40.1% inference saving is not a fixed property of video DiT but a consequence of where the conventional rule lands on the flat optimum; comparing the two rules at several compute budgets would show how the saving changes with budget.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies scaling laws for video diffusion transformers (Cross-DiT architecture). It proposes two power-law fits for the optimal batch size and learning rate as functions of model size N and training tokens T, and a three-term validation-loss law L(T,N) = (Tc/T)^α_T + (Nc/N)^α_N + L∞. The authors claim that training with these optimal hyperparameters yields more precise loss fits, allows prediction of optimal model size versus compute budget, and leads to a 39.9% parameter saving and a 40.1% inference-cost reduction at a 1e10 TFlops budget. Out-of-sample checks are reported at 1.07B parameters / 10B tokens and 0.72B parameters / 140B tokens (also described as 5.85e20 FLOPs), with deviations of 0.03% and 0.15%.

Significance. If the proposed laws hold, the paper would provide practically useful guidance for allocating compute between model size and data in video diffusion training, and the explicit treatment of batch size and learning rate is a useful step beyond standard LLM-style scaling laws. The strongest evidence is the out-of-sample validation of L(T,N) at 1.07B/10B and the claimed 0.72B point, which go beyond internal curve fitting. The paper also gives a detailed FLOPs-per-token accounting and honestly states limitations (constant learning rate, low resolution, small models, validation-loss-only evaluation). However, the algebraic derivation of the predicted optimal model size contains a serious error, and one flagship validation point is described inconsistently in terms of tokens versus compute, so the headline quantitative claims need to be re-derived and re-stated before the paper can be accepted.

major comments (3)
  1. [Sec. 4.3.2, Eq. (17)] Equation (17) drops the exponent α_T when substituting C = C_token T into Eq. (15). The correct substitution is L(N) = (Tc · 3N(7+nctx/d)/(4C))^{α_T} + (Nc/N)^{α_N} + L∞, not the printed linear first term Tc · 3N(7+nctx/d)/(4C). This is not cosmetic: minimizing the printed form gives Nopt ∝ C^{1/(1+α_N)} ≈ C^{0.758}, while restoring the exponent and holding (7+nctx/d) fixed gives Nopt ∝ C^{α_T/(α_T+α_N)} ≈ C^{0.478}; neither equals the stated Eq. (18), Nopt = 0.8705·C^{0.4294}. The derivation of Eq. (18), the 3.57% slope agreement in Fig. 7, and the associated parameter/inference saving estimates therefore rest on an algebraic error. Please re-derive Eq. (18) or state explicitly how it was obtained (e.g., as a numerical power-law fit to the corrected minimization, including the d(N) dependence from d/nlayer = 128).
  2. [Sec. 4.2 and Fig. 5b vs. Sec. 4.3.1] The 0.72B out-of-sample validation point is internally inconsistent. The text says this model was trained on 140B tokens, while Sec. 4.3.1 and Fig. 5b assign the same point a compute budget of 5.85e20 FLOPs. Using the paper's own compute model from Appendix D (Table 6), a 719.3M-parameter model with d=1792 and nctx=1,113,632 has C_token ≈ (3/4)(7+nctx/d)N ≈ 3.4e11 FLOPs per token. That gives 140B tokens ≈ 4.8e22 FLOPs, whereas 5.85e20 FLOPs corresponds to about 1.7B tokens. These two descriptions cannot both be right, and the 0.15% deviation claim depends on which run was actually performed. Please reconcile the token count, the FLOPs budget, and the figure labels.
  3. [Sec. 4.3.1 and Abstract (40.1% claim)] The headline parameter-saving and inference-cost numbers are obtained by extrapolating the Nopt-vs-C scaling laws to C = 1e10 TFlops = 1e22 FLOPs, which is more than three orders of magnitude beyond the largest fitted compute budget (6e18 FLOPs). The 0.72B run at 5.85e20 FLOPs provides partial support, but no experiment at 1e22 FLOPs validates the slope of the extrapolated Nopt(C) law. In addition, the 3.57% agreement in Fig. 7 compares the predicted Nopt(C) from Eq. (18) with the empirical Nopt(C) from Eq. (14) on the same five IsoFLOP budgets used to fit both curves; this is an internal-consistency check, not an out-of-sample validation. The paper should present the 1e22 extrapolation as a prediction and state the untested range explicitly, or add an intermediate out-of-sample compute budget.
minor comments (4)
  1. [Supplementary Eq. (23)] Equation (23) appears dimensionally inconsistent: from Eq. (22) and C = C_token T, the correct expression is Topt = (4/(3(7+nctx/d)·0.8705))·C^{0.5706}, not (4/3)(7+nctx/d)·C^{0.5706} as printed.
  2. [Sec. 3.4 / Tables 1 and 2] The fitted coefficients for β_B, γ_B, α_η, etc. are presented without confidence intervals or sensitivity to the 0.02% near-optimal threshold used to define 'optimal' hyperparameters; adding this would help readers judge the stability of the fitted exponents.
  3. [Figures and text] There are several typos and missing spaces in figure labels and text, e.g., 'Traing T okens' in Fig. 1 and Fig. 5, 'FixedSuboptimalHyperparametersOptimalHyperparameters' in Fig. 7, and 'video-specfic' in Sec. 5; these should be corrected.
  4. [Appendix E] The sentence 'which can also make accurate predictions for image generation' is ungrammatical and should be rewritten, e.g., 'showing that the approach also gives accurate predictions for image generation.'

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: the Figure 7 Nopt comparison is an in-sample consistency check, but the central loss law has independent out-of-sample support.

  1. fitted input called prediction [Section 4.3.2, Equations (14)-(18), Figure 7]
    "Additionally, we validate the accuracy of our predictions by fitting the predicted optimal model size based on the loss function derived under both optimal and fixed suboptimal hyperparameters ( Table 3 and Appendix C). We then compare the predicted optimal model size with empirical optimal model size to assess the accuracy of the loss function."

    The 'empirical optimal model size' (Eq. 14, Fig. 6a) is the argmin of parabolic fits to the same loss-vs-N runs used to fit the loss law L(T,N) in Eq. 15/Table 3 (Section 4.1 states the IsoFLOP budgets 3e17-6e18 were run using the paper's own optimal-hyperparameter formulas). The 'predicted optimal model size' (Eq. 18) is obtained by minimizing that same fitted L(T,N). Comparing the two therefore checks the fitted surface against the data it was fit on; agreement is an in-sample consistency condition, not an independent verification. The 3.57% slope deviation is thus not evidence that optimal hyperparameters make loss prediction more accurate than the MSE numbers already provide, and the 39.9%/40.1% parameter/inference-saving numbers inherit this in-sample character.

full rationale

The central loss law L(T,N) (Eq. 15) is genuinely tested out-of-sample at N=1.07B/T=10B and N=0.72B/T=140B, with deviations 0.03% and 0.15%; these are independent evidence and break any full circularity. The Bopt/eta_opt laws are also validated by an out-of-sample grid search at 1.07B (Figure 4). What is circular is the Figure 7 / Section 4.3.2 comparison: the 'empirical optimal model size' and the 'predicted optimal model size' are both derived from the same loss runs, so the 3.57% agreement is an internal consistency check rather than a fresh prediction. Separately, Eq. (17) appears to drop the exponent alpha_T = 0.2917 when substituting C = C_token T, so Eq. (18) and the slope comparison built on it are mathematically unsupported; this is a derivational error rather than a circular reduction, so it is noted but not added to the circularity score. The paper's own stated limitations (constant LR, validation-loss-only evaluation, low-resolution small models) are correctness/robustness caveats, not circularity. Overall score 4: partial circularity in the internal Nopt comparison, but the claim's independent content (out-of-sample loss predictions) remains.

Assumptions & free parameters 14 free parameters · 7 assumptions · 0 invented entities

The paper's quantitative claims rest on fourteen or more fitted constants and three posited power-law forms. The SGD theory provides motivation but does not derive the exponents. No new physical entities are introduced. The axioms are a mix of standard optimization assumptions and domain-specific modeling choices that are not independently validated.

free parameters (14)
  • alpha_B (coefficient in Bopt scaling law) = 2.1797e4 (tokens) / 17.0287 (samples)
    Fitted coefficient in Bopt = alpha_B * T^beta_B * N^gamma_B; the paper reports two values depending on whether batch size is measured in tokens or samples.
  • beta_B = 0.8080
    Fitted exponent on training tokens T in the Bopt scaling law (Table 1).
  • gamma_B = 0.1906
    Fitted exponent on model size N in the Bopt scaling law (Table 1).
  • alpha_eta = 0.0002
    Fitted coefficient in eta_opt = alpha_eta * T^beta_eta * N^gamma_eta (Table 2).
  • beta_eta = -0.0453
    Fitted exponent on T in the eta_opt scaling law (Table 2).
  • gamma_eta = -0.1619
    Fitted exponent on N in the eta_opt scaling law (Table 2).
  • Tc = 0.0373
    Fitted coefficient in the loss law L(T,N) = (Tc/T)^alpha_T + (Nc/N)^alpha_N + L_infinity (Table 3).
  • alpha_T = 0.2917
    Fitted exponent on T in the loss law (Table 3).
  • Nc = 0.0082
    Fitted coefficient in the loss law (Table 3).
  • alpha_N = 0.3188
    Fitted exponent on N in the loss law (Table 3).
  • L_infinity = 0.4856
    Fitted asymptotic validation loss, interpreted as data entropy (Table 3).
  • Empirical Nopt vs C prefactor and exponent = 1.5787 * C^0.4146
    Power-law fit to empirical optimal model sizes from IsoFLOP profiles (Eq 14).
  • Predicted Nopt vs C prefactor and exponent = 0.8705 * C^0.4294
    Power-law fit to predicted optimal model sizes obtained by minimizing the fitted loss law (Eq 18).
  • Near-optimal threshold = 0.02%
    Hand-chosen loss window used to define which hyperparameter configurations count as optimal for fitting Bopt and eta_opt (Section 3.4).
assumptions (7)
  • standard math Mini-batch SGD gradient estimate is unbiased with bounded variance (Eqs 2-3).
    Standard assumptions in stochastic optimization, from McCandlish et al. [38], used for the convergence bound in Eq 4 and the stepwise loss expansion in Eq 7.
  • standard math Loss L(theta) is L-smooth along the training trajectory.
    Standard smoothness assumption for SGD convergence analysis, cited from [56], used in Eq 4 and Eq 7.
  • ad hoc to paper Power-law forms Bopt = alpha_B * T^beta_B * N^gamma_B, eta_opt = alpha_eta * T^beta_eta * N^gamma_eta, and L(T,N) = (Tc/T)^alpha_T + (Nc/N)^alpha_N + L_infinity hold over the extrapolated range.
    These functional forms are posited, not derived. Section 3.2 says 'we posit' for Bopt, and Section 4.2 says the loss formula 'is designed' to adhere to two guiding principles. The SGD theory only motivates directions, not the exponents. The extrapolation to 0.72B/140B tokens and to 1e22 FLOPs depends on these forms remaining valid.
  • domain assumption Validation loss on a 2000-sample subset of Panda-70M is a faithful, comparable proxy for video generation quality.
    Section 2 states external benchmarks were not used due to alignment challenges, and validation loss is justified by citations to [13,43]. This is an assumption about the metric, not demonstrated for this specific setting.
  • domain assumption Constant learning rate training in the stable phase is representative for scaling law derivation.
    Section 2 focuses on the stable phase with a constant learning rate, and the limitation section acknowledges that learning rate decay could yield different results. The fitted laws may not transfer to cosine or other schedules.
  • domain assumption Architecture scaling rule d/nlayer = 128 and number of attention heads equal to nlayer is held fixed across model sizes.
    Appendix D specifies this proportional scaling rule. The scaling laws are fit under this rule and may not hold for other depth-width aspect ratios.
  • domain assumption Compute model C = 3 * N * (7 + nctx/d) / 4 * T approximates actual training cost.
    Appendix D derives FLOP counts for the Cross-DiT forward pass, with the factor 3 accounting for forward and backward. This excludes optimizer, communication, and data-loading overheads, and assumes the FLOP model is accurate for all model sizes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Precise Scaling Laws for Video Diffusion Transformers." pith.science (2026). https://pith.science/paper/GEUEKERH

@misc{pith2026241117470,
  author       = {Pith},
  title        = {Pith review of: Towards Precise Scaling Laws for Video Diffusion Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GEUEKERH}},
  note         = {Machine review of arXiv:2411.17470}
}
read the original abstract

Achieving optimal performance of video diffusion transformers within given data and compute budget is crucial due to their high training costs. This necessitates precisely determining the optimal model size and training hyperparameters before large-scale training. While scaling laws are employed in language models to predict performance, their existence and accurate derivation in visual generation models remain underexplored. In this paper, we systematically analyze scaling laws for video diffusion transformers and confirm their presence. Moreover, we discover that, unlike language models, video diffusion models are more sensitive to learning rate and batch size, two hyperparameters often not precisely modeled. To address this, we propose a new scaling law that predicts optimal hyperparameters for any model size and compute budget. Under these optimal settings, we achieve comparable performance and reduce inference costs by 40.1% compared to conventional scaling methods, within a compute budget of 1e10 TFlops. Furthermore, we establish a more generalized and precise relationship among validation loss, any model size, and compute budget. This enables performance prediction for non-optimal model sizes, which may also be appealed under practical inference cost constraints, achieving a better trade-off.

Figures

Figures reproduced from arXiv: 2411.17470 by the authors.

Figure 1
Figure 1. Validation loss across various model sizes with different amounts of training tokens and hyperparameters. Each panel represents [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Optimal learning rate scaling curve. Left: Optimal learning rate scaling curves fitted on four different model sizes (0.02B, 0.06B, 0.13B, and 0.26B parameters). “Observations” indicates values within 0.02% of the minimum loss for each model size. Right: Extrapolated scaling curves for learning rate, predicting optimal values for a 1.07B model to achieve minimal validation loss. properties under the framework of min… view at source ↗
Figure 3
Figure 3. Optimal batch size scaling curve. Left: Optimal batch size scaling curves fitted on four different model sizes (0.02B, 0.06B, 0.13B, and 0.26B parameters). “Observations” indicates values within 0.02% of the minimum loss for each model size. Right: Extrapolated scaling curves for batch size, predicting optimal values for a 1.07B model to achieve minimal validation loss. update steps S decreases, which limits the cum… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Predictions of optimal hyperparameters on 1.07B model size with 4B and 10B training tokens. The red pentagrams indicate the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Loss scaling with optimal hyperparameters across varying model and compute scales. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparison of empirical loss v.s. N under optimal and fixed suboptimal hyperparameters. Left: Empirical loss as a function of model size N for various compute budgets C under optimal hyperparameters, with a parabolic fit to identify minimum loss points. Middle: Results…
Figure 7
Figure 7. Figure 7: Comparison of empirical and predicted optimal model size scaling law under optimal and fixed suboptimal hyperparameters. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Loss scaling with fixed suboptimal hyperparameters across varying model and compute scales. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Empirical and predicted optimal model size on fixed suboptimal batch size and learning rate. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 15 canonical work pages

  1. [1]

    Deepseek llm: Scaling open- source language models with longtermism

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open- source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. 1, 2, 3, 4, 5, 6, 8

    arXiv 2024

  2. [2]

    u-mup: The unit-scaled maximal update parametrization

    Charlie Blake, Constantin Eichenberg, Josef Dean, Lukas Balles, Luke Y Prince, Bj¨orn Deiseroth, Andres Felipe Cruz- Salinas, Carlo Luschi, Samuel Weinbach, and Douglas Orr. u-mup: The unit-scaled maximal update parametrization. arXiv preprint arXiv:2407.17465, 2024. 3

    arXiv 2024

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 1

    arXiv 2023

  4. [4]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023. 9

    work page 2023

  5. [5]

    Video generation models as world simulators, 2024

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, et al. Video generation models as world simulators, 2024. 1

    2024

  6. [6]

    Videocrafter2: 9 Overcoming data limitations for high-quality video diffu- sion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: 9 Overcoming data limitations for high-quality video diffu- sion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7310– 7320, 2024. 1

    work page 2024

  7. [8]

    Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 3

    arXiv

  8. [9]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13320–13331, 2024. 1

    work page 2024

Show all 63 references
  1. [10]

    Seine: Short-to-long video diffu- sion model for generative transition and prediction

    Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffu- sion model for generative transition and prediction. In The Twelfth International Conference on Learning Representa- ti...

    2023

  2. [11]

    Adver- sarial video generation on complex datasets

    Aidan Clark, Jeff Donahue, and Karen Simonyan. Adver- sarial video generation on complex datasets. arXiv preprint arXiv:1907.06571, 2019. 9

    1907 arXiv

  3. [12]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  4. [13]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  5. [14]

    Lumina-t2x: Transforming text into any modality, reso- lution, and duration via flow-based large diffusion transform- ers

    Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Transforming text into any modality, reso- lution, and duration via flow-based large diffusion transform- ers. arXiv preprint arXiv:2405.05945, 2024. 1, 9

    2024 arXiv

  6. [16]

    Emu video: Factoriz- ing text-to-video generation by explicit image conditioning

    Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Du- val, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factoriz- ing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709, 2023. 9

    2023 arXiv

  7. [17]

    Accurate, large minibatch sgd: training imagenet in 1 hour

    P Goyal. Accurate, large minibatch sgd: training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017. 3

    2017 arXiv

  8. [18]

    Latent video diffusion models for high-fidelity video generation with arbitrary lengths

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221, 2022. 9

    2022 arXiv

  9. [20]

    Scaling laws for autoregressive generative modeling

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020. 9

    2010 arXiv

  10. [21]

    Streamingt2v: Con- sistent, dynamic, and extendable long video generation from text

    Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Streamingt2v: Con- sistent, dynamic, and extendable long video generation from text. arXiv preprint arXiv:2403.14773, 2024. 9

    2024 arXiv

  11. [22]

    Lo- cal lipschitz bounds of deep neural networks

    Calypso Herrera, Florian Krach, and Josef Teichmann. Lo- cal lipschitz bounds of deep neural networks. arXiv preprint arXiv:2004.13135, 2020. 4

    2004 arXiv

  12. [23]

    Deep learning scaling is predictable, empirically.arXiv preprint arXiv:1712.00409,

    Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically.arXiv preprint arXiv:1712.00409,

    arXiv

  13. [24]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1

    2020

  14. [25]

    Imagen video: High definition video generation with diffusion mod- els

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion mod- els. arXiv preprint arXiv:2210.02303, 2022. 1

    arXiv

  15. [26]

    Training compute-optimal large language mod- els

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language mod- els. arXiv preprint arXiv:2203.15556 , 2022. 1, 2, 3, 6, 7, 9

    2022 arXiv

  16. [27]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022. 9

    2022 arXiv

  17. [28]

    Depthcrafter: Generating consistent long depth sequences for open-world videos

    Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-world videos. arXiv preprint arXiv:2409.02095, 2024. 9

    2024 arXiv

  18. [29]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  19. [30]

    Klingai, 2024

    KlingAI. Klingai, 2024. Accessed: 2024-11-14. 1

    2024

  20. [31]

    Videopoet: A large language model for zero-shot video generation

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos ´e Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan 10 Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023. 9

    2023 arXiv

  21. [32]

    On the scalability of diffusion-based text-to-image generation

    Hao Li, Yang Zou, Ying Wang, Orchid Majumder, Yusheng Xie, R Manmatha, Ashwin Swaminathan, Zhuowen Tu, Ste- fano Ermon, and Stefano Soatto. On the scalability of diffusion-based text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern ...

    2024

  22. [33]

    Scaling laws for diffusion transformers

    Zhengyang Liang, Hao He, Ceyuan Yang, and Bo Dai. Scaling laws for diffusion transformers. arXiv preprint arXiv:2410.08184, 2024. 9

    2024

  23. [34]

    Mardini: Masked autoregres- sive diffusion for video generation at scale

    Haozhe Liu, Shikun Liu, Zijian Zhou, Mengmeng Xu, Yan- ping Xie, Xiao Han, Juan C P ´erez, Ding Liu, Kumara Ka- hatapitiya, Menglin Jia, et al. Mardini: Masked autoregres- sive diffusion for video generation at scale. arXiv preprint arXiv:2410.20280, 2024. 9

    arXiv

  24. [35]

    Vdt: General-purpose video diffusion transformers via mask modeling

    Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding. Vdt: General-purpose video diffusion transformers via mask modeling. arXiv preprint arXiv:2305.13311, 2023. 1, 9

    2023 arXiv

  25. [36]

    Sit: Explor- ing flow and diffusion-based generative models with scalable interpolant transformers

    Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Explor- ing flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint arXiv:2401.08740,

    arXiv

  26. [37]

    Latte: Latent diffusion transformer for video generation

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Zi- wei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024. 1, 9

    2024 arXiv

  27. [38]

    An empirical model of large-batch training

    Sam McCandlish, Jared Kaplan, Dario Amodei, and Ope- nAI Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018. 3, 4, 2

    2018 arXiv

  28. [39]

    Bigger is not always better: Scaling properties of latent diffusion models

    Kangfu Mei, Zhengzhong Tu, Mauricio Delbracio, Hossein Talebi, Vishal M Patel, and Peyman Milanfar. Bigger is not always better: Scaling properties of latent diffusion models. arXiv preprint arXiv:2404.01367, 2024. 1, 9

    2024 arXiv

  29. [40]

    Sora, 2024

    OpenAI. Sora, 2024. Accessed: 2024-11-13. 1

    2024

  30. [42]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  31. [43]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  32. [44]

    Tempo- ral generative adversarial nets with singular value clipping

    Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Tempo- ral generative adversarial nets with singular value clipping. In Proceedings of the IEEE international conference on com- puter vision, pages 2830–2839, 2017. 9

    2017

  33. [45]

    Fast high- resolution image synthesis with latent adversarial diffusion distillation

    Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high- resolution image synthesis with latent adversarial diffusion distillation. arXiv preprint arXiv:2403.12015, 2024. 1

    2024 arXiv

  34. [46]

    Measuring the effects of data parallelism on neural network training

    Christopher J Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research , 20(112): 1–49, 2019. 3

    2019

  35. [47]

    Power scheduler: A batch size and token number agnostic learning rate sched- uler

    Yikang Shen, Matthew Stallone, Mayank Mishra, Gaoyuan Zhang, Shawn Tan, Aditya Prasad, Adriana Meza Soria, David D Cox, and Rameswar Panda. Power scheduler: A batch size and token number agnostic learning rate sched- uler. arXiv preprint arXiv:2408.13359, 2024. 2

    2024 arXiv

  36. [48]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  37. [49]

    Don’t decay the learning rate, increase the batch size

    SL Smith. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017. 3

    2017 arXiv

  38. [50]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 1

    2010 arXiv

  39. [51]

    Video-infinity: Distributed long video generation

    Zhenxiong Tan, Xingyi Yang, Songhua Liu, and Xinchao Wang. Video-infinity: Distributed long video generation. arXiv preprint arXiv:2406.16260, 2024. 9

    2024 arXiv

  40. [52]

    Mocogan: Decomposing motion and content for video generation

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526–1535,

  41. [53]

    Phenaki: Variable length video generation from open domain textual descriptions

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kin- dermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Le...

    2022

  42. [54]

    Generating videos with scene dynamics

    Carl V ondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. Advances in neu- ral information processing systems, 29, 2016. 9

    2016

  43. [55]

    Gen-l-video: Multi-text to long video generation via temporal co-denoising

    Fu-Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen-l-video: Multi-text to long video generation via temporal co-denoising. arXiv preprint arXiv:2305.18264, 2023. 9

    2023 arXiv

  44. [56]

    Adapting to smoothness: A more universal algorithm for on- line convex optimization

    Guanghui Wang, Shiyin Lu, Yao Hu, and Lijun Zhang. Adapting to smoothness: A more universal algorithm for on- line convex optimization. In Proceedings of the AAAI Con- ference on Artificial Intelligence, pages 6162–6169, 2020. 3

    2020

  45. [57]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024. 9

    2024 arXiv

  46. [58]

    Videogpt: Video generation using vq-vae and trans- formers

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and trans- formers. arXiv preprint arXiv:2104.10157, 2021. 9

    2021 arXiv

  47. [59]

    Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer

    Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, 11 Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022. 3

    2022 arXiv

  48. [60]

    Rerender a video: Zero-shot text-guided video-to-video translation

    Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation. In SIGGRAPH Asia 2023 Conference Papers , pages 1–11, 2023. 9

    2023

  49. [61]

    Space-time diffusion features for zero-shot text-driven motion transfer

    Danah Yatim, Rafail Fridman, Omer Bar-Tal, Yoni Kasten, and Tali Dekel. Space-time diffusion features for zero-shot text-driven motion transfer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8466–8476, 2024. 9

    2024

  50. [62]

    Nuwa-xl: Diffusion over diffusion for extremely long video generation.arXiv preprint arXiv:2303.12346, 2023

    Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, et al. Nuwa-xl: Diffusion over diffusion for extremely long video generation.arXiv preprint arXiv:2303.12346, 2023. 9

    2023 arXiv

  51. [63]

    Generating videos with dynamics-aware implicit generative adversarial net- works

    Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial net- works. arXiv preprint arXiv:2202.10571, 2022. 9

    2022 arXiv

  52. [64]

    Tora: Trajectory-oriented diffu- sion transformer for video generation

    Zhenghao Zhang, Junchao Liao, Menghao Li, Long Qin, and Weizhi Wang. Tora: Trajectory-oriented diffu- sion transformer for video generation. arXiv preprint arXiv:2407.21705, 2024. 1

    2024 arXiv

  53. [65]

    Moviedreamer: Hier- archical generation for coherent long visual sequence

    Canyu Zhao, Mingyu Liu, Wen Wang, Jianlong Yuan, Hao Chen, Bo Zhang, and Chunhua Shen. Moviedreamer: Hier- archical generation for coherent long visual sequence. arXiv preprint arXiv:2407.16655, 2024. 9

    2024

  54. [66]

    Open-sora: Democratizing efficient video production for all, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024. 1

    2024

  55. [67]

    Storydiffusion: Consistent self- attention for long-range image and video generation

    Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou. Storydiffusion: Consistent self- attention for long-range image and video generation. arXiv preprint arXiv:2405.01434, 2024. 9 12 Towards Precise Scaling Laws for Video Diffusion Transformers Supplementary ...

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.