Pith. sign in

REVIEW 4 major objections 4 minor 87 references

PoM: Efficient Image and Video Generation with the Polynomial Mixer

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Polynomial mixer replaces attention in diffusion models at linear cost

desk verdict Interesting linear-complexity attention replacement with honest ImageNet results, but the universality proof has a real gap and the video numbers are far off baselines. read the letter →

arxiv 2411.12663 v1 pith:X42MHDVB submitted 2024-11-19 cs.CV cs.AIcs.LG

David Picard , Nicolas Dufour This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords PolynomialMixerdiffusiontransformerlinearcomplexityuniversalapproximationimagegenerationvideoblock-causalmaskingstate-spacemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes replacing multi-head attention (MHA) in transformer-based diffusion models with a linear-complexity building block called the Polynomial Mixer (PoM). Instead of computing pairwise token interactions, PoM compresses the whole sequence into an explicit state—an average of high-order polynomial expansions of the tokens—that every token then queries through a learned gate. The authors prove, by adapting the standard transformer universality argument, that PoM is a universal sequence-to-sequence approximator, and they show experimentally that DiT-style image and video generators built on PoM match the quality of attention-based ones while using less compute, especially at high resolution. If correct, this means the quadratic cost of attention is not needed for state-of-the-art generative quality, and scaling to higher resolutions or longer videos becomes far cheaper.

What carries the argument

The load-bearing object is the explicit state $H(X)$, a vector obtained by averaging, over all tokens, the elementwise products of activations $h(W_1 X), h(W_1 X)\circ h(W_2 X), \ldots$ up to degree $k$. Each token independently queries this shared state through a sigmoid gate $\sigma(W_s X)$, and the gated state is projected back by $W_o$. This 'blackboard' summary replaces the pairwise attention matrix: because $H(X)$ is computed once per sequence, complexity drops from $O(n^2)$ to $O(n)$ in token count. The universality proof leans on Lemma 3 (contextual mapping), which asserts that a sufficiently high-degree polynomial state distinguishes every pair of distinct sequences token by token, mirroring the lemma in the standard transformer universality proof.

What would settle it

Compute the polynomial state $H(X)$ for two distinct token matrices $X$ and $X'$ that differ in one column, and check whether any column of $\mathrm{PoM}(X)$ equals any column of $\mathrm{PoM}(X')$ across a range of degrees $k$; a concrete scalar example from the appendix's own simplification—$X=\{1,2\}$, $X'=\{1,5\}$ with $k=1$, where the sums differ (3 vs 6) yet $2\cdot 3 = 1\cdot 6$—shows distinct sums need not yield distinct columns, so such a search (or a proof that the full matrix case avoids it) would settle whether Lemma 3 and the universality theorem hold.

Watch

Extended reading notes

Core claim

The paper's central claim is that the Polynomial Mixer—defined by $\mathrm{PoM}(X) = W_o[\sigma(W_s X) \circ H(X)\mathbf{1}^\top]$, where $H(X)$ aggregates products of activations up to degree $k$ across all tokens—has the same expressive power as multi-head attention for sequence-to-sequence mapping, including a universal approximation theorem, while having linear rather than quadratic complexity in sequence length. It further claims that this replacement is drop-in: adapting DiT image models and text-to-video models by swapping MHA for PoM yields samples of comparable quality (ImageNet 256×256 FID 2.46 vs DiT's 2.27 with half the training steps) and, because PoM exposes an explicit state, video frames can be generated sequentially with constant per-frame cost.

Load-bearing premise

The universal-approximation result rests on Lemma 3's assumption that a high-degree polynomial state can distinguish every pair of distinct input sequences token by token; the paper's proof establishes this only for a simplified scalar, linear-activation version, not for the full nonlinear matrix mixer.

Editorial extensions

If this is right

  • Training a DiT-XL/2-scale model with PoM at high resolutions becomes cheaper than even inference with the attention-based DiT-XL/2, according to the paper's timing measurements.
  • Image generation on ImageNet 256×256 reaches FID 2.46 with 950M parameters, close to DiT-XL/2's 2.27, using half the training steps (the paper also reports a flow-matching variant at FID 3.70).
  • For video, block-causal masking lets the model generate frames sequentially with constant per-frame cost, improving temporal metrics like dynamic degree and human action at the price of some consistency scores.
  • PoM shows the same log-linear scaling of FID with compute as transformer-based DiT, indicating it retains favorable scaling laws.
  • Because PoM is a general sequence-to-sequence block, the same mixer can replace attention in any transformer, including autoregressive language models, where causal masking yields O(1) per-token inference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the universality theorem holds, PoM-like state mixers could be used in place of attention in long-context and multimodal LLMs, where O(1) per-token inference and parallel training could reduce serving costs without a fixed raster scan—an extension the paper mentions but does not test.
  • The explicit state suggests a natural streaming-video setup: keep $H(X)$ fixed for past frames and only update the state with the new frame's polynomial expansion, which would give bounded memory for arbitrarily long videos; this follows from the recurrence in Eq. (9) but is not evaluated.
  • The paper's proof gap (the appendix's scalar argument does not cover the full matrix nonlinear case) implies the universality claim should be tested empirically by probing whether distinct inputs ever collide in state space; a practical consequence is that polynomial degree $k$ may need to grow with sequence length or dimension.
  • Since the mixer is permutation equivariant and length-agnostic, it may transfer to other set-structured tasks—point clouds, object sets, or graphs—where attention's pairwise cost is prohibitive and variable cardinality matters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces the Polynomial Mixer (PoM), a sequence-to-sequence block intended as a drop-in replacement for multi-head attention (MHA). PoM computes a fixed-size state by applying a degree-k polynomial expansion to each token and pooling across tokens, then gates this state per token with a sigmoid; the block has linear complexity in the number of tokens and can be trained in parallel while supporting sequential inference via causal or block-causal masks. The authors prove permutation equivariance, claim a universal approximation theorem for Polymorphers, and validate the approach by replacing MHA in DiT-style image and video diffusion models, reporting an ImageNet FID of 2.46, ablations over polynomial degree, scaling-law experiments, and VBench video results.

Significance. If fully established, the paper would make a meaningful contribution: a linear-complexity attention replacement with an explicit state, universal approximation guarantees, and competitive generative performance would be of broad interest. The paper has notable strengths: the code is released, the degree ablation and scaling-law experiments are informative, the block-causal video formulation is clearly described, and the authors are candid about limitations of their FID reference set and video training data. However, the central theoretical claim depends on Lemma 3, whose appendix proof contains a genuine logical gap, and the headline experimental comparisons are not directly comparable to the cited baselines. The current evidence therefore does not fully support the strong claims made in the abstract and Section 6.

major comments (4)
  1. [Appendix C, Lemma 3] The proof of Lemma 3 is not valid for the actual PoM defined in Eq. (3). It begins by removing the learned matrices and treating the weights as identity or full-rank, then replaces all activations with linear ones. This discards the sigmoid gating in Eq. (3) and the nonlinear activation h in Eq. (4), which are essential to the PoM operation. The statement that a similar argument holds for full-rank matrices and piecewise-linear activations is not demonstrated and does not preserve the polynomial structure used in the rest of the proof. Consequently, the contextual mapping property is not established for the true PoM.
  2. [Appendix C, Eq. (21)-(26)] The scalar argument after Eq. (24) is logically inverted. The proof assumes there exist x in X and x' in X' such that x S_k = x' S'_k and then claims this would force the ratio x/x' to be constant for all pairs. The assumption is existential, not universal, so the contradiction does not follow. A concrete counterexample to this step is X={2,3}, X'={1,9}, k=1, where S_k=5 and S'_k=10 and 2*5=1*10. Since Theorem 2 is inherited from [81] only through Lemma 3, the universal approximation claim is currently unsupported.
  3. [Table 1] The comparison between the reported FID of 2.46 and DiT's 2.27 is not meaningful as presented. The footnote states that DiT, SiT, and other marked methods are evaluated against the ImageNet training set, while the authors use the ADM evaluation archive. Using different reference sets can change FID substantially, and the paper itself notes that FID is highly sensitive to the reference set. Without re-evaluation on the same reference set and ideally with multiple seeds or error bars, the statement in Section 5.1 that the results are "on par with the literature" is not quantitatively supported.
  4. [Table 3] The video results are far below all reported published baselines on most VBench metrics, for example Multiple Objects (1.9% vs. 25.9-33.3%), Appearance Style (2.8% vs. 39.3-52.7%), and Overall Consistency (15.1% vs. 25.2-26.4%). The authors' disclaimer that the comparison is unfair because the baselines were trained on larger and richer datasets is honest, but it also directly undermines the claim in Section 1 that the video models do not sacrifice visual quality. The section should be reframed as a study of the block-causal mask and a proof of concept, not as evidence of competitive quality.
minor comments (4)
  1. [Section 6] There is a typo in "universal senquence-to-sequence approximator;" it should read "sequence-to-sequence."
  2. [Appendix C] In the proof of Lemma 3, the sentence "Assuming ker(Wo) =" is missing the right-hand side; it should be something like "ker(Wo) = {0}" or "ker(Wo) is trivial." Also, the notation x' is used both for an element of X' and for the set X' in Eq. (21), which is confusing.
  3. [Table 3] The entry "39/0%" for ModeScope's Multiple Objects appears to be a typo and should likely be "39.0%".
  4. [Figure 1] The timing plot shows a single measurement setup without error bars or details on batch size, token count, or number of warm-up iterations; reporting these details or repeated runs would strengthen the complexity claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PoM's universality claim is adapted from an external theorem and experiments are measured against independent benchmarks; the flagged Lemma 3 issue is a proof gap, not a reduction to inputs.

full rationale

No circularity found. The Polymorpher/PoM architecture is defined independently in Eqs. (3)-(7), and the universality claim in Theorem 2 is explicitly adapted from the external result of Yun et al. [81], with Lemma 3 serving as an original mathematical bridge rather than a restatement of the architecture's definition. The appendix proof of Lemma 3 may contain a logical gap (the step from existence of one pair with x S_k = x' S'_k to constancy of the ratio x/x' for all pairs is invalid), but that is a correctness risk, not a circularity: the theorem is not used to define PoM, and no fitted parameter is recycled as a prediction. The experimental FID, IS, and VBench numbers are measured against independent external benchmarks and evaluation code, and the scaling-law plot is an empirical fit rather than a 'prediction' derived from a fitted parameter. Self-citations such as [41] for high-order pooling, [60] for seed sensitivity, and [18] for coherence-aware training are background context and do not carry the load-bearing universality claim; the main external citation [81] is transparently used as the proof scheme. Therefore the derivation chain does not reduce to its own inputs, and the paper is not circular under the definitions used here.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claims rest on several hand-chosen hyperparameters (degree, expansion, block size, guidance, sampling steps) and on a theorem whose proof inherits a substantial external framework. The invented H(X) state is an architectural construct rather than an empirically discovered entity, so it carries no independent evidence outside the paper's own experiments.

free parameters (5)
  • Polynomial degree k = 2 for main image and video models
    Chosen via ablation in Table 2; a degree of at least 2 is needed for good performance, and the paper selects 2 without a full tuning study.
  • Order expansion factor = 2 for main models
    Fixed by design and stated as default; the paper does not tune it independently of degree.
  • Block size K for block-causal video mask = not reported
    The block-causal video experiment requires choosing a temporal block size, but the paper does not state the value or provide a sensitivity analysis.
  • Classifier-free guidance weight omega = 0.7 for ImageNet, 1.0 for scaling laws, 4s/s0 for high-resolution
    Guidance strength is chosen per experiment and strongly affects the FID/IS trade-off; it is a hand-tuned hyperparameter.
  • Sampling steps and sampler = 250 DDIM for diffusion loss, 125 Heun for flow matching
    Evaluation protocol choices that materially affect FID; they are selected per model rather than derived.
assumptions (5)
  • standard math The Yun et al. (2020) transformer universality scheme: if a layer provides contextual mapping, feed-forward networks can approximate any continuous sequence-to-sequence function on a compact domain.
    Theorem 2 is stated to follow this scheme; the paper does not reprove this background result.
  • standard math Power sums of distinct finite multisets of reals differ for some degree k.
    Appendix C uses P(k) != P'(k) to build a contextual mapping; this is plausible for positive reals but is not proved in the paper.
  • ad hoc to paper Activations can be replaced by piecewise linear approximations without losing the contextual mapping property.
    Appendix C states linear activations can be made as close as desired, but it does not show that the resulting PoM remains a contextual mapping for the actual GELU and sigmoid activations.
  • ad hoc to paper Weight matrices can be taken as identity or full-rank without loss of generality for the contextual mapping proof.
    The proof assumes identity matrices and states full-rank matrices preserve injectivity, but the trained PoM does not use identity weights and the extension is not demonstrated.
  • domain assumption The universal approximation theorem is for fixed sequence length n on a compact domain in R^{d x n}.
    Theorem 2 is stated for that setting; the paper does not prove approximation for arbitrary-length sequences, which matters for the variable-length video generation claim.
invented entities (1)
  • High-order state H(X), an explicit fixed-size summary of the entire token sequence
    purpose: Replaces pairwise attention by a learned global state that each token queries with a gate, enabling linear complexity.
    The state is a designed latent with no external measurable signature; its validity rests solely on in-paper benchmarks, and the paper provides no independent falsifiable prediction for it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PoM: Efficient Image and Video Generation with the Polynomial Mixer." pith.science (2026). https://pith.science/paper/X42MHDVB

@misc{pith2026241112663,
  author       = {Pith},
  title        = {Pith review of: PoM: Efficient Image and Video Generation with the Polynomial Mixer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X42MHDVB}},
  note         = {Machine review of arXiv:2411.12663}
}
read the original abstract

Diffusion models based on Multi-Head Attention (MHA) have become ubiquitous to generate high quality images and videos. However, encoding an image or a video as a sequence of patches results in costly attention patterns, as the requirements both in terms of memory and compute grow quadratically. To alleviate this problem, we propose a drop-in replacement for MHA called the Polynomial Mixer (PoM) that has the benefit of encoding the entire sequence into an explicit state. PoM has a linear complexity with respect to the number of tokens. This explicit state also allows us to generate frames in a sequential fashion, minimizing memory and compute requirement, while still being able to train in parallel. We show the Polynomial Mixer is a universal sequence-to-sequence approximator, just like regular MHA. We adapt several Diffusion Transformers (DiT) for generating images and videos with PoM replacing MHA, and we obtain high quality samples while using less computational resources. The code is available at https://github.com/davidpicard/HoMM.

Figures

Figures reproduced from arXiv: 2411.12663 by the authors.

Figure 1
Figure 1. Comparison between the speed of PoM and Multi￾Head Attention (MHA) in the same DiT-XL/2 architecture for different image resolutions. We use an H100 GPU and compute the average time on 100 synthetic training batches to perform the forward or forward+backward passes. We use synthetic data to remove the influence from data loading. Training with PoM is less costly than inference with MHA at higher resolutions. ating a… view at source ↗
Figure 2
Figure 2. Diagram for the Polynomial Mixer. The input se￾quence is split into two paths. The top path expands each token using a polynomial before they are mixed (averaged)² into a single representation. The bottom path expands the tokens into gating coefficients. Both paths are recombined and projected back into the input dimension. Contrarily to MHA that computes all pairwise exchanges of information between tokens in the s… view at source ↗
Figure 3
Figure 3. Building blocks for our diffusion models using PoM. For class-conditional image generation (a), we follow strictly DiT[58] in the AdaLN variant, replacing multi-head attention with PoM. For text to video generation (b), we follow a hybrid approach in which the encoded text tokens are incorporated into the video tokens using PoM instead of cross attention, while the time is used as a modulation. Modulation means comp… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Qualitative results on class-conditional generation. We show images sampled with the model DiPoM-XL/2 trained with the flow-matching loss LFM at several resolutions for different classes. We use classifier-free guidance with ω = 4s/s0 with s the scale of the image and …
Figure 5
Figure 5. Figure 5: Scaling laws for a DiT-like architecture with attention replaced by PoM. FIDs and Inception Scores (IS) are computed on 10k samples with classifier free guidance (ω = 1), and shown with a linear regression in log space. Performances scale with the computation budget, s…
Figure 6
Figure 6. Figure 6: Image quality versus condition adherence trade-off. FID/IS curve for the L2 model with 250 DDIM sampling steps. Values are computed on 10k images against the validation set of ImageNet. IS comes at the cost of FID. This is typical of mode collapse with the model genera…
Figure 7
Figure 7. Figure 7: Uncurated 256² images for the class magpie (18). 15 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Uncurated 256² images for the class loggerhead, loggerhead turtle, Caretta caretta (33). 16 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Uncurated 256² images for the class macaw (88). 17 [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Uncurated 256² images for the class otter (360). 18 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Uncurated 256² images for the class balloon (417). 19 [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Uncurated 256² images for the class ice cream, icecream (928). 20 [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Uncurated 256² images for the class seashore, coast, seacoast, sea-coast (978). 21 [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Uncurated 256² images for the class volcano (980). 22 [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

87 extracted references · 55 canonical work pages

  1. [81]

    Yun, C., Bhojanapalli, S., Rawat, A.S., Reddi, S., Kumar, S.: Are transformers universal approximators of sequence- to-sequence functions? In: ICLR (2020) 4

    work page 2020

  2. [1]

    arXiv preprint arXiv:2303.08774 (2023) 2

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023) 2

    arXiv 2023

  3. [2]

    In: CVPR (2024) 2

    Bai, X., Melas-Kyriazi, L.: Fixed point diffusion models. In: CVPR (2024) 2

    2024

  4. [3]

    In: ICCV (2021) 6, 8

    Bain, M., Nagrani, A., Varol, G., Zisserman, A.: Frozen in time: A joint video and image encoder for end-to-end re- trieval. In: ICCV (2021) 6, 8

    2021

  5. [4]

    arXiv preprint arXiv:2211.01324 (2022) 2

    Balaji, Y ., Nah, S., Huang, X., Vahdat, A., Song, J., Zhang, Q., Kreis, K., Aittala, M., Aila, T., Laine, S., Catanzaro, B., Karras, T., Liu, M.Y .: ediff-i: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 (2022) 2

    arXiv 2022

  6. [5]

    In: CVPR (2023) 2

    Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S.W., Fidler, S., Kreis, K.: Align Your Latents: High- Resolution Video Synthesis with Latent Diffusion Models . In: CVPR (2023) 2

    2023

  7. [6]

    arXiv preprint arXiv:2405.12981 (2024) 2

    Brandon, W., Mishra, M., Nrusimha, A., Panda, R., Kelly, J.R.: Reducing transformer key-value cache size with cross- layer attention. arXiv preprint arXiv:2405.12981 (2024) 2

    arXiv 2024

  8. [7]

    Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y ., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luh- man, E., Ng, C., Wang, R., Ramesh, A.: Video gen- eration models as world simulators (2024), https : //openai.com/research/video- generation- models-as-world-simulators 1

    2024

Show all 87 references
  1. [8]

    In: CVPR (2022) 6, 7

    Chang, H., Zhang, H., Jiang, L., Liu, C., Freeman, W.T.: Maskgit: Masked generative image transformer. In: CVPR (2022) 6, 7

    2022

  2. [9]

    In: ECCV (2024) 2

    Chen, J., Ge, C., Xie, E., Wu, Y ., Yao, L., Ren, X., Wang, Z., Luo, P., Lu, H., Li, Z.: Pixart-\sigma: Weak-to-strong train- ing of diffusion transformer for 4k text-to-image generation. In: ECCV (2024) 2

    2024

  3. [10]

    arXiv (2023) 2

    Chen, T., Li, L.: Fit: Far-reaching interleaved transformers. arXiv (2023) 2

    2023

  4. [11]

    arXiv preprint arXiv:1904.10509 (2019) 1, 2

    Child, R., Gray, S., Radford, A., Sutskever, I.: Generat- ing long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 (2019) 1, 2

    2019 arXiv

  5. [12]

    In: ICML (2024) 2

    Crowson, K., Baumann, S.A., Birch, A., Abraham, T.M., Kaplan, D.Z., Shippole, E.: Scalable high-resolution pixel- space image synthesis with hourglass diffusion transformers. In: ICML (2024) 2

    2024

  6. [13]

    arXiv preprint arXiv:2307.08691 (2023) 2

    Dao, T.: Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.08691 (2023) 2

    2023 arXiv

  7. [14]

    In: NeurIPS (2022) 2

    Dao, T., Fu, D., Ermon, S., Rudra, A., R´e, C.: Flashattention: Fast and memory-efficient exact attention with io-awareness. In: NeurIPS (2022) 2

    2022

  8. [15]

    Dao, T., Gu, A.: Transformers are ssms: Generalized models and efficient algorithms through structured state space dual- ity. In: Int. Conf. Mach. Learn. (2024) 3

    2024

  9. [16]

    In: NeurIPS (2021) 6, 7

    Dhariwal, P., Nichol, A.: Diffusion models beat gans on im- age synthesis. In: NeurIPS (2021) 6, 7

    2021

  10. [17]

    arXiv preprint arXiv:2407.21783 (2024) 2

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024) 2

    2024 arXiv

  11. [18]

    In: CVPR (2024) 2

    Dufour, N., Besnier, V ., Kalogeiton, V ., Picard, D.: Don’t drop your samples! coherence-aware training benefits con- ditional diffusion. In: CVPR (2024) 2

    2024

  12. [19]

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., M ¨uller, J., Saini, H., Levi, Y ., Lorenz, D., Sauer, A., Boesel, F., et al.: Scaling rectified flow transformers for high-resolution image synthesis. In: Int. Conf. Mach. Learn. (2024) 1, 2

    2024

  13. [20]

    arXiv preprint arXiv:2405.05945 (2024) 2

    Gao, P., Zhuo, L., Lin, Z., Liu, C., Chen, J., Du, R., Xie, E., Luo, X., Qiu, L., Zhang, Y ., et al.: Lumina-t2x: Trans- forming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.05945 (2024) 2

    2024 arXiv

  14. [21]

    In: ICCV (2023) 2

    Ge, S., Nah, S., Liu, G., Poon, T., Tao, A., Catanzaro, B., Jacobs, D., Huang, J.B., Liu, M.Y ., Balaji, Y .: Preserve your own correlation: A noise prior for video diffusion models. In: ICCV (2023) 2

    2023

  15. [22]

    In: ECCV (2024) 2

    Girdhar, R., Singh, M., Brown, A., Duval, Q., Azadi, S., Rambhatla, S.S., Shah, A., Yin, X., Parikh, D., Misra, I.: Factorizing text-to-video generation by explicit image con- ditioning. In: ECCV (2024) 2

    2024

  16. [23]

    arXiv preprint arXiv:2405.16712 (2024) 3

    Glorioso, P., Anthony, Q., Tokpanov, Y ., Whittington, J., Pi- lault, J., Ibrahim, A., Millidge, B.: Zamba: A compact 7b ssm hybrid model. arXiv preprint arXiv:2405.16712 (2024) 3

    2024 arXiv

  17. [24]

    In: CVPR (2024) 2

    Gokaslan, A., Cooper, A.F., Collins, J., Seguin, L., Jacob- son, A., Patel, M., Frankle, J., Stephenson, C., Kuleshov, V .: Commoncanvas: Open diffusion models trained on creative- commons images. In: CVPR (2024) 2

    2024

  18. [25]

    In: ICLR (2024) 1

    Gu, A., Dao, T.: Mamba: Linear-time sequence modeling with selective state spaces. In: ICLR (2024) 1

    2024

  19. [26]

    In: ICLR (2021) 1, 3

    Gu, A., Goel, K., Re, C.: Efficiently modeling long se- quences with structured state spaces. In: ICLR (2021) 1, 3

    2021

  20. [27]

    In: NeurIPS (2021) 1, 3

    Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., R´e, C.: Combining recurrent, convolutional, and continuous- time models with linear state space layers. In: NeurIPS (2021) 1, 3

    2021

  21. [28]

    In: ICLR (2023) 2

    Gu, J., Zhai, S., Zhang, Y ., Susskind, J.M., Jaitly, N.: Ma- tryoshka diffusion models. In: ICLR (2023) 2

    2023

  22. [29]

    In: ECCV (2025) 2

    Gupta, A., Yu, L., Sohn, K., Gu, X., Hahn, M., Li, F.F., Essa, I., Jiang, L., Lezama, J.: Photorealistic video generation with diffusion models. In: ECCV (2025) 2

    2025

  23. [30]

    In: NeurIPS (2024), http://arxiv.org/abs/2405.18392 6 9

    H ¨agele, A., Bakouch, E., Kosson, A., Allal, L.B., Werra, L.V ., Jaggi, M.: Scaling Laws and Compute-Optimal Train- ing Beyond Fixed Training Durations. In: NeurIPS (2024), http://arxiv.org/abs/2405.18392 6 9

    2024 arXiv

  24. [31]

    arXiv preprint arXiv:2407.03297 (2024) 2

    Hang, T., Gu, S.: Improved noise schedule for diffusion training. arXiv preprint arXiv:2407.03297 (2024) 2

    2024 arXiv

  25. [32]

    In: ECCV (2024) 2

    Hatamizadeh, A., Song, J., Liu, G., Kautz, J., Vahdat, A.: Diffit: Diffusion vision transformers for image generation. In: ECCV (2024) 2

    2024

  26. [33]

    arXiv preprint arXiv:2211.13221 (2022) 8

    He, Y ., Yang, T., Zhang, Y ., Shan, Y ., Chen, Q.: Latent video diffusion models for high-fidelity long video genera- tion. arXiv preprint arXiv:2211.13221 (2022) 8

    2022 arXiv

  27. [34]

    arXiv preprint arXiv:2210.02303 (2022) 2

    Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D.P., Poole, B., Norouzi, M., Fleet, D.J., et al.: Imagen video: High definition video generation with diffu- sion models. arXiv preprint arXiv:2210.02303 (2022) 2

    2022 arXiv

  28. [35]

    In: NeurIPS (2020) 2

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: NeurIPS (2020) 2

    2020

  29. [36]

    In: NeurIPS (2022) 2

    Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., Fleet, D.J.: Video diffusion models. In: NeurIPS (2022) 2

    2022

  30. [37]

    In: ICLR (2023) 2, 8

    Hong, W., Ding, M., Zheng, W., Liu, X., Tang, J.: Cogvideo: Large-scale pretraining for text-to-video gener- ation via transformers. In: ICLR (2023) 2, 8

    2023

  31. [38]

    In: ECCV (2024) 1, 3

    Hu, V .T., Baumann, S.A., Gui, M., Grebenkova, O., Ma, P., Fischer, J., Ommer, B.: Zigma: A dit-style zigzag mamba diffusion model. In: ECCV (2024) 1, 3

    2024

  32. [39]

    In: CVPR (2024) 8

    Huang, Z., He, Y ., Yu, J., Zhang, F., Si, C., Jiang, Y ., Zhang, Y ., Wu, T., Jin, Q., Chanpaisit, N., et al.: Vbench: Com- prehensive benchmark suite for video generative models. In: CVPR (2024) 8

    2024

  33. [40]

    Jabri, A., Fleet, D.J., Chen, T.: Scalable adaptive compu- tation for iterative generation. In: Int. Conf. Mach. Learn. (2023) 2, 7

    2023

  34. [41]

    In: ICCV (2019) 3

    Jacob, P., Picard, D., Histace, A., Klein, E.: Metric learning with horde: High-order regularizer for deep embeddings. In: ICCV (2019) 3

    2019

  35. [42]

    In: ICLR (2022) 2

    Jaegle, A., Borgeaud, S., Alayrac, J.B., Doersch, C., Ionescu, C., Ding, D., Koppula, S., Zoran, D., Brock, A., Shelhamer, E., et al.: Perceiver io: A general architecture for structured inputs & outputs. In: ICLR (2022) 2

    2022

  36. [43]

    arXiv preprint arXiv:2410.05954 (2024) 2

    Jin, Y ., Sun, Z., Li, N., Xu, K., Jiang, H., Zhuang, N., Huang, Q., Song, Y ., Mu, Y ., Lin, Z.: Pyramidal flow match- ing for efficient video generative modeling. arXiv preprint arXiv:2410.05954 (2024) 2

    2024

  37. [44]

    arXiv preprint arXiv:2001.08361 (2020) 1, 2

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D.: Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020) 1, 2

    2020 arXiv

  38. [46]

    In: CVPR (2024) 2

    Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., Laine, S.: Analyzing and improving the training dynamics of diffusion models. In: CVPR (2024) 2

    2024

  39. [47]

    In: ICLR (2020) 1, 2

    Kitaev, N., Kaiser, L., Levskaya, A.: Reformer: The efficient transformer. In: ICLR (2020) 1, 2

    2020

  40. [48]

    In: ECCV (2024) 2

    Kwon, M., Oh, S.W., Zhou, Y ., Liu, D., Lee, J.Y ., Cai, H., Liu, B., Liu, F., Uh, Y .: Harivo: Harnessing text-to-image models for video generation. In: ECCV (2024) 2

    2024

  41. [49]

    In: ECCV (2025) 2

    Lee, S.H., Li, Y ., Ke, J., Yoo, I., Zhang, H., Yu, J., Wang, Q., Deng, F., Entis, G., He, J., et al.: Parrot: Pareto-optimal multi-reward reinforcement learning framework for text-to- image generation. In: ECCV (2025) 2

    2025

  42. [50]

    arXiv preprint arXiv:2403.19887 (2024) 3

    Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedi- gos, I., Safahi, E., Meirom, S., Belinkov, Y ., Shalev-Shwartz, S., et al.: Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887 (2024) 3

    2024 arXiv

  43. [51]

    In: ICLR (2022) 2

    Lipman, Y ., Chen, R.T., Ben-Hamu, H., Nickel, M., Le, M.: Flow matching for generative modeling. In: ICLR (2022) 2

    2022

  44. [52]

    In: ICLR (2023) 2

    Liu, X., Gong, C., et al.: Flow straight and fast: Learning to generate and transfer data with rectified flow. In: ICLR (2023) 2

    2023

  45. [53]

    Liu, Y ., Tian, Y ., Zhao, Y ., Yu, H., Xie, L., Wang, Y ., Ye, Q., Liu, Y .: Vmamba: Visual state space model (2024) 3

    2024

  46. [54]

    In: CVPR (2024) 2

    Liu, Y ., Zhang, Y ., Jaakkola, T., Chang, S.: Correcting diffu- sion generation through resampling. In: CVPR (2024) 2

    2024

  47. [55]

    arXiv preprint arXiv:2407.18003 (2024) 2

    Luohe, S., Hongyi, Z., Yao, Y ., Zuchao, L., Hai, Z.: Keep the cost down: A review on methods to optimize llm’s kv-cache consumption. arXiv preprint arXiv:2407.18003 (2024) 2

    2024 arXiv

  48. [56]

    In: ECCV (2024) 2, 7

    Ma, N., Goldstein, M., Albergo, M.S., Boffi, N.M., Vanden- Eijnden, E., Xie, S.: Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In: ECCV (2024) 2, 7

    2024

  49. [57]

    Nichol, A.Q., Dhariwal, P.: Improved denoising diffusion probabilistic models. In: Int. Conf. Mach. Learn. (2021) 2

    2021

  50. [58]

    In: ICCV (2023) 1, 2, 5, 7, 8

    Peebles, W., Xie, S.: Scalable diffusion models with trans- formers. In: ICCV (2023) 1, 2, 5, 7, 8

    2023

  51. [59]

    Pei, X., Huang, T., Xu, C.: Efficientvmamba: Atrous selec- tive scan for light weight visual mamba (2024) 3

    2024

  52. [60]

    Picard, D.: Torch.manual seed(3407) is all you need: On the influence of random seeds in deep learning ar- chitectures for computer vision (2023) 7

    2023

  53. [61]

    arXiv preprint arXiv:2410.13720 (2024) 1

    Polyak, A., Zohar, A., Brown, A., Tjandra, A., Sinha, A., Lee, A., Vyas, A., Shi, B., Ma, C.Y ., Chuang, C.Y ., et al.: Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720 (2024) 1

    2024 arXiv

  54. [62]

    JMLR (2020) 5

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y ., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR (2020) 5

    2020

  55. [63]

    In: CVPR (2022) 2, 5, 7

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Om- mer, B.: High-resolution image synthesis with latent diffu- sion models. In: CVPR (2022) 2, 5, 7

    2022

  56. [64]

    In: NeurIPS (2024) 2

    Shi, Y ., De Bortoli, V ., Campbell, A., Doucet, A.: Diffusion schr¨odinger bridge matching. In: NeurIPS (2024) 2

    2024

  57. [65]

    In: CVPR (2024) 2

    Si, C., Huang, Z., Jiang, Y ., Liu, Z.: Freeu: Free lunch in diffusion u-net. In: CVPR (2024) 2

    2024

  58. [66]

    In: ICLR (2023) 2

    Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al.: Make-a- video: Text-to-video generation without text-video data. In: ICLR (2023) 2

    2023

  59. [67]

    In: International Conference on Learning Representations (2021) 2

    Song, Y ., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. In: International Conference on Learning Representations (2021) 2

    2021

  60. [68]

    arXiv preprint arXiv:2312.11805 (2023) 2 10

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A.M., Hauth, A., Millican, K., et al.: Gemini: a family of highly capable multimodal mod- els. arXiv preprint arXiv:2312.11805 (2023) 2 10

    2023 arXiv

  61. [69]

    arXiv preprint arXiv:2405.14224 (2024) 1, 7

    Teng, Y ., Wu, Y ., Shi, H., Ning, X., Dai, G., Wang, Y ., Li, Z., Liu, X.: Dim: Diffusion mamba for efficient high-resolution image synthesis. arXiv preprint arXiv:2405.14224 (2024) 1, 7

    2024 arXiv

  62. [70]

    In: NeurIPS (2021) 3

    Tolstikhin, I.O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al.: Mlp-mixer: An all-mlp architecture for vision. In: NeurIPS (2021) 3

    2021

  63. [71]

    IEEE TPAMI (2022) 3

    Touvron, H., Bojanowski, P., Caron, M., Cord, M., El- Nouby, A., Grave, E., Izacard, G., Joulin, A., Synnaeve, G., Verbeek, J., et al.: Resmlp: Feedforward networks for im- age classification with data-efficient training. IEEE TPAMI (2022) 3

    2022

  64. [72]

    In: NeurIPS (2017) 1, 2

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polosukhin, I.: Attention is all you need. In: NeurIPS (2017) 1, 2

    2017

  65. [73]

    In: ICLR (2022) 2

    Villegas, R., Babaeizadeh, M., Kindermans, P.J., Moraldo, H., Zhang, H., Saffar, M.T., Castro, S., Kunze, J., Erhan, D.: Phenaki: Variable length video generation from open domain textual descriptions. In: ICLR (2022) 2

    2022

  66. [74]

    In: CVPR (2024) 2

    Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Pu- rushwalkam, S., Ermon, S., Xiong, C., Joty, S., Naik, N.: Diffusion model alignment using direct preference optimiza- tion. In: CVPR (2024) 2

    2024

  67. [75]

    Wang, J., Yuan, H., Chen, D., Zhang, Y ., Wang, X., Zhang, S.: Modelscope text-to-video technical report (2023) 8

    2023

  68. [76]

    arXiv preprint arXiv:2006.04768 (2020) 1, 2

    Wang, S., Li, B.Z., Khabsa, M., Fang, H., Ma, H.: Lin- former: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 (2020) 1, 2

    2020 arXiv

  69. [77]

    arXiv preprint arXiv:2309.15103 (2023) 8

    Wang, Y ., Chen, X., Ma, X., Zhou, S., Huang, Z., Wang, Y ., Yang, C., He, Y ., Yu, J., Yang, P., et al.: Lavie: High- quality video generation with cascaded latent diffusion mod- els. arXiv preprint arXiv:2309.15103 (2023) 8

    2023 arXiv

  70. [78]

    In: ECCV (2024) 2

    Wei, F., Zeng, W., Li, Z., Yin, D., Duan, L., Li, W.: Pow- erful and flexible: Personalized text-to-image generation via reinforcement learning. In: ECCV (2024) 2

    2024

  71. [79]

    In: CVPR

    Yan, J.N., Gu, J., Rush, A.M.: Diffusion models without at- tention. In: CVPR. pp. 8239–8249 (2024) 1, 3, 7

    2024

  72. [80]

    arXiv preprint arXiv:2408.06072 (2024) 2, 5

    Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y ., Hong, W., Zhang, X., Feng, G., et al.: Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072 (2024) 2, 5

    2024 arXiv

  73. [82]

    In: CVPR (2022) 1, 2

    Zhai, X., Kolesnikov, A., Houlsby, N., Beyer, L.: Scaling vision transformers. In: CVPR (2022) 1, 2

    2022

  74. [83]

    In: ECCV (2024) 2

    Zhao, H., Lu, T., Gu, J., Zhang, X., Zheng, Q., Wu, Z., Xu, H., Jiang, Y .G.: Magdiff: Multi-alignment diffusion for high-fidelity video generation and editing. In: ECCV (2024) 2

    2024

  75. [84]

    In: ECCV (2024) 2

    Zhao, Y ., Xu, Y ., Xiao, Z., Jia, H., Hou, T.: Mobilediffu- sion: Instant text-to-image generation on mobile devices. In: ECCV (2024) 2

    2024

  76. [85]

    Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., Levy, O.: Transfusion: Predict the next token and diffuse images with one multi-modal model (2024) 8

    2024

  77. [86]

    In: CVPR (2024) 2

    Zhou, Z., Chen, D., Wang, C., Chen, C.: Fast ode-based sampling for diffusion models in around 5 steps. In: CVPR (2024) 2

    2024

  78. [87]

    arXiv preprint arXiv:2401.09417 (2024) 3

    Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., Wang, X.: Vision mamba: Efficient visual representation learn- ing with bidirectional state space model. arXiv preprint arXiv:2401.09417 (2024) 3

    2024 arXiv

  79. [88]

    arXiv preprint arXiv:2410.05355 (2024) 3 11 A

    Zuo, J., Velikanov, M., Rhaiem, D.E., Chahed, I., Belkada, Y ., Kunsch, G., Hacid, H.: Falcon mamba: The first com- petitive attention-free 7b language model. arXiv preprint arXiv:2410.05355 (2024) 3 11 A. PoM pytorch code In this section, we provide code in Pytorch for the ma...

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.