Pith. sign in

REVIEW 3 major objections 6 minor 104 references

A hybrid linear–global diffusion backbone plus module-wise hyperparameter transfer yields Chinchilla laws and multi-fold pretraining compute gains for long visual sequences.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 02:13 UTC pith:4DX6ML67

load-bearing objection Solid systems paper: hybrid NoPE visual DiT plus carefully controlled Chinchilla laws; the 1.7×/7.3× numbers are real FLOP-to-loss gains but not yet quality-calibrated across architectures. the 3 major comments →

arxiv 2607.28611 v1 pith:4DX6ML67 submitted 2026-07-30 cs.CV

Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers

Chongjian Ge , Hanwen Jiang , Tianyu Wang , Jiuxiang Gu , Yiran Xu , Ziwen Chen , Shaoteng Liu , Jing Shi
show 4 more authors
Yicong Hong Zefan Cai Hailin Jin Hao Tan
This is my paper
classification cs.CV
keywords visual diffusionhybrid attentionlinear attentionmixture-of-expertshyperparameter transferChinchilla scalinglong-context videono positional embeddings
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Visual generation is hitting sequence lengths where full self-attention becomes too expensive, and recipes built for language models do not transfer cleanly because images and video need local grid structure, bidirectional multimodal interaction, and a denoising objective. Chimera answers with a single raster-ordered stream of text, image, and video tokens that needs no positional embeddings: most layers use linear-complexity Kimi Delta Attention for long-range state tracking, a minority use Multi-head Latent Attention for direct global links, and modality-aware short convolutions supply local spatiotemporal structure. Sparse MoE, identity hyper-connections, and sandwich norms add capacity and stability at controlled activated compute. Because the backbone is heterogeneous, the authors introduce HeteroP, which transfers learning rates, init, and residual scales module by module from each tensor’s fan-in and depth, producing a fairly tuned model family. From that family they fit compute-optimal laws over activated size, training tokens, and image–video mix, then train an 11B/2B-activated model. On pretraining diffusion loss the dense backbone is about 1.7× as compute-efficient as a matched full-attention baseline and the full system about 7.3×; the same model, trained only on 5-second clips, generates 30-second video zero-shot with small FID degradation on the tail.

Core claim

Under matched pretraining diffusion loss, a dense Chimera backbone is roughly 1.7× as compute-efficient as a matched full-attention Wan-2.1 2B baseline, and the complete MoE + iHC + HeteroP system reaches about 7.3×; the same NoPE hybrid design extrapolates from 5-second training clips to 30-second videos without length-specific fine-tuning, with only about 6.5% FID degradation on the final five seconds. Fitted laws say image pretraining should split compute nearly evenly between activated model size and tokens, while video pretraining modestly favors model size at higher budgets.

What carries the argument

HeteroP: a module-wise hyperparameter-transfer scheme that assigns each tensor its own width multiplier from functional fan-in (and a shared depth multiplier), so proxy-tuned learning rates, initialization, weight decay, residual scales, and AdamW ε stay near-optimal across width and depth for a hybrid of KDA, MLA, short convolutions, MoE, iHC, and diffusion conditioning—yielding a comparable model family for Chinchilla-style fits.

Load-bearing premise

That fixed-objective pretraining diffusion loss is a fair, sampler-independent stand-in for generative knowledge, and that simple FLOP accounting still ranks hybrid linear–global MoE models correctly for compute-optimal allocation.

What would settle it

Train matched full-attention and Chimera models to the same diffusion loss and FLOP budget, then check whether the claimed 1.7×/7.3× efficiency and the Nopt/Dopt frontiers still hold on held-out generation metrics (FID/FVD, GenEval, DPG) and on independent FLOP measurements that fully cost KDA state updates, MLA, and MoE routing.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Long-context visual generators can drop explicit positional embeddings if order comes from a causal recurrent scan plus modality-aware short convolutions.
  • Compute-optimal image diffusion pretraining should grow activated parameters and visual tokens nearly in lockstep; video runs should lean slightly more toward model size as budgets rise.
  • Image-heavy mixtures remain compute-efficient early, then slowly admit more video as scale increases.
  • Module-wise fan-in transfer is a practical prerequisite for trustworthy scaling curves on heterogeneous diffusion backbones.
  • Zero-shot multi-minute video and higher-resolution generation become architectural targets rather than only fine-tuning targets.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If diffusion loss and downstream quality diverge under hybrid attention, the field may need a separate “quality-optimal” law alongside the loss-optimal Chinchilla fit.
  • The same NoPE + linear-state recipe could transfer to single-stream models that both understand and generate long multimodal context.
  • Weaker MoE gains than in language suggest visual denoising may need routing or expert designs that exploit timestep or spatial structure, not only sparsity.
  • Holding MoE sparsity and KV-compression fixed across scale leaves open whether those knobs should themselves follow HeteroP-style transfer.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. Chimera proposes a hybrid single-stream visual diffusion Transformer that packs text, image, and video tokens in a NoPE raster order, combining KDA (linear long-context state tracking), periodic MLA (global interaction), modality-aware short convolutions (local spatiotemporal structure), sparse MoE, identity hyper-connections, and sandwich normalization. To scale this heterogeneous stack, the authors introduce HeteroP, a module-wise µP/CompleteP-style transfer that sets per-tensor LR/init/WD/ε from functional fan-in and depth, then use the resulting family to fit Chinchilla-style laws over activated size N, visual tokens D, and image–video mixture. An 11B/2B-activated model is trained under those laws. Main empirical claims: under matched data and rectified-flow v-loss, dense Chimera reaches a fixed loss with ~1.7× fewer FLOPs than a Wan-2.1 2B full-attention baseline and the full MoE+iHC+HeteroP system with ~7.3× fewer; zero-shot length extrapolation from 5 s training clips to 30 s with only 6.5% FID rise on the final 5 s; and near-balanced N–D allocation for images (Nopt∝C^0.48–0.52) with a modest model-size tilt for video.

Significance. If the results hold, this is a substantial contribution on two fronts that the field currently lacks: (i) a carefully engineered hybrid linear/global backbone tailored to multimodal visual diffusion with a clean NoPE story and strong long-horizon extrapolation, and (ii) the first controlled Chinchilla-style analysis for a heterogeneous visual diffusion MoE, including an explicit image–video mixture axis and a demonstrated failure mode when transfer is omitted (SP biases exponents toward oversized/undertrained models). Strengths that should be credited include three mutually consistent scaling estimators, explicit HeteroP vs SP LR-stability sweeps, cumulative MoE/iHC/HeteroP ablations, a fixed-tail length-extrapolation protocol against Wan and Hunyuan, and unusually candid negative results (Muon underperforms; MoE gains only ~1.5×). The work is a credible foundation for long-context diffusion design and compute allocation even if some headline multipliers need sharper qualification.

major comments (3)
  1. [Sec. 5.5, Fig. 12, Abstract] Sec. 5.5 / Fig. 12 and the abstract define the 1.7× and 7.3× efficiency gains strictly as FLOPs to a shared rectified-flow v-loss of 0.149. That is a legitimate pretraining metric under a fixed objective (Eq. 1), but the manuscript never shows that equal v-loss implies comparable sample quality across operators (full MHA Wan vs KDA/MLA dense vs MoE+iHC). Downstream GenEval/DPG and length-extrapolation FID are reported only for final Chimera checkpoints, not for loss-matched baselines. Without at least one loss-matched quality panel (e.g., FID or GenEval at the common loss, or quality vs FLOPs curves), the cross-architecture ranking and the use of this loss for Nopt/Dopt allocation remain only partially validated. Please either add such a calibration or narrow the claim language so it cannot be read as generative-quality efficiency.
  2. [Abstract; Sec. 5.5; Fig. 19] Fig. 19 attributes the largest jump in the 7.3× factor to HeteroP, while Wan and Z-Image baselines are trained under standard parameterization without an equivalent transfer recipe. Chimera-dense (no MoE/iHC/HeteroP) already shows ~1.7×, which cleanly isolates architecture; the full-system multiplier conflates architecture, sparse capacity, and optimizer transfer. The abstract and efficiency discussion should report the decomposed factors side-by-side (dense arch vs +MoE vs +iHC vs +HeteroP) and avoid a single 7.3× headline that mixes tuning with architecture.
  3. [Sec. 4.2; Sec. 5.3] Sec. 4.2 asserts C≈6ND remains valid because KDA is O(N) and MLA uses compressed KV, so matmuls dominate. For a hybrid MoE backbone this is plausible but not demonstrated. A short empirical check—logged FLOPs vs 6ND across the width/depth family and vs the MHA/MLA counterpart at long sequences (cf. Fig. 18)—would confirm that the IsoFLOP and envelope frontiers are not distorted by attention or routing overheads that scale differently with N and L.
minor comments (6)
  1. [Sec. 5.3, Fig. 9] Video parametric/bootstrap intervals are wide ([0.47, 0.80] for the model-size exponent); the text already treats this cautiously, but Fig. 9 captions and the abstract’s “modestly favors model size” should explicitly flag the limited D dynamic range (0.7 decades).
  2. [Table 2; Sec. 5.5] Table 2 compares Chimera (600 H100-days) to models with much larger undisclosed or larger disclosed budgets; a column or footnote with approximate training compute where known (e.g., Z-Image-Turbo 12.4K H100-days) would make the efficiency narrative easier to interpret.
  3. [Sec. 3; Appendix A] Notation: L is used both for sequence length and for the diffusion loss; t is both scan index and (in places) diffusion time. A single clarifying sentence in Sec. 3 or Table 3 would help.
  4. [Fig. 7] Fig. 7c y-axis range differs from 7a/b/d, which exaggerates SP instability visually; use a shared scale or note the break.
  5. [Fig. 1, Fig. 9, Sec. 6] Typos/consistency: “Chichilla” in Fig. 9 watermark; “viusal” in Sec. 6; “min - 22 M ·59 M” formatting in Fig. 1; date “July 25th, 2026” is fine for arXiv but check camera-ready.
  6. [Sec. 3.4] The RoPE analysis (Sec. 3.3, App. C) is thorough and supports NoPE, but the main text could briefly state that mShortConv kernel sizes and causal temporal padding were ablated (only the causal vs non-causal claim is quantified).

Circularity Check

0 steps flagged

No significant circularity: efficiency multipliers and Chinchilla frontiers are empirical measurements and fits, not results forced by definition or self-citation.

full rationale

The paper’s load-bearing claims are architectural measurements and empirical scaling fits, not closed derivation loops. Compute-efficiency factors (1.7× dense / ~7.3× full vs Wan-2.1) are defined as FLOPs to a shared rectified-flow v-loss under matched data and objective (Fig. 12, Sec. 5.5); that is a stated metric, not a quantity derived from itself. Chinchilla-style laws fit ˆL(N,D)=E+A/N^a+B/D^b on a HeteroP-tuned model family and read Nopt(C), Dopt(C) under C≈6ND via three independent estimators (envelope, IsoFLOP, parametric surface)—standard Hoffmann-style methodology, not tautology. HeteroP transfers proxy-tuned base hyperparameters by functional fan-in; SP-without-HeteroP is an external control showing biased exponents, which strengthens rather than closes the loop. Length extrapolation and GenEval/DPG compare against external baselines (Wan, Hunyuan, FLUX, Z-Image). Citations (KDA, MLA, µP/CompleteP, Chinchilla) supply components or methodology, not author-unique theorems that forbid alternatives. Concerns that v-loss may not calibrate sample quality across operators are validity/correctness issues, not circularity under the stated criteria.

Axiom & Free-Parameter Ledger

6 free parameters · 7 axioms · 4 invented entities

Claims rest on standard diffusion/Transformer practice plus several modeling choices: diffusion loss as pretrain metric, C≈6ND, fixed structural ratios while scaling width/depth, infinite-data regime, and proprietary multimodal data. Free parameters are the usual fitted scaling coefficients and many architectural hyperparameters held fixed by design. Invented entities are named methods (Chimera, HeteroP, iHC, mShortConv), not physical objects.

free parameters (6)
  • Parametric loss surface {E,A,B,a,b} for image and video = image: E≈0.126,a≈0.315,b≈0.336; video: E≈0.124,a≈0.330,b≈0.394
    Five-parameter Chinchilla-style fit minimized with Huber loss over many initializations; exponents and irreducible loss are data-fitted, then used to prescribe Nopt(C), Dopt(C).
  • Envelope/IsoFLOP power-law exponents Nopt∝C^α, Dopt∝C^β = image α≈0.48–0.52; video α≈0.53–0.56
    Nonparametric frontiers fitted to winning checkpoints; central quantitative scaling claim.
  • Image:video mixture IsoFLOP surface optima = image-loss ratio slope ≈ -0.08 per compute decade
    Quadratic surfaces over size and ratio at fixed FLOPs; optimal sample ratio ~3:1–4:1 for image loss.
  • Proxy-tuned base LR η_base and AdamW hyperparameters = η_base=10^-3
    Tuned on 22M proxy then transferred; η_base=1e-3, betas (0.95,0.95), λ_base=0.01, etc.
  • Fixed architectural ratios (KDA:MLA 3:1, MoE 56/8, iHC M=4, MLA compression, patch factors) = 3:1; N=56,K=8; M=4
    Held constant across the scaling family so laws do not entangle structure; choices are design knobs, not derived.
  • Timestep noise shift s=√(L_vis/256) and logit-normal τ schedule = reference 256² bucket
    Hand-chosen training noise protocol affecting the loss being optimized and compared.
axioms (7)
  • domain assumption Under fixed diffusion objective and noise parameterization, training v-loss is a valid pretraining metric for architecture and scale comparison, preferable to FID/GenEval at this stage.
    Stated in Sec. 2 and used for all efficiency and scaling claims.
  • domain assumption Dominant training compute satisfies C≈6ND even for hybrid linear-attention + sparse MoE + periodic MLA.
    Sec. 4.2; authors argue matmuls dominate, but this is an accounting assumption for optima.
  • domain assumption Module-wise µP/CompleteP-style transfer (HeteroP) makes losses across widths/depths comparable enough to fit scaling laws.
    Core of Sec. 4.1; supported by LR sweeps but still an assumption about optimization dynamics.
  • domain assumption Rectified-flow v-prediction with frozen Wan VAE and umT5-XXL text encoder is an adequate generative front-end.
    Sec. 3.1 and 5.1 implementation; quality ceiling partly external to Chimera weights.
  • domain assumption Infinite-data / <1 epoch regime and deterministic loaders remove data-reuse confounds in scaling fits.
    Secs. 4.2, 5.1–5.3.
  • standard math Standard Transformer/MoE/linear-attention algebra and optimization (AdamW, sandwich RMSNorm, aux-loss-free balancing).
    Background methods cited throughout Sec. 3.
  • ad hoc to paper NoPE plus causal scan and modality-aware short conv fully replace RoPE’s position-selection, recency, and layout roles for visual diffusion.
    Motivated by RoPE analysis (Sec. 3.3) and ablations, but remains a design thesis tied to their stack.
invented entities (4)
  • Chimera hybrid backbone no independent evidence
    purpose: Single-stream text/image/video diffusion Transformer combining KDA, MLA, mShortConv, MoE, iHC.
    Named system under study; composition is the architectural claim.
  • HeteroP no independent evidence
    purpose: Per-tensor fan-in and depth hyperparameter transfer for heterogeneous diffusion modules.
    Enables the controlled model family and scaling laws; validated mainly via internal LR stability and SP contrast.
  • Identity Hyper-Connections (iHC) no independent evidence
    purpose: Simplified multi-stream residuals with identity stream transition for stable heterogeneous updates.
    Variant of mHC/HC; utility shown in ablations on their runs.
  • Modality-aware short convolution (mShortConv) no independent evidence
    purpose: 1D causal text / 3D causal visual local mixing under one KDA scan without PE.
    Key NoPE inductive bias; fused kernel speedups reported internally.

pith-pipeline@v1.2.0-daily-grok45 · 52650 in / 4445 out tokens · 84931 ms · 2026-07-31T02:13:31.592156+00:00 · methodology

0 comments
read the original abstract

Visual generation increasingly requires high-resolution images, long videos, and multimodal context, making the quadratic cost of full attention prohibitive. We introduce Chimera, a hybrid visual diffusion backbone with a principled scaling recipe. Chimera processes text, image, and video tokens in one raster-ordered stream without positional embeddings. It combines Kimi Delta Attention (KDA) for long-context state tracking with O(N) complexity, interleaved Multi-head Latent Attention (MLA) for direct global interaction, and modality-aware short convolutions for local spatiotemporal context. Sparse Mixture-of-Experts (MoE) layers expand capacity while controlling activated compute. To scale this heterogeneous architecture, we introduce HeteroP, a module-wise scheme that transfers hyperparameters across width and depth according to each tensor's functional fan-in and model depth. HeteroP yields a consistently tuned family used to fit Chinchilla-style compute-optimal laws for activated model size, training-token count, and image-video data ratio. Guided by these laws, we train an 11B-parameter Chimera with 2B activated parameters. Experiments show three results. First, measured by pretraining diffusion loss, the dense backbone is 1.7x as compute-efficient as a matched full-attention Wan-2.1 2B baseline, while the complete system reaches 7.3x. Second, without length-specific fine-tuning, Chimera extrapolates zero-shot from 5-second training clips to 30-second videos, with only 6.5% FID degradation in the last five seconds. Third, the fitted laws show that compute-optimal image pretraining divides compute nearly evenly between activated model size and training-token count, whereas video pretraining modestly favors model size at higher budgets. These results establish a foundation for designing and scaling efficient long-context diffusion architectures.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

104 extracted references · 49 linked inside Pith

  1. [1]

    Gqa: Training generalizedmulti-querytransformermodelsfrommulti-headcheckpoints

    Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalizedmulti-querytransformermodelsfrommulti-headcheckpoints. InProceedingsofthe2023ConferenceonEmpirical Methods in Natural Language Processing, pages 4895–4901, 2023

    2023

  2. [2]

    Physics of language models: Part 4.1, architecture design and the magic of canon layers.Advances in Neural Information Processing Systems, 38:42349–42369, 2026

    Zeyuan Allen-Zhu. Physics of language models: Part 4.1, architecture design and the magic of canon layers.Advances in Neural Information Processing Systems, 38:42349–42369, 2026

    2026

  3. [3]

    Round and round we go! what makes rotary positional encodings useful?arXiv preprint arXiv:2410.06205, 2024

    Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful?arXiv preprint arXiv:2410.06205, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    FLUX.2: Frontier Visual Intelligence.https://bfl.ai/blog/flux-2, 2025

    Black Forest Labs. FLUX.2: Frontier Visual Intelligence.https://bfl.ai/blog/flux-2, 2025

    2025

  5. [5]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Align your latents: High-resolution video synthesis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22563–22575, 2023

    2023

  7. [7]

    Video generation models as world simulators, 2024

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators, 2024. https://openai.com/research/video-generation-models-as-world-simulators

    2024

  8. [8]

    Z-image: An efficient image generation foundation model with single-stream diffusion transformer.arXiv preprint arXiv:2511.22699, 2025

    Huanqia Cai, Sihan Cao, Ruoyi Du, Peng Gao, Steven Hoi, Zhaohui Hou, Shijie Huang, Dengyang Jiang, Xin Jin, Liangchen Li, et al. Z-image: An efficient image generation foundation model with single-stream diffusion transformer.arXiv preprint arXiv:2511.22699, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    The effect of mini-batch noise on the implicit bias of adam.arXiv preprint arXiv:2602.01642, 2026

    Matias D Cattaneo and Boris Shigida. The effect of mini-batch noise on the implicit bias of adam.arXiv preprint arXiv:2602.01642, 2026

    Pith/arXiv arXiv 2026

  10. [10]

    Aligning visual foundation encoders to tokenizers for diffusion models

    Bowei Chen, Sai Bi, Hao Tan, He Zhang, Tianyuan Zhang, Zhengqi Li, Yuanjun Xiong, Jianming Zhang, and Kai Zhang. Aligning visual foundation encoders to tokenizers for diffusion models. InThe Fourteenth International Conference on Learning Representations, 2025

    2025

  11. [11]

    Diffusion forcing: Next-token prediction meets full-sequence diffusion.Advances in Neural Information Processing Systems, 37:24081–24125, 2024

    Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion.Advances in Neural Information Processing Systems, 37:24081–24125, 2024

    2024

  12. [12]

    Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. InInternational conference on learning representations, volume 2024, pages 57611–57640, 2024

    2024

  13. [13]

    Sana-video: Efficient video generation with block linear diffusion transformer.arXiv preprint arXiv:2509.24695, 2025

    Junsong Chen, Yuyang Zhao, Jincheng Yu, Ruihang Chu, Junyu Chen, Shuai Yang, Xianbang Wang, Yicheng Pan, Daquan Zhou, Huan Ling, et al. Sana-video: Efficient video generation with block linear diffusion transformer.arXiv preprint arXiv:2509.24695, 2025

    arXiv 2025

  14. [14]

    Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

    Pith/arXiv arXiv 1904

  15. [15]

    Conditional positional encodings for vision transformers

    Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers. InInternational Conference on Learning Representations, 2023

    2023

  16. [16]

    Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining.arXiv preprint arXiv:2304.09151, 2023

    Hyung Won Chung, Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, and Orhan Firat. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining.arXiv preprint arXiv:2304.09151, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1280–1297, 2024

    2024

  18. [18]

    Don’t be lazy: Completep enables compute-efficient deep transformers.Advances in Neural Information Processing Systems, 38:137707–137739, 2026

    Nolan Dey, Bin Zhang, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestness. Don’t be lazy: Completep enables compute-efficient deep transformers.Advances in Neural Information Processing Systems, 38:137707–137739, 2026. 32

    2026

  19. [19]

    Diffusion models beat gans on image synthesis.Advances in neural information processing systems, 34:8780–8794, 2021

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis.Advances in neural information processing systems, 34:8780–8794, 2021

    2021

  20. [20]

    Diffusion is spectral autoregression.https://sander.ai/2024/09/02/spectral-autoregression

    Sander Dieleman. Diffusion is spectral autoregression.https://sander.ai/2024/09/02/spectral-autoregression. html, September 2024. Blog post

    2024

  21. [21]

    Efficient-vdit: Efficient video diffusion transformers with attention tile.arXiv preprint arXiv:2502.06155, 2025

    Hangliang Ding, Dacheng Li, Runlong Su, Peiyuan Zhang, Zhijie Deng, Ion Stoica, and Hao Zhang. Efficient-vdit: Efficient video diffusion transformers with attention tile.arXiv preprint arXiv:2502.06155, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Cogview: Mastering text-to-image generation via transformers

    Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. Cogview: Mastering text-to-image generation via transformers. InAdvances in Neural Information Processing Systems, volume 34, pages 19822–19835, 2021

    2021

  23. [23]

    A mathematical framework for transformer circuits.Transformer Circuits Thread, 1(1):12, 2021

    NelsonElhage,NeelNanda,CatherineOlsson,TomHenighan,NicholasJoseph,BenMann,AmandaAskell,YuntaoBai,Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits.Transformer Circuits Thread, 1(1):12, 2021

    2021

  24. [24]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. InForty-first international conference on machine learning, 2024

    2024

  25. [25]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    2022

  26. [26]

    Scaling diffusion transformers to 16 billion parameters, 2024.URL https://arxiv

    Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, and Junshi Huang. Scaling diffusion transformers to 16 billion parameters, 2024.URL https://arxiv. org/abs/2407.116, 33, 2024

    2024

  27. [27]

    Why adam works better with beta1=beta2: The missing gradient scale invariance principle.arXiv preprint arXiv:2601.21739, 2026

    Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I Mestre, Manuel F Dolz, and Enrique S Quintana-Ortí. Why adam works better with beta1=beta2: The missing gradient scale invariance principle.arXiv preprint arXiv:2601.21739, 2026

    Pith/arXiv arXiv 2026

  28. [28]

    Geneval: Anobject-focusedframeworkforevaluatingtext-to-image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

    DhrubaGhosh,HannanehHajishirzi,andLudwigSchmidt. Geneval: Anobject-focusedframeworkforevaluatingtext-to-image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

    2023

  29. [29]

    Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    Scaling laws for autoregressive generative modeling.arXiv preprint arXiv:2010.14701, 2020

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling.arXiv preprint arXiv:2010.14701, 2020

    Pith/arXiv arXiv 2010

  31. [31]

    Deep learning scaling is predictable, empirically.arXiv preprint arXiv:1712.00409, 2017

    Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically.arXiv preprint arXiv:1712.00409, 2017

    Pith/arXiv arXiv 2017

  32. [32]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Pith/arXiv arXiv 2022

  33. [33]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. InAdvances in Neural Information Processing Systems, 2020

    2020

  34. [34]

    Imagen video: High definition video generation with diffusion models.arXiv preprint arXiv:2210.02303, 2022

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models.arXiv preprint arXiv:2210.02303, 2022

    Pith/arXiv arXiv 2022

  35. [35]

    Videodiffusionmodels

    JonathanHo,TimSalimans,AlexeyGritsenko,WilliamChan,MohammadNorouzi,andDavidJFleet. Videodiffusionmodels. Advances in neural information processing systems, 35:8633–8646, 2022

    2022

  36. [36]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 10, 2022

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, DDL Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 10, 2022

    Pith/arXiv arXiv 2022

  37. [37]

    Pushing the boundaries of state space models for image and video generation, 2025.https://arxiv.org/abs/2502.00972

    Yicong Hong, Long Mai, Yuan Yao, and Feng Liu. Pushing the boundaries of state space models for image and video generation, 2025.https://arxiv.org/abs/2502.00972

    Pith/arXiv arXiv 2025

  38. [38]

    Relic: Interactive video world model with long-horizon memory.arXiv preprint arXiv:2512.04040, 2025

    YicongHong,YiqunMei,ChongjianGe,YiranXu,YangZhou,SaiBi,YannickHold-Geoffroy,MikeRoberts,MatthewFisher, Eli Shechtman, et al. Relic: Interactive video world model with long-horizon memory.arXiv preprint arXiv:2512.04040, 2025

    arXiv 2025

  39. [39]

    Zigma: A dit-style zigzag mamba diffusion model

    Vincent Tao Hu, Stefan Andreas Baumann, Ming Gui, Olga Grebenkova, Pingchuan Ma, Johannes Fischer, and Björn Ommer. Zigma: A dit-style zigzag mamba diffusion model. InEuropean conference on computer vision, pages 148–166. Springer, 2024

    2024

  40. [40]

    Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024. 33

    Pith/arXiv arXiv 2024

  41. [41]

    Self forcing: Bridging the train-test gap in autoregressive video diffusion.Advances in Neural Information Processing Systems, 38:167283–167308, 2026

    Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion.Advances in Neural Information Processing Systems, 38:167283–167308, 2026

    2026

  42. [42]

    Muon: An optimizer for hidden layers in neural networks, 2024.URL https://kellerjordan

    Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024.URL https://kellerjordan. github. io/posts/muon, 6(3):4, 2024

    2024

  43. [43]

    Editverse: Unifying image and video editing and generation with in-context learning.arXiv preprint arXiv:2509.20360, 2025

    Xuan Ju, Tianyu Wang, Yuqian Zhou, He Zhang, Qing Liu, Nanxuan Zhao, Zhifei Zhang, Yijun Li, Yuanhao Cai, Shaoteng Liu, Daniil Pakhomov, Zhe Lin, Soo Ye Kim, and Qiang Xu. Editverse: Unifying image and video editing and generation with in-context learning.arXiv preprint arXiv:2509.20360, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  45. [45]

    Transformers are RNNs: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Francois Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. InProceedings of the International Conference on Machine Learning, 2020

    2020

  46. [46]

    Theimpactofpositional encoding on length generalization in transformers.Advances in Neural Information Processing Systems, 36:24892–24928, 2023

    AmirhosseinKazemnejad,InkitPadhi,KarthikeyanNatesanRamamurthy,PayelDas,andSivaReddy. Theimpactofpositional encoding on length generalization in transformers.Advances in Neural Information Processing Systems, 36:24892–24928, 2023

    2023

  47. [48]

    Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020

    Pith/arXiv arXiv 2006

  48. [49]

    Back to basics: Let denoising generative models denoise

    Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 36115–36125, 2026

    2026

  49. [50]

    Scalinglawsfordiffusiontransformers.arXivpreprintarXiv:2410.08184, 2024

    ZhengyangLiang,HaoHe,CeyuanYang,andBoDai. Scalinglawsfordiffusiontransformers.arXivpreprintarXiv:2410.08184, 2024

    arXiv 2024

  50. [51]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    Pith/arXiv arXiv 2022

  51. [52]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    Pith/arXiv arXiv 2024

  52. [53]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  53. [54]

    Deepseek-v3

    AixinLiu, AoxueMei, BangcaiLin, BingXue, BingxuanWang, BingzhengXu, BochaoWu, BoweiZhang, ChaofanLin, Chen Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556, 2025

    Pith/arXiv arXiv 2025

  54. [55]

    Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022

    Pith/arXiv arXiv 2022

  55. [56]

    Vmamba: Visual state space model.Advances in neural information processing systems, 37:103031–103063, 2024

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model.Advances in neural information processing systems, 37:103031–103063, 2024

    2024

  56. [57]

    Olmo hybrid: From theory to practice and back.arXiv preprint arXiv:2604.03444, 2026

    William Merrill, Yanhong Li, Tyler Romero, Anej Svete, Caia Costello, Pradeep Dasigi, Dirk Groeneveld, David Heineman, Bailey Kuehl, Nathan Lambert, et al. Olmo hybrid: From theory to practice and back.arXiv preprint arXiv:2604.03444, 2026

    Pith/arXiv arXiv 2026

  57. [58]

    Completed hyperparameter transfer across modules, width, depth, batch and duration.arXiv preprint arXiv:2512.22382, 2025

    Bruno Mlodozeniec, Pierre Ablin, Louis Béthune, Dan Busbridge, Michal Klein, Jason Ramapuram, and Marco Cuturi. Completed hyperparameter transfer across modules, width, depth, batch and duration.arXiv preprint arXiv:2512.22382, 2025

    arXiv 2025

  58. [59]

    Glide: Towardsphotorealisticimagegenerationandeditingwithtext-guideddiffusionmodels.arXivpreprintarXiv:2112.10741, 2021

    AlexNichol,PrafullaDhariwal,AdityaRamesh,PranavShyam,PamelaMishkin,BobMcGrew,IlyaSutskever,andMarkChen. Glide: Towardsphotorealisticimagegenerationandeditingwithtext-guideddiffusionmodels.arXivpreprintarXiv:2112.10741, 2021

    Pith/arXiv arXiv 2021

  59. [60]

    Improveddenoisingdiffusionprobabilisticmodels

    AlexanderQuinnNicholandPrafullaDhariwal. Improveddenoisingdiffusionprobabilisticmodels. InInternationalconference on machine learning, pages 8162–8171. PMLR, 2021

    2021

  60. [61]

    In-context learning and induction heads.arXiv preprint arXiv:2209.11895, 2022

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads.arXiv preprint arXiv:2209.11895, 2022

    Pith/arXiv arXiv 2022

  61. [62]

    In search of adam’s secret sauce.Advances in Neural Information Processing Systems, 38: 63404–63442, 2026

    Antonio Orvieto and Robert Gower. In search of adam’s secret sauce.Advances in Neural Information Processing Systems, 38: 63404–63442, 2026. 34

    2026

  62. [63]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, 2023

    2023

  63. [64]

    Smith, and Mike Lewis

    Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. InInternational Conference on Learning Representations, 2022

    2022

  64. [65]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21 (140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21 (140):1–67, 2020

    2020

  65. [66]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    Pith/arXiv arXiv 2022

  66. [67]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  67. [68]

    Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural information processing systems, 35:36479–36494, 2022

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural information processing systems, 35:36479–36494, 2022

    2022

  68. [69]

    Fast transformer decoding: One write-head is all you need.arXiv preprint arXiv:1911.02150, 2019

    Noam Shazeer. Fast transformer decoding: One write-head is all you need.arXiv preprint arXiv:1911.02150, 2019

    Pith/arXiv arXiv 1911

  69. [70]

    Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Noam Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Pith/arXiv arXiv 2002

  70. [71]

    Make-a-video: Text-to-video generation without text-video data.arXiv preprint arXiv:2209.14792, 2022

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data.arXiv preprint arXiv:2209.14792, 2022

    Pith/arXiv arXiv 2022

  71. [72]

    History-guidedvideodiffusion

    KiwhanSong,BoyuanChen,MaxSimchowitz,YilunDu,RussTedrake,andVincentSitzmann. History-guidedvideodiffusion. arXiv preprint arXiv:2502.06764, 2025

    Pith/arXiv arXiv 2025

  72. [73]

    Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

    Pith/arXiv arXiv 2011

  73. [74]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    2024

  74. [75]

    Retentivenetwork: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    YutaoSun,LiDong,ShaohanHuang,ShumingMa,YuqingXia,JilongXue,JianyongWang,andFuruWei. Retentivenetwork: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    Pith/arXiv arXiv 2023

  75. [76]

    Dsv: Exploiting dynamic sparsity to accelerate large-scale video dit training.arXiv preprint arXiv:2502.07590, 2025

    Xin Tan, Yuetao Chen, Yimin Jiang, Xing Chen, Kun Yan, Nan Duan, Yibo Zhu, Daxin Jiang, and Hong Xu. Dsv: Exploiting dynamic sparsity to accelerate large-scale video dit training.arXiv preprint arXiv:2502.07590, 2025

    arXiv 2025

  76. [77]

    Kimi k3: Open frontier intelligence, 2026.https://arxiv.org/abs/2607.24653

    Kimi Team. Kimi k3: Open frontier intelligence, 2026.https://arxiv.org/abs/2607.24653

    arXiv 2026

  77. [78]

    Kimi linear: An expressive, efficient attention architecture.arXiv preprint arXiv:2510.26692, 2025

    KimiTeam, Yu Zhang, ZongyuLin, XingchengYao, Jiaxi Hu, Fanqing Meng, ChengyinLiu, Xin Men, SonglinYang, Zhiyuan Li, et al. Kimi linear: An expressive, efficient attention architecture.arXiv preprint arXiv:2510.26692, 2025

    Pith/arXiv arXiv 2025

  78. [79]

    Qwen Team. Qwen3. 5-omni technical report.arXiv preprint arXiv:2604.15804, 2026

    Pith/arXiv arXiv 2026

  79. [80]

    Hunyuanvideo 1.5 technical report, 2025.https://arxiv.org/abs/2511

    Tencent Hunyuan Foundation Model Team. Hunyuanvideo 1.5 technical report, 2025.https://arxiv.org/abs/2511. 18870

    2025

  80. [81]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems, 2017

    2017

Showing first 80 references.