Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LazyDiT claims that about half the per-step compute of a diffusion transformer can be skipped by reusing cached outputs from the previous denoising step, with a small trained linear layer deciding when to skip.

desk verdict Useful cache-based DiT acceleration with credible matched-compute FID/latency gains, but the theoretical guarantees as written do not hold up and should be cut or rewritten. read the letter →

arxiv 2412.12444 v3 pith:A64OMOMP submitted 2024-12-17 cs.LG cs.AI

Xuan Shen , Zhao Song , Yufa Zhou , Bo Chen , Yanyu Li , Yifan Gong , Kai Zhang , Hao Tan
show 7 more authors
Jason Kuen Henghui Ding Zhihao Shu Wei Niu Pu Zhao Yanzhi Wang Jiuxiang Gu
This is my paper · ORCID
classification cs.LGcs.AI MSC 68T07
keywords diffusiontransformerslazycachinginferenceaccelerationDDIMsamplingsimilaritylowerboundlineargateapproximationImageNetgenerationmobiledeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LazyDiT claims that a diffusion transformer does not need to run every attention and feedforward module at every denoising step, because the outputs of these modules at consecutive steps are very similar. The paper derives a lower bound on that similarity, shows the similarity can be approximated by a linear function of the current input, and turns this into a trainable gate: a small linear layer decides per module whether to recompute or just reuse the previous step's cached output. At a 50 percent lazy ratio, LazyDiT matches or beats the DDIM sampler at equal compute, for example FID 2.70 versus 2.95 on DiT-XL/2 at 256 resolution. If the claim holds, roughly half the per-step compute of large diffusion transformers can be skipped without hurting sample quality, which matters for deploying these models on phones and GPUs.

What carries the argument

The load-bearing object is the lazy learning layer: a linear map $W^{\Phi}_l \in \mathbb{R}^{D\times 1}$ inserted before each MHSA and feedforward module, producing a scalar gate $s^{\Phi}_{l,t} = \operatorname{sigmoid}((Z^{\Phi}_{l,t} W^{\Phi}_l)\,1_N)$, trained with a lazy loss that rewards skipping. Its justification is the paper's three-step chain: Theorem 13 states that time- and condition-dependent scaling and shifting vectors can make consecutive-step inputs arbitrarily close; Theorem 18 uses Lipschitz continuity of attention and feedforward modules to convert that closeness into a high cosine-similarity lower bound; Theorem 19 uses a Taylor expansion around the current input to represent the similarity as an inner product with a weight matrix. During sampling, the gate replaces the module with the previous step's cached output when the similarity score exceeds 0.5.

What would settle it

Measure the actual cosine similarity $f(Y^{\Phi}_{l,t-1}, Y^{\Phi}_{l,t})$ for every MHSA and feedforward module of a pretrained DiT-XL/2 during 50-step DDIM sampling. If the average similarity is far from $1-\alpha$ (say, below 0.9 at any stage), or if the measured distance $\|Z^{\Phi}_{l,t-1} - Z^{\Phi}_{l,t}\|$ under the real fixed scaling and shifting parameters is not small, the theoretical premise of the paper fails even if the learned gates still work.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that the cosine similarity between consecutive-step outputs of an MHSA or feedforward module is bounded below by $1-\alpha$ with small $\alpha$ (Theorem 2 / Theorem 18), because scaling and shifting can bring consecutive inputs close and the modules are Lipschitz. It then shows that this similarity admits a linear approximation, $f(Y^{\Phi}_{l,t-1}, Y^{\Phi}_{l,t}) = \langle W^{\Phi}_l, Z^{\Phi}_{l,t}\rangle$ (Theorem 3 / Theorem 19), so a linear layer can predict when a module's output would be redundant. Feeding that prediction through a sigmoid yields a per-module skip decision: recompute when $s \le 0.5$, reuse the cached previous-step output when $s > 0.5$. With the original model frozen and only these small linear layers trained for 500 steps, LazyDiT improves FID over DDIM at matched or lower TMACs (2.70 vs 2.95 at 2.87 TMACs on DiT-XL/2), and similar gains appear on Large-DiT-3B/7B and on mobile latency.

Load-bearing premise

The load-bearing premise is that after the model's scaling and shifting, the inputs to a module at consecutive denoising steps are close enough that the outputs are nearly identical. The proof constructs scaling and shifting vectors directly from the hidden states, but the real model obtains them as fixed linear functions of the timestep and class embedding, and the paper never shows that those real vectors satisfy the required closeness bound.

Editorial extensions

If this is right

  • At a 50% lazy ratio, LazyDiT reaches better FID than DDIM at the same TMACs (2.70 vs 2.95 on DiT-XL/2 at 256x256), meaning users can keep 50 sampling steps but pay the compute of a 25-step run.
  • In few-step regimes the gap is larger: a 10-step LazyDiT run with 30% lazy ratio beats a 7-step DDIM at similar or less compute (FID 17.05 vs 34.14 at 256x256).
  • The gains transfer across model families and resolutions: DiT-XL/2, DiT-L/2, Large-DiT-3B and Large-DiT-7B all match or beat DDIM at equal cost, with more redundancy in the billion-parameter models.
  • On a Snapdragon 8 Gen 3 phone, LazyDiT delivers a better inception score than DDIM at nearly identical latency, e.g. 136.81 vs 91.67 at 0.80 TMACs, so the method is directly deployable on mobile.
  • Layer-wise lazy-ratio patterns show no layer is ever fully skipped, and MHSA redundancy is concentrated in early layers while feedforward redundancy is concentrated in later layers, implying that layer pruning would be a worse strategy than per-step caching.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implicit consequence the paper does not develop: if consecutive-step similarity is as high as claimed, lazy skipping should compose with step-distillation and fast ODE solvers, potentially cutting the 10-step regime further.
  • The learned gate could be analyzed as a function of timestep and layer; a testable extension is to see whether gate decisions form a stable schedule that transfers across resolutions and classes without retraining.
  • The theory's construction of scaling and shifting vectors from the hidden states themselves suggests the empirical similarity should be measured directly on pretrained models; if it is high, a non-learned threshold schedule could obtain similar speedups without any training.
  • Because the linear approximation carries a Taylor remainder, the strongest practical headroom may sit at moderate lazy ratios, and the uniform 0.5 threshold may be suboptimal; per-layer thresholds could squeeze out more skips.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. LazyDiT inserts lightweight trained linear gates before each MHSA and feedforward module of DiT/Large-DiT; at inference, a module is skipped and replaced by its previous-timestep cache whenever the gate's predicted similarity exceeds 0.5. The paper claims a theoretical guarantee that consecutive module outputs are highly similar and that the similarity is linearly approximable, and supports this with a lazy-loss training procedure. Empirically, on ImageNet with cfg=1.5 and 50k generated images, LazyDiT at 20-50% lazy ratios achieves FID/IS better than DDIM at similar or lower TMACs across DiT-XL/2 (256/512), DiT-L/2, Large-DiT-3B, and Large-DiT-7B, and mobile latency is profiled on a Snapdragon 8 Gen 3 device.

Significance. If the empirical results are taken at face value, the contribution is practically significant: it is a cheap, post-hoc cache-skip method that improves the FID-vs-compute tradeoff of strong diffusion transformers, and the authors provide unusually detailed mobile latency measurements, a comparison against Learn2Cache, and public code. The evaluation is anchored to the external ImageNet-FID benchmark and the lazy ratios are measured after training, so the central system claim is not circular. However, the theoretical support in Section 3.2 and Appendix C is not established: the existence proofs are not tied to DiT's AdaLN parameterization, the formal similarity bound is vacuous without control of R2, and the implemented linear gate differs from the Taylor-derived weight. The paper would be stronger if the theory were either repaired or repositioned as empirical motivation rather than as proof.

major comments (4)
  1. [Appendix C.2, Lemma 12 and Theorem 13] The proof constructs uniform scaling vectors a and b from the hidden states X1 and X2, with c = 0, but DiT's actual scaling and shifting are at = W_{l,a} y_t + v_{l,a} and bt = W_{l,b} y_t + v_{l,b}, where y_t = SiLU(emb(t)+emb(c)) depends only on the timestep and class embedding (Section 3.2). The paper never shows that the pretrained AdaLN projection matrices and embeddings can realize these constructed a, b, or that a single y_t can simultaneously satisfy the scaling and shifting conditions. Therefore Theorem 1, and through Eq. (4) also Theorem 2, does not apply to the actual model as written.
  2. [Appendix C.4, Theorem 18] The formal similarity bound defines α := 0.5 C^2 R2^2 min{N,D}, where R2 is an assumed upper bound on ||X^Φ_{l,t-1} - X^Φ_{l,t}||, but no argument shows R2 is small. Moreover, Eq. (4) bounds Dist by C||Z^Φ_{l,t-1} - Z^Φ_{l,t}||, whereas Theorem 18 bounds the distance in terms of ||X^Φ_{l,t-1} - X^Φ_{l,t}||; the informal Theorem 2 in the main text combines the Z-distance bound with Theorem 1, so the formal and informal statements do not match. As written, the theorem is vacuous unless an additional control on R2 is supplied.
  3. [Appendix C.4 and C.5, unit-norm assumption] Both Theorem 18, Eq. (8), and Theorem 19, second equality, use the assumption ||Y^Φ_{l,t}||_F = 1 for every t. This normalization is not established for the actual DiT modules, and the implementation does not appear to enforce it. Without this assumption the cosine similarity is not equal to 1 - 0.5||Y_{t-1} - Y_t||^2, so the derived lower bound and the linear approximation do not follow.
  4. [Appendix C.5, Theorem 19] The proof defines W^Φ_l := (Y^Φ_{l,t-1})^T J, which is an N x D matrix that depends on the timestep t, while the implemented gate is a D x 1 weight vector shared across all timesteps (Section 3.3, s^Φ_{l,t} = sigmoid((Z^Φ_{l,t} · W^Φ_l) · 1_N)). In addition, the Taylor expansion in Eq. (9) contains an O(1) error term that is dropped in the main-text Theorem 3 equality. Consequently, the paper has not demonstrated that the implemented linear layer approximates cosine similarity.
minor comments (4)
  1. [Section 3.3, after Eq. (5)] The text says 'lazy ratio ΓΦ ∈ ZB of MSHA or Feedforward'; 'MSHA' should be 'MHSA'.
  2. [Section 3.2, Fact 7 paraphrase] The statement 'f (Y_{l,t-1}, Y_{l,t}) = 1 - Dist/2' is dimensionally inconsistent; the actual relation from Fact 7 is f = 1 - 0.5||Y_{t-1} - Y_t||^2_F when both matrices have unit Frobenius norm. Please correct this in the main text.
  3. [Section 4.1, Model Family] The citation 'LargeDiT (Zhang et al. 2023)' points to LLaMA-Adapter, which is not the source of the Large-DiT models used in the experiments; the reference appears mismatched and should be corrected.
  4. [Tables 1 and 2] The matched-compute comparison is central to the empirical claim, but Tables 1 and 2 do not report TMACs; only Table 3 reports TMACs for a subset of configurations. Adding TMAC columns to the main tables would make the comparison transparent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical results are externally benchmarked; theoretical gaps are correctness risks rather than circular reductions.

full rationale

The central empirical claim—LazyDiT beating DDIM at matched TMACs on ImageNet FID—is anchored to external benchmarks and to measured post-training lazy ratios, so it does not reduce to a fitted parameter or to the paper's own definitions. The only load-bearing citation from a coauthor's prior work is Lemma H.5 of (Deng et al. 2023), restated as Lemma 14; it is a parameter-free Lipschitz bound with explicit assumptions (∥W∥, ∥W_V∥, ∥X∥ ≤ R), so under the review rules it counts as independent mathematical support rather than circular self-citation. The theoretical weaknesses identified by the skeptical reading—Theorem 18's bound depends on an unquantified R2, Lemma 12 constructs scalings outside DiT's AdaLN parameterization, and Theorem 19 drops the O(1) Taylor error—are gaps between the informal and formal statements and matters of correctness or rigor, not cases where the conclusion is assumed as an input or a prediction is forced by construction. Theorem 19's W is defined from the Jacobian, making the existence claim a Taylor-theorem tautology, but the trained gate weights are learned against the diffusion and lazy losses and are validated by external FID, so the empirical result is not circular. No self-definitional reduction, no fitted-input-called-prediction, and no load-bearing self-citation chain was found.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The practical system rests on trained gate weights and three hand-set hyperparameters. The theoretical narrative rests on four unverified assumptions: input-adaptive scalings matching DiT's actual AdaLN, unit-norm outputs, bounded Lipschitz constants with no usable bound, and a Taylor remainder treated as negligible. No new physical entities are introduced.

free parameters (3)
  • Skip threshold 0.5 = 0.5 (fixed)
    The decision to skip uses a hand-set threshold on the sigmoid gate output; no sensitivity analysis is given.
  • Lazy penalty ratios rho_attn, rho_feed = tuned over 1e-7 to 1e-2, kept identical
    These hyperparameters control the achieved lazy ratio in each reported experiment; different rows in Tables 1-2 use different settings.
  • Lazy layer training budget = 500 steps, lr 1e-4, batch 256
    The linear gate weights are fit to ImageNet with this fixed budget; the resulting gate quality is part of the reported results.
assumptions (4)
  • ad hoc to paper DiT's actual timestep- and class-dependent AdaLN scaling/shifting parameters satisfy the closeness bound proved for input-adaptive scaling vectors in Lemma 12.
    Theorem 13 only proves existence of input-dependent a,b, but the architecture computes a_t,b_t from y_t = SiLU(emd(t)+emd(c)); the paper does not prove these coincide.
  • ad hoc to paper Module outputs have unit Frobenius norm, ||Y_Phi_{l,t}||_F = 1 for all layers and steps.
    Assumed in Theorems 18 and 19 to convert cosine similarity to 1 - 0.5||A-B||_F^2; hidden states are not unit norm in real DiT runs.
  • domain assumption Attention and feedforward modules are Lipschitz with constants C = 5R^4 N D and C = R under bounded spectral norms R.
    Taken from Lemma H.5 of Deng et al. 2023 and assumed bounded norms; the attention constant grows with N D, making the lower bound vacuous in practice.
  • domain assumption Taylor expansion of the module output around zero with O(1) remainder is a valid linear approximation of the similarity function.
    The hidden states Z_{l,t} are not near zero, so the remainder need not be small; the proof treats O(1) as negligible without bounding it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers." pith.science (2026). https://pith.science/paper/A64OMOMP

@misc{pith2026241212444,
  author       = {Pith},
  title        = {Pith review of: LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A64OMOMP}},
  note         = {Machine review of arXiv:2412.12444}
}
read the original abstract

Diffusion Transformers have emerged as the preeminent models for a wide array of generative tasks, demonstrating superior performance and efficacy across various applications. The promising results come at the cost of slow inference, as each denoising step requires running the whole transformer model with a large amount of parameters. In this paper, we show that performing the full computation of the model at each diffusion step is unnecessary, as some computations can be skipped by lazily reusing the results of previous steps. Furthermore, we show that the lower bound of similarity between outputs at consecutive steps is notably high, and this similarity can be linearly approximated using the inputs. To verify our demonstrations, we propose the \textbf{LazyDiT}, a lazy learning framework that efficiently leverages cached results from earlier steps to skip redundant computations. Specifically, we incorporate lazy learning layers into the model, effectively trained to maximize laziness, enabling dynamic skipping of redundant computations. Experimental results show that LazyDiT outperforms the DDIM sampler across multiple diffusion transformer models at various resolutions. Furthermore, we implement our method on mobile devices, achieving better performance than DDIM with similar latency. Code: https://github.com/shawnricecake/lazydit

Figures

Figures reproduced from arXiv: 2412.12444 by the authors.

Figure 1
Figure 1. Image generated by DiT-XL/2 in 512×512 and 256×256 resolutions when lazily skipping 50% computation. The upper rows display results from original model and the lower rows showcase outcomes of our method. Our method generates distinct lighting effects for background and color compared to the baseline, as demonstrated in dog and marmot, respectively. proach designed to dynamically reduce computational costs and accele… view at source ↗
Figure 2
Figure 2. Overview framework. We skip the computation of [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Visualization for the laziness in MHSA and Feedforward at each layer generated through DDIM 20 steps on DiT-XL. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Upper figure: ablation for the generation perfor [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visualization for the strategies of skipping MHSA or Feedforward only. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Visualization with 512×512 resolution with DDIM and our method with 50% lazy ratio [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depths

    cs.CV 2025-02 conditional novelty 6.0 of 10

    UniMoD prunes tokens with task-specific routers in unified multimodal transformers, cutting training FLOPs by 15-40% while roughly maintaining benchmark performance.

  2. High-Order Matching for One-Step Shortcut Diffusion Models

    cs.CV 2025-02 reject novelty 4.0 of 10

    HOMO extends shortcut diffusion with acceleration and jerk supervision, but the proof of superior approximation is not supported and experiments lack error bars.

  3. RichSpace: Enriching Text-to-Video Prompt Space via Text Embedding Interpolation

    cs.CV 2025-01 reject novelty 3.0 of 10

    Picking an interpolated text embedding with cosine similarity to a perpendicular foot can make a text-to-video model generate composite-feature videos that direct prompts miss.

Reference graph

Works this paper leans on

162 extracted references · 79 linked inside Pith · cited by 3 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; et al. 2016. TensorFlow: A system for large-scale machine learning. In OSDI 2016, 265--283. USA: USENIX Association

    2016

  4. [4]

    Alman, J.; and Song, Z. 2024. How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computation. In The Twelfth International Conference on Learning Representations

    2024

  5. [5]

    A.; and Wainwright, M

    Amini, A. A.; and Wainwright, M. J. 2009. High-dimensional analysis of semidefinite relaxations for sparse principal components. The Annals of Statistics, 37(5B)

    2009

  6. [6]

    Anstreicher, K. M. 2000. The Volumetric Barrier for Semidefinite Programming. Math. Oper. Res., 25(3): 365–380

    2000

  7. [7]

    Bao, F.; Li, C.; Zhu, J.; and Zhang, B. 2022. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201.06503

    arXiv 2022

  8. [8]

    Bao, F.; Nie, S.; Xue, K.; Cao, Y.; Li, C.; Su, H.; and Zhu, J. 2023. All are worth words: A vit backbone for diffusion models. In CVPR, 22669--22679

    2023

Show all 162 references
  1. [9]

    L.; Bousquet, O.; and Mendelson, S

    Bartlett, P. L.; Bousquet, O.; and Mendelson, S. 2005. Local rademacher complexities

    2005

  2. [10]

    Belinkov, Y. 2022. Probing Classifiers: Promises, Shortcomings, and Advances . Computational Linguistics, 48(1): 207--219

    2022

  3. [11]

    Bian, S.; Song, Z.; and Yin, J. 2023. Federated Empirical Risk Minimization via Second-Order Method. arXiv preprint arXiv:2305.17482

    2023 arXiv

  4. [12]

    Bottou, L.; and Bousquet, O. 2007. The tradeoffs of large scale learning. NeurIPS

    2007

  5. [13]

    Brand, J. v. d. 2020. A deterministic linear program solver in current matrix multiplication time. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms, 259--278. SIAM

    2020

  6. [14]

    Brand, J. v. d.; Lee, Y. T.; Sidford, A.; and Song, Z. 2020. Solving tall dense linear programs in nearly linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, 775--788

    2020

  7. [15]

    Brand, J. v. d.; Song, Z.; and Zhou, T. 2023. Algorithm and hardness for dynamic attention maintenance in large language models. arXiv preprint arXiv:2304.02207

    2023 arXiv

  8. [16]

    Castells, T.; Song, H.-K.; Kim, B.-K.; and Choi, S. 2024. LD-Pruner: Efficient Pruning of Latent Diffusion Models using Task-Agnostic Insights. arXiv:2404.11936

    2024 arXiv

  9. [17]

    Chang, C.-C.; and Lin, C.-J. 2001. Training v-support vector classifiers: theory and algorithms. Neural computation, 13(9): 2119--2147

    2001

  10. [18]

    Chen, B.; Li, X.; Liang, Y.; Long, J.; Shi, Z.; and Song, Z. 2024 a . Circuit Complexity Bounds for RoPE-based Transformer Architecture. arXiv preprint arXiv:2411.07602

    2024 arXiv

  11. [19]

    Chen, B.; Li, X.; Liang, Y.; Shi, Z.; and Song, Z. 2024 b . Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent. arXiv preprint arXiv:2410.11268

    2024 arXiv

  12. [20]

    Chen, B.; Liang, Y.; Sha, Z.; Shi, Z.; and Song, Z. 2024 c . HSR-Enhanced Sparse Attention Acceleration. arXiv preprint arXiv:2410.10165

    2024 arXiv

  13. [21]

    Chen, S.; Xu, M.; Ren, J.; et al. 2023. Gentron: Delving deep into diffusion transformers for image and video generation. arXiv preprint arXiv:2312.04557

    2023 arXiv

  14. [22]

    Chen, T.; Moreau, T.; Jiang, Z.; et al. 2018. TVM: An automated end-to-end optimizing compiler for deep learning. In OSDI 2018, 578--594

    2018

  15. [23]

    Chen, Y.; Zhong, R.; Zha, S.; Karypis, G.; and He, H. 2021. Meta-learning via language model in-context tuning. arXiv preprint arXiv:2110.07814

    2021 arXiv

  16. [24]

    Clark, K.; Khandelwal, U.; Levy, O.; and Manning, C. D. 2019. What Does BERT Look At? An Analysis of BERT's Attention. arXiv:1906.04341

    2019 arXiv

  17. [25]

    B.; Lee, Y

    Cohen, M. B.; Lee, Y. T.; and Song, Z. 2021. Solving linear programs in the current matrix multiplication time. Journal of the ACM (JACM), 68(1): 1--39

    2021

  18. [26]

    E.; Jordan, M

    d'Aspremont, A.; Ghaoui, L. E.; Jordan, M. I.; and Lanckriet, G. R. G. 2006. A direct formulation for sparse PCA using semidefinite programming. arXiv:cs/0406021

    2006 arXiv

  19. [27]

    D \'e fossez, A.; and Bach, F. 2014. Constant step size least-mean-square: Bias-variance trade-offs and optimal sampling distributions. arXiv preprint arXiv:1412.0156

    2014 arXiv

  20. [28]

    Deng, Y.; Li, Z.; and Song, Z. 2023. Attention scheme inspired softmax regression. arXiv preprint arXiv:2304.10411

    2023 arXiv

  21. [29]

    Deng, Y.; Mahadevan, S.; and Song, Z. 2023. Randomized and deterministic attention sparsification algorithms for over-parameterized feature dimension. arXiv preprint arXiv:2304.04397

    2023 arXiv

  22. [30]

    Deng, Y.; Song, Z.; Xie, S.; and Yang, C. 2023. Unmasking transformers: A theoretical approach to data recovery via attention weights. arXiv preprint arXiv:2310.12462

    2023 arXiv

  23. [31]

    Dhariwal, P.; and Nichol, A. 2021. Diffusion models beat gans on image synthesis. NeurIPS

    2021

  24. [32]

    Diakonikolas, I.; Kamath, G.; Kane, D.; Li, J.; Moitra, A.; and Stewart, A. 2019. Robust Estimators in High Dimensions without the Computational Intractability. arXiv:1604.06443

    2019 arXiv

  25. [33]

    T.; and Ye, G

    Dong, S.; Lee, Y. T.; and Ye, G. 2023. A Nearly-Linear Time Algorithm for Linear Programs with Small Treewidth: A Multiscale Representation of Robust Central Path. arXiv:2011.05365

    2023 arXiv

  26. [34]

    B.; and Li, J

    Dong, Y.; Hopkins, S. B.; and Li, J. 2019. Quantum Entropy Scoring for Fast Robust Mean Estimation and Improved Outlier Detection. arXiv:1906.11366

    2019 arXiv

  27. [35]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; et al. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010.11929

    2021 arXiv

  28. [36]

    Du, M.; He, F.; Zou, N.; Tao, D.; and Hu, X. 2023. Shortcut learning of large language models in natural language understanding. Communications of the ACM, 67(1): 110--120

    2023

  29. [37]

    Fan, A.; Grave, E.; and Joulin, A. 2019. Reducing Transformer Depth on Demand with Structured Dropout. arXiv:1909.11556

    2019 arXiv

  30. [38]

    Fang, G.; Ma, X.; and Wang, X. 2023. Structural pruning for diffusion models. In NeurIPS

    2023

  31. [39]

    Feldman, V.; Guruswami, V.; Raghavendra, P.; and Wu, Y. 2012. Agnostic learning of monomials by halfspaces is hard. SIAM Journal on Computing, 41(6): 1558--1590

    2012

  32. [40]

    M.; and Sidford, A

    Frostig, R.; Ge, R.; Kakade, S. M.; and Sidford, A. 2015. Competing with the empirical risk minimizer in a single pass. In Conference on learning theory, 728--763. PMLR

    2015

  33. [41]

    Gao, Y.; Mahadevan, S.; and Song, Z. 2023. An over-parameterized exponential regression. arXiv preprint arXiv:2303.16504

    2023 arXiv

  34. [42]

    Gao, Y.; Song, Z.; Wang, W.; and Yin, J. 2023 a . A fast optimization view: Reformulating single layer attention in llm based on tensor and svm trick, and solving it in matrix multiplication time. arXiv preprint arXiv:2309.07418

    2023 arXiv

  35. [43]

    Gao, Y.; Song, Z.; Yang, X.; Zhang, R.; and Zhou, Y. 2023 b . Fast quantum algorithm for attention computation. arXiv preprint arXiv:2307.08045

    2023

  36. [44]

    Gao, Y.; Song, Z.; Yang, X.; and Zhou, Y. 2023 c . Differentially private attention computation. arXiv preprint arXiv:2305.04701

    2023 arXiv

  37. [45]

    Gao, Y.; Song, Z.; and Yin, J. 2023. An iterative algorithm for rescaled hyperbolic functions regression. arXiv preprint arXiv:2305.00660

    2023 arXiv

  38. [46]

    Gao, Y.; Song, Z.; Zhang, R.; and Zhou, Y. 2024. Quantum Speedup for Spectral Approximation of Kronecker Products. arXiv preprint arXiv:2402.07027

    2024 arXiv

  39. [47]

    Gong, Y.; Zhan, Z.; Jin, Q.; et al. 2024. E ^ 2 GAN: Efficient Training of Efficient GANs for Image-to-Image Translation. In ICML

    2024

  40. [48]

    Gu, J.; Li, C.; Liang, Y.; Shi, Z.; and Song, Z. 2024 a . Exploring the frontiers of softmax: Provable optimization, applications in diffusion model, and beyond. arXiv preprint arXiv:2405.03251

    2024

  41. [49]

    Gu, J.; Li, C.; Liang, Y.; Shi, Z.; Song, Z.; and Zhou, T. 2024 b . Fourier circuits in neural networks: Unlocking the potential of large language models in mathematical reasoning and modular arithmetic. arXiv preprint arXiv:2402.09469

    2024 arXiv

  42. [50]

    Gu, J.; Liang, Y.; Sha, Z.; Shi, Z.; and Song, Z. 2024 c . Differential Privacy Mechanisms in Neural Tangent Kernel Regression. arXiv preprint arXiv:2407.13621

    2024 arXiv

  43. [51]

    Gu, Y.; and Song, Z. 2022. A Faster Small Treewidth SDP Solver. arXiv:2211.06033

    2022 arXiv

  44. [52]

    Gu, Y.; Song, Z.; and Zhang, L. 2023. A nearly-linear time algorithm for structured support vector machines. arXiv preprint arXiv:2307.07735

    2023 arXiv

  45. [53]

    He, Y.; Liu, L.; Liu, J.; Wu, W.; Zhou, H.; and Zhuang, B. 2023. PTQD : Accurate Post-Training Quantization for Diffusion Models. In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  46. [54]

    Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS

    2017

  47. [55]

    Hewitt, J.; and Liang, P. 2019. Designing and Interpreting Probes with Control Tasks. arXiv:1909.03368

    2019 arXiv

  48. [56]

    Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising diffusion probabilistic models. NeurIPS

    2020

  49. [57]

    Ho, J.; and Salimans, T. 2022. Classifier-Free Diffusion Guidance. arXiv:2207.12598

    2022 arXiv

  50. [58]

    Huang, B.; Jiang, S.; Song, Z.; Tao, R.; and Zhang, R. 2021. Solving SDP Faster: A Robust IPM Framework and Efficient Implementation. arXiv:2101.08208

    2021 arXiv

  51. [59]

    Jacq, A.; Ferret, J.; Pietquin, O.; and Geist, M. 2022. Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Act. arXiv:2203.08542

    2022 arXiv

  52. [60]

    Jagielski, M.; Thakkar, O.; Tramèr, F.; Ippolito, D.; Lee, K.; Carlini, N.; Wallace, E.; Song, S.; Thakurta, A.; Papernot, N.; and Zhang, C. 2023. Measuring Forgetting of Memorized Training Examples. arXiv:2207.00099

    2023 arXiv

  53. [61]

    Jambulapati, A.; Li, J.; and Tian, K. 2020. Robust Sub-Gaussian Principal Component Analysis and Width-Independent Schatten Packing. arXiv:2006.06980

    2020 arXiv

  54. [62]

    T.; Padmanabhan, S.; and Song, Z

    Jiang, H.; Kathuria, T.; Lee, Y. T.; Padmanabhan, S.; and Song, Z. 2020 a . A faster interior point method for semidefinite programming. In 2020 IEEE 61st annual symposium on foundations of computer science (FOCS), 910--918. IEEE

    2020

  55. [63]

    T.; Song, Z.; and wai Wong, S

    Jiang, H.; Lee, Y. T.; Song, Z.; and wai Wong, S. C. 2020 b . An Improved Cutting Plane Method for Convex Optimization, Convex-Concave Games and its Applications. arXiv:2004.04250

    2020 arXiv

  56. [64]

    Jiang, S.; Song, Z.; Weinstein, O.; and Zhang, H. 2020 c . Faster dynamic matrix inverse for faster lps. arXiv preprint arXiv:2004.07470

    2020 arXiv

  57. [65]

    T.; Ge, R.; and Jordan, M

    Jin, C.; Liu, L. T.; Ge, R.; and Jordan, M. I. 2018. On the local minima of the empirical risk. NeurIPS

    2018

  58. [66]

    Joachims, T. 2006. Training linear SVMs in linear time. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, 217--226

    2006

  59. [67]

    Johnson, R.; and Zhang, T. 2013. Accelerating stochastic gradient descent using predictive variance reduction. NeurIPS

    2013

  60. [68]

    Kacham, P.; Mirrokni, V.; and Zhong, P. 2023. Polysketchformer: Fast transformers via sketches for polynomial kernels. arXiv preprint arXiv:2310.01655

    2023 arXiv

  61. [69]

    Kim, B.-K.; Song, H.-K.; Castells, T.; and Choi, S. 2023. BK - SDM : Architecturally Compressed Stable Diffusion for Efficient Text-to-Image Generation. In Workshop on Efficient Systems for Foundation Models @ ICML2023

    2023

  62. [70]

    Kong, Z.; Dong, P.; Ma, X.; et al. 2022. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In ECCV

    2022

  63. [71]

    Kong, Z.; Ma, H.; Yuan, G.; et al. 2023. Peeling the onion: Hierarchical reduction of data redundancy for efficient vision transformer training. In AAAI

    2023

  64. [72]

    Kynkäänniemi, T.; Karras, T.; Laine, S.; Lehtinen, J.; and Aila, T. 2019. Improved Precision and Recall Metric for Assessing Generative Models. CoRR, abs/1904.06991

    2019 arXiv

  65. [73]

    Lab, P.-Y.; and etc., T. A. 2024. Open-Sora-Plan

    2024

  66. [74]

    T.; Song, Z.; and Zhang, Q

    Lee, Y. T.; Song, Z.; and Zhang, Q. 2019. Solving empirical risk minimization in the current matrix multiplication time. In Conference on Learning Theory, 2140--2157. PMLR

    2019

  67. [75]

    Li, S.; Song, Z.; Xia, Y.; Yu, T.; and Zhou, T. 2023 a . The closeness of in-context learning and weight shifting for softmax regression. arXiv preprint arXiv:2304.13276

    2023 arXiv

  68. [76]

    Li, T.; Tian, Y.; Li, H.; Deng, M.; and He, K. 2024 a . Autoregressive Image Generation without Vector Quantization. arXiv preprint arXiv:2406.11838

    2024 arXiv

  69. [77]

    Li, X.; Liang, Y.; Shi, Z.; Song, Z.; and Zhou, Y. 2024 b . Fine-grained Attention I/O Complexity: Comprehensive Analysis for Backward Passes. arXiv preprint arXiv:2410.09397

    2024

  70. [78]

    Li, X.; Liu, Y.; Lian, L.; Yang, H.; Dong, Z.; Kang, D.; Zhang, S.; and Keutzer, K. 2023 b . Q-Diffusion: Quantizing Diffusion Models. In ICCV, 17535--17545

    2023

  71. [79]

    Li, X.; Long, J.; Song, Z.; and Zhou, T. 2024 c . Fast Second-order Method for Neural Network under Small Treewidth Setting. In 2024 IEEE International Conference on Big Data (BigData). IEEE

    2024

  72. [80]

    Li, Y.; Wang, H.; Jin, Q.; Hu, J.; Chemerys, P.; Fu, Y.; Wang, Y.; Tulyakov, S.; and Ren, J. 2023 c . SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Seconds. arXiv preprint arXiv:2306.00980

    2023 arXiv

  73. [81]

    Li, Y.; Yang, C.; Zhao, P.; et al. 2023 d . Towards real-time segmentation on the edge. AAAI'23/IAAI'23/EAAI'23. AAAI Press. ISBN 978-1-57735-880-0

    2023

  74. [82]

    Li, Y.; Zhang, Y.; Liu, S.; and Lin, X. 2024 d . Pruning then Reweighting: Towards Data-Efficient Training of Diffusion Models. arXiv preprint arXiv:2409.19128

    2024 arXiv

  75. [83]

    Li, Y.; Zhao, P.; Ding, R.; Zhou, T.; Fei, Y.; Xu, X.; and Lin, X. 2024 e . Neural architecture search for adversarial robustness via learnable pruning. Frontiers in High Performance Computing, 2: 1301384

    2024

  76. [84]

    Li, Y.; Zhao, P.; Yuan, G.; et al. 2022. Pruning-as-Search: Efficient Neural Architecture Search via Channel Pruning and Structural Reparameterization. In IJCAI

    2022

  77. [85]

    Li, Z.; Song, Z.; Wang, Z.; and Yin, J. 2023 e . Local Convergence of Approximate Newton Method for Two Layer Nonlinear Regression. arXiv preprint arXiv:2311.15390

    2023 arXiv

  78. [86]

    Li, Z.; Song, Z.; and Zhou, T. 2023. Solving regularized exp, cosh and sinh regression problems. arXiv preprint arXiv:2303.15725

    2023 arXiv

  79. [87]

    Liang, Y.; Long, J.; Shi, Z.; Song, Z.; and Zhou, Y. 2024 a . Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix. arXiv preprint arXiv:2410.11261

    2024 arXiv

  80. [88]

    Liang, Y.; Sha, Z.; Shi, Z.; Song, Z.; and Zhou, Y. 2024 b . Looped ReLU MLPs May Be All You Need as Practical Programmable Computers. arXiv preprint arXiv:2410.09375

    2024 arXiv

  81. [89]

    Liang, Y.; Sha, Z.; Shi, Z.; Song, Z.; and Zhou, Y. 2024 c . Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time. arXiv preprint arXiv:2408.13233

    2024 arXiv

  82. [90]

    Liang, Y.; Shi, Z.; Song, Z.; and Zhou, Y. 2024 d . Differential Privacy of Cross-Attention with Provable Guarantee. arXiv preprint arXiv:2407.14717

    2024

  83. [91]

    Liang, Y.; Shi, Z.; Song, Z.; and Zhou, Y. 2024 e . Tensor Attention Training: Provably Efficient Learning of Higher-order Transformers. arXiv preprint arXiv:2405.16411

    2024

  84. [92]

    Liang, Y.; Shi, Z.; Song, Z.; and Zhou, Y. 2024 f . Unraveling the Smoothness Properties of Diffusion Models: A Gaussian Mixture Perspective. arXiv preprint arXiv:2405.16418

    2024 arXiv

  85. [93]

    Lianke, Q.; Song, Z.; Zhang, L.; and Zhuo, D. 2023. An online and unified algorithm for projection matrix vector multiplication with application to empirical risk minimization. In International Conference on Artificial Intelligence and Statistics, 101--156. PMLR

    2023

  86. [94]

    Lin, S.; Wang, A.; and Yang, X. 2024. SDXL-Lightning: Progressive Adversarial Diffusion Distillation. arXiv:2402.13929

    2024 arXiv

  87. [95]

    Liu, L.; Ren, Y.; Lin, Z.; and Zhao, Z. 2022. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778

    2022 arXiv

  88. [96]

    Lu, C.; Zhou, Y.; Bao, F.; Chen, J.; Li, C.; and Zhu, J. 2022. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. arXiv:2206.00927

    2022 arXiv

  89. [97]

    Lu, H.; Yang, G.; Fei, N.; Huo, Y.; Lu, Z.; Luo, P.; and Ding, M. 2023. Vdt: General-purpose video diffusion transformers via mask modeling. arXiv preprint arXiv:2305.13311

    2023 arXiv

  90. [98]

    Luo, S.; Tan, Y.; Huang, L.; Li, J.; and Zhao, H. 2024. Latent Consistency Models: Synthesizing High-Resolution Images with Few-step Inference

    2024

  91. [99]

    B.; and Wang, X

    Ma, X.; Fang, G.; Mi, M. B.; and Wang, X. 2024. Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching. arXiv:2406.01733

    2024 arXiv

  92. [100]

    Ma, X.; Fang, G.; and Wang, X. 2024. DeepCache: Accelerating Diffusion Models for Free. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition

    2024

  93. [101]

    Moulines, E.; and Bach, F. 2011. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. NeurIPS

    2011

  94. [102]

    Nash, C.; Menick, J.; Dieleman, S.; and Battaglia, P. W. 2021. Generating images with sparse representations. arXiv preprint arXiv:2103.03841

    2021 arXiv

  95. [103]

    Nemirovski, A.; Juditsky, A.; Lan, G.; and Shapiro, A. 2009. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization, 19(4): 1574--1609

    2009

  96. [104]

    Nesterov, Y. 1983. A method for solving the convex programming problem with convergence rate O (1/k2). In Dokl akad nauk Sssr, volume 269, 543

    1983

  97. [105]

    Nesterov, Y. 2013. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media

    2013

  98. [106]

    OpenAI. 2024. Video generation models as world simulators. https://openai.com/index/video-generation-models-as-world-simulators/

    2024

  99. [107]

    Peebles, W.; and Xie, S. 2023. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4195--4205

    2023

  100. [108]

    T.; and Juditsky, A

    Polyak, B. T.; and Juditsky, A. B. 1992. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4): 838--855

    1992

  101. [109]

    Qin, L.; Song, Z.; and Sun, B. 2023. Is Solving Graph Neural Tangent Kernel Equivalent to Training Graph Neural Network? arXiv preprint arXiv:2309.07452

    2023 arXiv

  102. [110]

    Raposo, D.; Ritter, S.; Richards, B.; et al. 2024. Mixture-of-Depths: Dynamically allocating compute in transformer-based language models. arXiv:2404.02258

    2024 arXiv

  103. [111]

    Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022 a . High-resolution image synthesis with latent diffusion models. In CVPR, 10684--10695

    2022

  104. [112]

    Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022 b . High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2112.10752

    2022 arXiv

  105. [113]

    Ronneberger, O.; Fischer, P.; and Brox, T. 2015. U-net: Convolutional networks for biomedical image segmentation. In MICCAI

    2015

  106. [114]

    Salimans, T.; Goodfellow, I.; Zaremba, W.; Cheung, V.; Radford, A.; and Chen, X. 2016. Improved techniques for training gans. NeurIPS

    2016

  107. [115]

    Salimans, T.; and Ho, J. 2022. Progressive Distillation for Fast Sampling of Diffusion Models. In International Conference on Learning Representations

    2022

  108. [116]

    Shalev-Shwartz, S.; and Zhang, T. 2013. Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research, 14(1)

    2013

  109. [117]

    Shen, X.; Dong, P.; Lu, L.; et al. 2024 a . Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge. In AAAI

    2024

  110. [118]

    Shen, X.; Han, Z.; Lu, L.; et al. 2024 b . HotaQ: Hardware Oriented Token Adaptive Quantization for Large Language Models. TCAD

    2024

  111. [119]

    Shen, X.; Kong, Z.; Qin, M.; et al. 2023 a . Data level lottery ticket hypothesis for vision transformers. IJCAI

    2023

  112. [120]

    Shen, X.; Kong, Z.; Yang, C.; et al. 2024 c . EdgeQAT: Entropy and Distribution Guided Quantization-Aware Training for the Acceleration of Lightweight LLMs on the Edge. arXiv preprint arXiv:2402.10787

    2024 arXiv

  113. [121]

    Shen, X.; Wang, Y.; Lin, M.; et al. 2023 b . DeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Network. In CVPR

    2023

  114. [122]

    Shen, X.; Zhao, P.; Gong, Y.; et al. 2024 d . Search for Efficient Large Language Models. In NeurIPS

    2024

  115. [123]

    Shrivastava, A.; Song, Z.; and Xu, Z. 2023. A Theoretical Analysis Of Nearest Neighbor Search On Approximate Near Neighbor Graph. arXiv preprint arXiv:2303.06210

    2023 arXiv

  116. [124]

    A.; Maheswaranathan, N.; and Ganguli, S

    Sohl-Dickstein, J.; Weiss, E. A.; Maheswaranathan, N.; and Ganguli, S. 2015. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. arXiv:1503.03585

    2015 arXiv

  117. [125]

    Song, J.; Meng, C.; and Ermon, S. 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502

    2020 arXiv

  118. [126]

    Song, Y.; and Ermon, S. 2019. Generative modeling by estimating gradients of the data distribution. NeurIPS

    2019

  119. [127]

    P.; Kumar, A.; Ermon, S.; and Poole, B

    Song, Y.; Sohl-Dickstein, J.; Kingma, D. P.; Kumar, A.; Ermon, S.; and Poole, B. 2020. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456

    2020 arXiv

  120. [128]

    Song, Z.; Wang, W.; and Yin, J. 2023. A unified scheme of resnet and softmax. arXiv preprint arXiv:2309.13482

    2023 arXiv

  121. [129]

    Song, Z.; Xu, G.; and Yin, J. 2023. The Expressibility of Polynomial based Attention Scheme. arXiv preprint arXiv:2310.20051

    2023 arXiv

  122. [130]

    Song, Z.; Ye, M.; and Zhang, L. 2023. Streaming Semidefinite Programs: O( sqrt \ n\ ) Passes, Small Space and Fast Runtime. arXiv preprint arXiv:2309.05135

    2023 arXiv

  123. [131]

    Song, Z.; and Yu, Z. 2021. Oblivious sketching-based central path method for solving linear programming problems

    2021

  124. [132]

    M.; Rae, J

    Sprechmann, P.; Jayakumar, S. M.; Rae, J. W.; Pritzel, A.; Badia, A. P.; Uria, B.; Vinyals, O.; Hassabis, D.; Pascanu, R.; and Blundell, C. 2018. Memory-based Parameter Adaptation. arXiv:1802.10542

    2018 arXiv

  125. [133]

    Sun, P.; Jiang, Y.; Chen, S.; Zhang, S.; Peng, B.; Luo, P.; and Yuan, Z. 2024. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation. arXiv preprint arXiv:2406.06525

    2024 arXiv

  126. [134]

    A.; Li, Y.; Thrampoulidis, C.; and Oymak, S

    Tarzanagh, D. A.; Li, Y.; Thrampoulidis, C.; and Oymak, S. 2023. Transformers as support vector machines. arXiv preprint arXiv:2308.16898

    2023 arXiv

  127. [135]

    Tenney, I.; Das, D.; and Pavlick, E. 2019. BERT Rediscovers the Classical NLP Pipeline. arXiv:1905.05950

    2019 arXiv

  128. [136]

    Trench, W. F. 2013. Introduction to real analysis

    2013

  129. [137]

    van den Brand, J. 2020. Unifying Matrix Data Structures: Simplifying and Speeding up Iterative Algorithms. arXiv:2010.13888

    2020 arXiv

  130. [138]

    Vapnik, V. 1991. Principles of risk minimization for learning theory. NeurIPS

    1991

  131. [139]

    Vapnik, V. 2013. The nature of statistical learning theory. Springer science & business media

    2013

  132. [140]

    N.; Kaiser, .; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. NeurIPS, 30

    2017

  133. [141]

    Vig, J.; and Belinkov, Y. 2019. Analyzing the Structure of Attention in a Transformer Language Model. arXiv:1906.04284

    2019 arXiv

  134. [142]

    Wang, K.; Chen, J.; Li, H.; Mi, Z.; and Zhu, J. 2024 a . SparseDM: Toward Sparse Efficient Diffusion Models. arXiv:2404.10445

    2024 arXiv

  135. [143]

    Wang, Y.; Chen, Z.; Zhong, L.; Ding, Z.; Sha, Z.; and Tu, Z. 2023 a . Dolfin: Diffusion Layout Transformers without Autoencoder. arXiv preprint arXiv:2310.16305

    2023 arXiv

  136. [144]

    Wang, Y.; Xu, H.; Zhang, X.; Chen, Z.; Sha, Z.; Wang, Z.; and Tu, Z. 2024 b . OmniControlNet: Dual-stage Integration for Conditional Image Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7436--7448

    2024

  137. [145]

    Wang, Z.; Sha, Z.; Ding, Z.; Wang, Y.; and Tu, Z. 2023 b . Tokencompose: Grounding diffusion with token-level supervision. arXiv preprint arXiv:2312.03626

    2023 arXiv

  138. [146]

    Wimbauer, F.; Wu, B.; Schoenfeld, E.; et al. 2023. Cache Me if You Can: Accelerating Diffusion Models through Block Caching. arXiv preprint arXiv:2312.03209

    2023 arXiv

  139. [147]

    Wu, Y.; Gong, Y.; Zhao, P.; et al. 2022. Compiler-aware neural architecture search for on-mobile real-time super-resolution. In ECCV, 92--111. Springer

    2022

  140. [148]

    Yang, C.; Zhao, P.; Li, Y.; et al. 2023 a . Pruning parameterization with bi-level optimization for efficient semantic segmentation on the edge. In CVPR, 15402--15412

    2023

  141. [149]

    Yang, X.; Shih, S.-M.; Fu, Y.; Zhao, X.; and Ji, S. 2022. Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Model. arXiv:2208.07791

    2022 arXiv

  142. [150]

    Yang, X.; Zhou, D.; Feng, J.; and Wang, X. 2023 b . Diffusion probabilistic model made slim. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 22552--22562

    2023

  143. [151]

    T.; and Park, T

    Yin, T.; Gharbi, M.; Zhang, R.; Shechtman, E.; Durand, F.; Freeman, W. T.; and Park, T. 2024. One-step Diffusion with Distribution Matching Distillation. In CVPR

    2024

  144. [152]

    Zhan, Z.; Gong, Y.; Zhao, P.; et al. 2021. Achieving on-mobile real-time super-resolution with neural architecture and pruning search. In ICCV, 4821--4831

    2021

  145. [153]

    Zhan, Z.; Kong, Z.; Gong, Y.; et al. 2024 a . Exploring Token Pruning in Vision State Space Models. In NeurIPS

    2024

  146. [154]

    Zhan, Z.; Wu, Y.; Gong, Y.; et al. 2024 b . Fast and Memory-Efficient Video Diffusion Using Streamlined Inference. In NeurIPS

    2024

  147. [155]

    Zhan, Z.; Wu, Y.; Kong, Z.; et al. 2024 c . Rethinking Token Reduction for State Space Models. In EMNLP, 1686--1697. Miami, Florida, USA: ACL

    2024

  148. [156]

    Zhang, D.; Li, S.; Chen, C.; Xie, Q.; and Lu, H. 2024. LAPTOP-Diff: Layer Pruning and Normalized Distillation for Compressing Diffusion Models. arXiv:2404.11098

    2024 arXiv

  149. [157]

    Zhang, R.; et al. 2023. LLaMA-Adapter: Efficient Finetuning of Language Models with Zero-init Attention. arXiv preprint arXiv:2303.16199

    2023 arXiv

  150. [158]

    Zhang, Y.; and Xiao, L. 2017. Stochastic primal-dual coordinate method for regularized empirical risk minimization. Journal of Machine Learning Research, 18(84): 1--42

    2017

  151. [159]

    Zhang, Y.; Yao, Y.; Ram, P.; et al. 2022. Advancing model pruning via bi-level optimization. NeurIPS

    2022

  152. [160]

    Zhao, P.; Sun, F.; Shen, X.; Yu, P.; Kong, Z.; Wang, Y.; and Lin, X. 2024. Pruning Foundation Models for High Accuracy without Retraining. In Findings of EMNLP 2024, 9681--9694. Miami, Florida, USA: ACL

    2024

  153. [161]

    Zheng, Z.; Peng, X.; Yang, T.; Shen, C.; Li, S.; Liu, H.; Zhou, Y.; Li, T.; and You, Y. 2024. Open-Sora: Democratizing Efficient Video Production for All

    2024

  154. [162]

    Zhu, L. 2022. pytorch-OpCounter. https://github.com/Lyken17/pytorch-OpCounter

    2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.