Pith. sign in

REVIEW 4 major objections 5 minor 55 references

Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Acceleration

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Diffusion transformers can be accelerated 1.55x at no quality cost by merging exactly the tokens their own attention maps ignore, with no fine-tuning.

desk verdict Solid empirical token-merging paper for DiT acceleration; the mechanism story is heuristic but the experiments are worth refereeing. read the letter →

arxiv 2505.11707 v1 pith:MMG2K2XE submitted 2025-05-16 cs.CV

Haipeng Fang , Sheng Tang , Juan Cao , Enshuo Zhang , Fan Tang , Tong-Yee Lee This is my paper · ORCID
classification cs.CV
keywords diffusiontransformerstokenmerginginferenceaccelerationpost-trainingstructure-then-detailpriorsattentionmapstext-to-imagegenerationrectifiedflow
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that diffusion transformers carry hidden feature redundancy in exactly the places the denoising process is not looking, and that pruning those spots yields large speedups with no measurable quality loss. The authors first show, through a wavelet analysis of the estimated noise across denoising steps, that DiTs still follow a structure-then-detail prior: early steps build low-frequency layout, later steps refine high-frequency detail. They then introduce SDTM, a post-training token merging scheme that compresses locally similar, weakly structured tokens early on and globally inattentive, weakly detailed tokens later, with a time-varying compression ratio and prompt-token reweighting to keep the output aligned with the text. On Stable Diffusion 3 Medium with 50-step rectified flow, SDTM reaches 1.55x acceleration while FID improves slightly, from 29.02 to 28.97. A sympathetic reader would care because the method needs no fine-tuning and drops into existing DiT backbones, schedulers, and step counts, making it a near-free inference speedup for text-to-image models.

What carries the argument

The load-bearing mechanism is the two-stage token merging policy built on the structure-then-detail prior. In the structure stage (first 40% of steps) SDTM applies similarity-prioritized structure merging (SSM), which partitions the feature grid into m×m windows, scores each window by average pairwise cosine similarity (Eq. 1) plus a recency term penalizing tokens that were just merged (Eq. 2), and averages the tokens in the top-scoring windows before the attention and MLP blocks. In the detail stage it switches to inattentive-prioritized detail merging (IDM), which scores each token by how little influence it exerts on others via the attention map (Eq. 3, 1 minus the mean attention column), then merges the weakest such tokens into their most similar attentive partner by a softmax-weighted convex combination (Eq. 4), acting only before MLP blocks. Compression ratio adjusting (CRA) modulates the merge ratio over timesteps with a cosine decay, with an adaptive-threshold variant that builds a ratio-to-threshold table from a small calibration batch, and prompt token reweighting (PTR) rescales attention of prompt words classified, via a large language model, as structure- versus detail-bearing. Together these pieces translate the measured redundancy curves directly into compute savings: they decide which tokens to drop, when to drop them, and how strongly the prompt should steer the compressed model.

What would settle it

Apply SDTM to a diffusion transformer pretrained with a noise schedule or distillation that visibly reshapes the wavelet decomposition of the estimated noise; if the fixed 0.6T stage boundary still gives lossless 1.55x acceleration, the structure-then-detail prior is not what carries the gain, whereas a failure would confirm the prior is load-bearing.

Watch

Extended reading notes

Core claim

The paper's central claim is that the tokens a diffusion transformer effectively ignores—those it attends to least—are precisely the redundant ones, so pruning them is nearly free. It establishes this by decomposing the denoising process in the wavelet domain and showing that DiTs preserve the structure-then-detail prior: the first 40% of steps concentrate on low-frequency layout, the remaining 60% on high-frequency detail, and token redundancy is high early and low late, clustered locally early and spread globally later. On that basis SDTM merges tokens in two regimes: similarity-prioritized structure merging (SSM), which averages tokens inside local windows with high cosine similarity and low recent merge frequency, and inattentive-prioritized detail merging (IDM), which merges the tokens with the smallest column-wise influence in the attention map into their most similar attentive counterparts. A compression-ratio adjusting schedule (cosine decay from ρ+d to ρ−d, with an adaptive-threshold variant) matches the measured redundancy curve, and a prompt-token reweighting step rescales attention to structure- versus detail-relevant prompt words so the compressed model stays aligned with the text. The authors report 1.55x acceleration on SD3 Medium at 50-step rectified flow with FID 28.97 versus 29.02 baseline, and similar preservation across SD3.5 Large, SD3.5 Large Turbo, and FLUX.1-dev, across schedulers and step counts, all without fine-tuning.

Load-bearing premise

The claim collapses if tokens that the denoising process attends to least are not actually the redundant ones; the paper infers this equivalence from a wavelet analysis on one backbone (SD3 Medium) and assumes it transfers to every architecture, scheduler, and step count tested.

Editorial extensions

If this is right

  • A post-training, fine-tuning-free 1.55x speedup transfers across backbones: SD3 Medium, SD3.5 Large, SD3.5 Large Turbo, and FLUX.1-dev, under both rectified-flow and DPM-Solver++ schedulers and step counts from 50 down to 4.
  • Token compression no longer needs to trade away text alignment: the prompt reweighting component keeps CLIP scores at or above baseline levels even when visual tokens are heavily pruned.
  • The structure-then-detail split gives a principled way to allocate compute over time: early denoising steps can tolerate aggressive merging, late steps require lighter merging, and the ratio schedule can be set once and reused.
  • Because the priority scores are computed from the model's own activations and attention maps, the method adapts to image complexity: the adaptive-threshold variant prunes more on simple images and less on complex ones.
  • The method is orthogonal to distillation and caching, so it can be stacked with those techniques for further acceleration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 'unattended tokens are redundant' equivalence holds generally, the same two-score recipe (local cosine similarity early, attention-column influence late) could be ported to video diffusion transformers, where the token budget is larger and the attention maps carry temporal structure.
  • The paper's frequency analysis suggests a testable prediction: models trained with different noise schedules or distilled few-step samplers, where the structure-then-detail split is compressed or reordered, should show a different optimal stage boundary than the fixed 0.6T threshold used here.
  • The attention-column influence score (Eq. 3) treats all query rows equally; weighting rows by their own importance (e.g., by saliency or by the magnitude of the output projection) might identify redundancies that a uniform column average misses.
  • A stronger falsifier-level check would compare SDTM's chosen tokens against an oracle that removes each token and measures the actual change in the denoised output; if the correlation is weak, the merging priorities are picking the wrong tokens even though end-to-end metrics look fine.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SDTM, a post-training token merging method for accelerating diffusion transformers. The method splits the denoising process into a structure stage (early steps) and a detail stage (later steps), merging locally similar tokens in the structure stage (Eq. 1) and globally inattentive tokens in the detail stage (Eq. 3), with a dynamic compression ratio schedule (Sec. 4.1.3) and a prompt-token reweighting mechanism (Sec. 4.2). The approach requires no fine-tuning. Experiments are reported on SD3 Medium, SD3.5 Large, SD3.5 Large Turbo, and FLUX.1-dev across RF and DPM-Solver++ schedulers, with FID and CLIP on COCO2017 and PartiPrompts. The headline result is a 1.55x latency reduction on SD3 Medium at 50 steps with FID 28.97 vs. 29.02 baseline.

Significance. The paper addresses a practically important problem: reducing the inference cost of DiT-based text-to-image models without retraining. If the results are reproducible, the proposed method could be a useful plug-in acceleration module. The empirical study is broad, covering several backbones, schedulers, step counts, and image resolutions, and includes an analysis of the overhead of the redundancy identification (Table 4). However, the conceptual contribution is heuristic rather than formal: the core premise that 'unattended' tokens are redundant is assumed rather than directly tested, and the evaluation methodology has issues (selection of hyperparameters on the evaluation set, missing error bars, inconsistency in GPU count, and an exception for FLUX). These issues currently limit confidence in the headline claims.

major comments (4)
  1. [Sec. 3, Eqs. (1) and (3)] The core premise that 'unattended' tokens are redundant is not directly validated. The preliminary analysis in Sec. 3 shows that the L2 norms of DWT subbands evolve differently at early vs. late denoising steps and that nearest-neighbor token cosine similarity varies over time, but it does not establish that merging tokens with high P_sim (Eq. 1) or low P_ina (Eq. 3) preserves the denoising trajectory. The statement in Sec. 3 that the hypothesis is validated by tracking the location of feature redundancies is circular, because the same cosine similarity is used to define redundancy and to select tokens for merging. A direct test would be to compare the proposed scores against random token selection at matched compression ratios, or to measure the per-token reconstruction error in the estimated noise when a token is removed. Without such evidence, the reported FID parity may be due to the carefully tuned ratio schedule (e.g., d=0.2 in Table 3) rather than to the 'attend to not attended' mechanism.
  2. [Sec. 5.1 and Sec. 5.4, Table 1 vs. Table 3] The hyperparameters basic ratio ρ, maximum deviation d, and the stage split (T-0.6T) are selected on the COCO2017 validation set, which is also the set used for the final FID numbers in Table 1. This amounts to model selection on the test set. Moreover, no error bars or multiple seeds are reported; the headline difference (FID 28.97 vs. 29.02) is 0.05, which is likely within FID estimation noise. The paper should either hold out a validation subset for ablations and report final numbers on a separate test set, or provide multiple-run statistics and confidence intervals.
  3. [Sec. 5.1 vs. Suppl. Sec. 7.3] The main text states that all experiments were performed using 4 NVIDIA A100 40G GPUs, while the supplementary material states that all experiments were conducted using two NVIDIA A100 GPUs. Since latency is a central metric, this inconsistency must be resolved for reproducibility.
  4. [Suppl. Table 5 (FLUX.1-dev)] For FLUX.1-dev, the authors left all MHSA blocks unpruned because RoPE is 'extremely sensitive to token reduction.' This contradicts the abstract's claim that the method can be integrated seamlessly into any DiT architecture, and it means that inattentive-prioritized detail merging (Eq. 3) is not applied to attention modules in that model. The paper should qualify the universality claim and specify the conditions under which the full method is applicable.
minor comments (5)
  1. [Fig. 1 caption] The caption contains a double negative: 'areas not unattended' should read 'areas not attended'.
  2. [Tables 1 and 2] The unit 'MACs(T)' is ambiguous: Table 2 uses 'W-MACs' for total computation across all steps, while Table 1 lists values that appear to be per-step MACs. Please define the units consistently.
  3. [Sec. 4.1.2, Eq. (4)] The notation for x_i and x_a is not fully defined in the text; please clarify which token is the inattentive one being merged into the attentive one.
  4. [Sec. 4] The hyperparameters α_s, α_d, α_p, and the window size m are not ablated; a sensitivity analysis would help assess robustness.
  5. [References] Reference [14] is missing the publication venue, and several other references are incompletely formatted.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SDTM's design is motivated heuristically and validated on external FID/CLIP benchmarks; no prediction reduces by construction to its inputs.

full rationale

No circular derivation chain is present. The paper's actual chain is: (1) observe structure-then-detail frequency evolution and token similarity/distance statistics in Sec. 3; (2) hypothesize that locally similar early tokens and globally low-attention later tokens are redundant; (3) instantiate this hypothesis as SSM and IDM scores in Eqs. 1 and 3; (4) evaluate FID/CLIP on COCO2017, PartiPrompts, and other external benchmarks. Step (2) is an empirical hypothesis, not a definitional equivalence: the scores are not defined in terms of the final FID, and the evaluation could have shown quality loss. The adaptive threshold variant in Algorithm 1 calibrates a similarity-to-ratio mapping on a sampled batch, which is a tuning step rather than a case where the reported prediction reduces to the fitted input by construction; the headline 28.97 vs 29.02 FID is an external benchmark result that can be independently reproduced. Citations to ToMe, ALGM, and AT-EDM supply components but are not load-bearing self-citations, and the structure-then-detail schedule is tested through ablations. The FLUX caveat about RoPE sensitivity to token reduction is an acknowledged limitation, not evidence of circularity.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The method rests on several empirical assumptions rather than proven theory. The main free parameters are the compression ratio, deviation, stage split, and unspecified weighting factors, plus the LLM prompt classification. No new entities are introduced.

free parameters (6)
  • basic ratio ρ = 0.5 (default, varied for trade-off curves)
    Controls the fraction of windows or tokens merged; set as a user demand in Sec 4.1.3 and tuned for the headline config in Sec 5.1.
  • max deviation d = 0.2
    Ablated over {0.1, 0.2, 0.3, 0.4} in Table 3; 0.2 with cosine decay is selected, which is the configuration used in the reported results.
  • stage split = 0.6T structure / 0.4T detail
    Fixed in Sec 5.1 as T to 0.6T for structure and 0.6T to 0 for detail; not ablated.
  • scaling factors α_s and α_d = unspecified
    Appear in priority score formulas (Sec 4.1.1 and 4.1.2) but their values are not reported.
  • prompt reweighting exponent α_p = unspecified
    Defines the attention reweighting range [α_p, 1/α_p] in Sec 4.2; value not stated.
  • SSM window size m = unspecified
    Window size for local similarity grouping in Eq. 1 is not given in the paper.
assumptions (6)
  • domain assumption DiT denoising follows a structure-then-detail prior.
    Sec 3 infers this from DWT of estimated noise on SD3 Medium with 50-step rectified flow, and assumes it holds across all tested models, schedulers, and step counts.
  • domain assumption Tokens unattended by the denoising process are redundant and safe to merge.
    Stated in Sec 3 as a hypothesis and validated indirectly by showing high similarity among some tokens; no causal proof that merging them preserves quality.
  • domain assumption Local cosine similarity identifies weak-structure tokens.
    SSM uses average cosine similarity within windows (Eq. 1) as the structure-stage priority, assuming similar tokens carry redundant structure.
  • domain assumption Low attention-column influence identifies weak-detail tokens.
    IDM uses P_ina = 1 - mean(A(:, x)) in Eq. 3, assuming tokens that others attend to little are information-sparse.
  • ad hoc to paper The adaptive ratio-threshold map built from a small sampled batch is representative.
    Algorithm 1 constructs M from a batch; the paper does not specify the batch source or size, and if it is from the evaluation set, it leaks test information.
  • ad hoc to paper ChatGPT's prompt-token categorization reflects structure and detail roles.
    Sec 4.2 relies on an LLM with a hand-written instruction to classify tokens; the classification is not validated and is non-deterministic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Acceleration." pith.science (2026). https://pith.science/paper/MMG2K2XE

@misc{pith2026250511707,
  author       = {Pith},
  title        = {Pith review of: Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Acceleration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MMG2K2XE}},
  note         = {Machine review of arXiv:2505.11707}
}
read the original abstract

Diffusion transformers have shown exceptional performance in visual generation but incur high computational costs. Token reduction techniques that compress models by sharing the denoising process among similar tokens have been introduced. However, existing approaches neglect the denoising priors of the diffusion models, leading to suboptimal acceleration and diminished image quality. This study proposes a novel concept: attend to prune feature redundancies in areas not attended by the diffusion process. We analyze the location and degree of feature redundancies based on the structure-then-detail denoising priors. Subsequently, we introduce SDTM, a structure-then-detail token merging approach that dynamically compresses feature redundancies. Specifically, we design dynamic visual token merging, compression ratio adjusting, and prompt reweighting for different stages. Served in a post-training way, the proposed method can be integrated seamlessly into any DiT architecture. Extensive experiments across various backbones, schedulers, and datasets showcase the superiority of our method, for example, it achieves 1.55 times acceleration with negligible impact on image quality. Project page: https://github.com/ICTMCG/SDTM.

Figures

Figures reproduced from arXiv: 2505.11707 by the authors.

Figure 1
Figure 1. Illustration. Upper: Our SDTM represents a dynamic multi-resolution generation process by reducing feature redundan￾cies in areas not unattended by the denoising process. Lower: Compared to the baseline method, our approach achieves 1.55× acceleration with negligible impact on generation quality. tural redundancy. Due to the inflexibility of these methods for diverse data and evolving requirements, various studies i… view at source ↗
Figure 2
Figure 2. Preliminaries. Left: Evolution of Denoising Process: The L2 norm of (a) LL, (b) LH, HL, and HH subbands of estimated noise during the DiT denoising process. Right: Evolution of Feature Redundancies: (c) Location and (d) Degree evolution of token redundancies across DiT different steps and layers. Experiments were conducted on MMDiT [9] with 50-step RF schedule [25] based on 10k samples. How to find these unattended … view at source ↗
Figure 3
Figure 3. Overview. Grey: Our SDTM compresses weak-structure redundancies in the early stage and weak-detail redundancies in the later stage. Blue: Compression ratio adjusting (CRA) dynamically adjusts the ratio or threshold to control the pruning degree. Yellow: Prompt token reweighting (PTR) categorizes each prompt token into structure or detail groups, optimizing the denoising direction by reweighting attention map. Here, … view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Visual Token Merging. By measuring structure similar [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison on COCO2017 and PartiPrompts under varying data complexities. For ToMeSD and AT-EDM, we use versions with approximately 1.3× acceleration, while others use approximately 1.5× versions. Best viewed when zoomed in. 5. Experiment 5.1. Experiments Se…
Figure 6
Figure 6. Figure 6: The trade-off of Latency vs. FID and CLIP. We obtain [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 9
Figure 9. Figure 9: Visualization of merged tokens selected. White masks [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Ablation of prompt token reweighting (PTR). From left [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: MACs of main components in different image sizes. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Reduced MACs of main components along timesteps. [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Qualitative comparison on SD3.5 Large under varying data complexities. For ToMeSD and AT-EDM, we use versions with [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Uncurated images generated using SD3 Medium and SD3.5 Large configurations under the [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 26 canonical work pages

  1. [1]

    Token merging for fast sta- ble diffusion

    Daniel Bolya and Judy Hoffman. Token merging for fast sta- ble diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4599– 4603, 2023. 1, 2, 3, 5

    work page 2023

  2. [2]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In ICLR, 2023. 1

    work page 2023

  3. [3]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  4. [4]

    Ld-pruner: Efficient pruning of latent diffu- sion models using task-agnostic insights

    Thibault Castells, Hyoung-Kyu Song, Bo-Kyeong Kim, and Shinkook Choi. Ld-pruner: Efficient pruning of latent diffu- sion models using task-agnostic insights. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 821–830, 2024. 1, 2

    work page 2024

  5. [5]

    Videocrafter1: Open diffu- sion models for high-quality video generation.arXiv preprint arXiv:2310.19512, 2023

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffu- sion models for high-quality video generation.arXiv preprint arXiv:2310.19512, 2023. 2

    arXiv

  6. [6]

    Pixart-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 1, 2

    arXiv

  7. [7]

    arXiv preprint arXiv:2406.01125, 2024

    Pengtao Chen, Mingzhu Shen, Peng Ye, Jianjian Cao, Chongjun Tu, Christos-Savvas Bouganis, Yiren Zhao, and Tao Chen.∆-dit: A training-free acceleration method tailored for diffusion transformers. arXiv preprint arXiv:2406.01125, 2024. 2

    arXiv 2024

  8. [8]

    Diffusion mod- els beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion mod- els beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021. 2

    2021

Show all 55 references
  1. [9]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

  2. [10]

    Struc- tural pruning for diffusion models

    Gongfan Fang, Xinyin Ma, and Xinchao Wang. Struc- tural pruning for diffusion models. Advances in neural information processing systems, 36, 2024. 1, 2

    2024

  3. [11]

    Flux.1.https://github.com/ black-forest-labs/flux, 2024

    Black forest labs. Flux.1.https://github.com/ black-forest-labs/flux, 2024. 1, 2

    2024

  4. [12]

    Ptqd: Accurate post-training quantiza- tion for diffusion models

    Yefei He, Luping Liu, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Ptqd: Accurate post-training quantiza- tion for diffusion models. Advances in Neural Information Processing Systems, 36, 2024. 1, 2

    2024

  5. [13]

    Prompt-to-prompt im- age editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 5

    2022 arXiv

  6. [14]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. 2017. 6

    2017

  7. [15]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. pages 6840–6851, 2020. 2

    2020

  8. [16]

    Bk-sdm: A lightweight, fast, and cheap ver- sion of stable diffusion

    Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi. Bk-sdm: A lightweight, fast, and cheap ver- sion of stable diffusion. arXiv preprint arXiv:2305.15798,

    arXiv

  9. [17]

    Distilling ode solvers of diffusion models into smaller steps

    Sanghwan Kim, Hao Tang, and Fisher Yu. Distilling ode solvers of diffusion models into smaller steps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9410–9419, 2024. 2

    2024

  10. [18]

    Open-sora-plan

    PKU-Yuan Lab and Tuzhan AI etc. Open-sora-plan. https://doi.org/10.5281/zenodo.10948109,

    doi

  11. [19]

    Faster diffusion: Rethinking the role of unet encoder in diffusion models

    Senmao Li, Taihang Hu, Fahad Shahbaz Khan, Linxuan Li, Shiqi Yang, Yaxing Wang, Ming-Ming Cheng, and Jian Yang. Faster diffusion: Rethinking the role of unet encoder in diffusion models. arXiv e-prints, pages arXiv–2312, 2023. 2

    2023

  12. [20]

    Q-diffusion: Quantizing diffusion models

    Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17535–17545, 2023. 1, 2

    2023

  13. [21]

    Snapfusion: Text-to-image diffusion model on mobile de- vices within two seconds

    Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile de- vices within two seconds. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  14. [22]

    Not all patches are what you need: Expediting vision transformers via token reorganizations

    Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. In ICLR, 2022. 1

    2022

  15. [23]

    Evit: Expediting vision transform- ers via token reorganizations

    Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Evit: Expediting vision transform- ers via token reorganizations. In ICLR, 2022. 4

    2022

  16. [24]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision, pages 740–755. Springer, 2014. 6

    2014

  17. [25]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022. 2, 3

    2022 arXiv

  18. [26]

    Token caching for diffusion transformer accel- eration

    Jinming Lou, Wenyang Luo, Yufan Liu, Bing Li, Xinmiao Ding, Weiming Hu, Jiajiong Cao, Yuming Li, and Chen- guang Ma. Token caching for diffusion transformer accel- eration. arXiv preprint arXiv:2409.18523, 2024. 1, 2

    2024

  19. [27]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775–5787,

  20. [28]

    Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongx- uan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022. 6

    2022 arXiv

  21. [29]

    Deepcache: Accelerating diffusion models for free

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15762–15772, 2024. 1, 2

    2024

  22. [30]

    Latte: Latent diffusion transformer for video generation

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Zi- wei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024. 1, 2

    2024 arXiv

  23. [31]

    On distillation of guided diffusion models

    Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297–14306, 2023. 1, 2

    2023

  24. [32]

    Algm: Adaptive local-then-global token merg- ing for efficient semantic segmentation with plain vision transformers

    Narges Norouzi, Svetlana Orlova, Daan de Geus, and Gijs Dubbelman. Algm: Adaptive local-then-global token merg- ing for efficient semantic segmentation with plain vision transformers. In CVPR, pages 15773–15782, 2024. 1, 3

    2024

  25. [33]

    Scalable diffusion mod- els with transformers

    William Peebles and Saining Xie. Scalable diffusion mod- els with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195– 4205, 2023. 1, 2

    2023

  26. [34]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 2

    2023 arXiv

  27. [35]

    Boosting diffusion models with moving average sampling in frequency domain

    Yurui Qian, Qi Cai, Yingwei Pan, Yehao Li, Ting Yao, Qibin Sun, and Tao Mei. Boosting diffusion models with moving average sampling in frequency domain. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8911–8920, 2024. 2

    2024

  28. [36]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  29. [37]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In NeurIPS,

  30. [38]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2

    2022

  31. [39]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022. 1, 2

    2022 arXiv

  32. [40]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792,

    arXiv

  33. [41]

    Temporal dynamic quantization for dif- fusion models

    Junhyuk So, Jungwon Lee, Daehyun Ahn, Hyungjun Kim, and Eunhyeok Park. Temporal dynamic quantization for dif- fusion models. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  34. [42]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In International Conference on Learning Representations, 2020. 1, 2

    2020

  35. [43]

    Score- based generative modeling through stochastic differen- tial equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score- based generative modeling through stochastic differen- tial equations. In International Conference on Learning Representations, 2020. 2

    2020

  36. [44]

    Attention-driven training-free ef- ficiency enhancement of diffusion models

    Hongjie Wang, Difan Liu, Yan Kang, Yijun Li, Zhe Lin, Ni- raj K Jha, and Yuchen Liu. Attention-driven training-free ef- ficiency enhancement of diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16080–16089, 2024. 1, 2, 5, 6

    2024

  37. [45]

    Cache me if you can: Accelerating diffusion models through block caching

    Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, et al. Cache me if you can: Accelerating diffusion models through block caching. In Proceedings of the IEEE/CVF Conference on Computer Visio...

    2024

  38. [46]

    Diffusion probabilistic model made slim

    Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusion probabilistic model made slim. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 22552–22562, 2023. 2

    2023

  39. [47]

    Scaling autoregres- sive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun- jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin- fei Yang, Burcu Karagol Ayan, et al. Scaling autoregres- sive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3):5, 2022. 6

    2022 arXiv

  40. [48]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2

    2023

  41. [49]

    Vidit-q: Efficient and accurate quantization of diffusion transformers for im- age and video generation

    Tianchen Zhao, Tongcheng Fang, Enshu Liu, Wan Rui, Widyadewi Soedarmadji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, Huazhong Yang, et al. Vidit-q: Efficient and accurate quantization of diffusion transformers for im- age and video generation. arXiv preprint arXiv:2406.02540,

    arXiv

  42. [50]

    Dynamic diffusion transformer

    Wangbo Zhao, Yizeng Han, Jiasheng Tang, Kai Wang, Yib- ing Song, Gao Huang, Fan Wang, and Yang You. Dynamic diffusion transformer. arXiv preprint arXiv:2410.03456,

    arXiv

  43. [51]

    Open-sora: Democratizing efficient video production for all.https://github.com/hpcaitech/Open- Sora, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all.https://github.com/hpcaitech/Open- Sora, 2024. 1, 2

    2024

  44. [52]

    Magicvideo: Efficient video generation with latent diffusion models

    Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022. 2 Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Accelera...

    2022 arXiv

  45. [53]

    Details of Preliminaries In Sec

    Implementation Details 7.1. Details of Preliminaries In Sec. 3, we examine the evolution of the LL, HL, LH, and HH subbands in the denoising process, along with the loca- tion and degree evolution of feature redundancies. Below, we detail the preliminary experiments. Feature C...

    2014

  46. [54]

    Additional Analyses 8.1. Reduced Computation via Timesteps In our methodology, we introduce similarity-prioritized structure merging to enhance the efficiency of MHSA and MLP blocks during the structure stage, and inattentive- prioritized detail merging to speed up MLP blocks ...

  47. [55]

    Two ceramic cups - one with a bird and the other with a fox

    Additional Results 9.1. More Comparisons with Baselines More Comparisons with Baselines.We expanded the integration ofSDTMandSDTM* into additional baselines to assess their adaptability. These included the SD3.5 Large Turbo, a distilled version of SD3.5 Large designed to enhan...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.