Pith. sign in

REVIEW 3 major objections 6 minor 60 references

Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DiffCR learns where to skip compute in diffusion transformers.

desk verdict DiffCR gives a real quality-efficiency win with a new differentiable per-layer/timestep compression idea, but the unvalidated interpolation surrogate is the main soft spot. read the letter →

arxiv 2412.16822 v2 pith:3LWRQL5D submitted 2024-12-22 cs.CV cs.AIcs.LG

Haoran You , Connelly Barnes , Yuqian Zhou , Yan Kang , Zhenbang Du , Wei Zhou , Lingzhi Zhang , Yotam Nitzan
show 5 more authors
Xiaoyang Liu Zhe Lin Eli Shechtman Sohrab Amirghodsi Yingyan Celine Lin
This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords diffusiontransformersdynamicinferencetokenroutingdifferentiablecompressionratiosmixture-of-depthstext-to-imagegenerationimageinpaintingmodelefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DiffCR is a way to make diffusion transformers spend computation only where it matters during image generation. Instead of applying one global token-pruning ratio to every layer and every denoising step, it lets each layer and each timestep region learn its own ratio, with a lightweight router per layer deciding which tokens skip that layer's attention and MLP entirely. The ratios become trainable through a differentiable surrogate: during training the output at a learned continuous ratio is a weighted blend of the outputs at the two nearest discrete compression bins, and at inference the ratio snaps to the nearest bin. On text-to-image and inpainting benchmarks the paper reports lower FID (a standard image-quality score) at comparable or lower latency than uniform token routing, token merging, and attention-based pruning baselines. If the claim holds, it gives a practical way to deploy expensive diffusion transformers with less computation and without hand-tuning per-layer or per-timestep compression schedules.

What carries the argument

The load-bearing mechanism is the differentiable compression-ratio surrogate built on top of mixture-of-depths (MoD) token routing. MoD is a routing scheme in which each layer's router scores tokens and only the top-k tokens are processed by the layer. DiffCR gives every layer a single learnable scalar compression ratio and, during training, queries the two nearest 10% ratio bins, runs the MoD layer at both ratios, and linearly combines the two outputs with weights proportional to the scalar's distance to each bin; a mean-squared-error loss drives the average ratio to a target. At inference the scalar snaps to the nearest bin, so no extra branches run. This mechanism is what makes the compression ratio differentiable, allowing routers, ratios, and model weights to be fine-tuned together.

What would settle it

Run the same fine-tuning recipe with the interpolation replaced by a straight-through estimator that snaps each learned ratio to the nearest discrete bin in the forward pass while keeping gradients flowing through the scalar, then compare FID and latency on the same 10K-image evaluation set; if the straight-through model matches or beats DiffCR, the interpolation proxy is not what delivers the gains, and if it is much worse, the proxy is carrying the method.

Watch

Extended reading notes

Core claim

The central claim is that the right amount of token compression in a diffusion transformer is not a global hyperparameter but something the model can learn jointly with its weights. DiffCR adds to each DiT layer a router that scores every token; only the top-k tokens run through the layer, and the rest bypass it. A continuous scalar per layer (and per timestep region) controls k through the differentiable surrogate described above, and a mean-squared-error loss pulls the batch-averaged ratios toward a target. On PixArt-Sigma for text-to-image, DiffCR-LT achieves FID 10.68 at 179.71 seconds latency versus FID 22.78 at 178.89 seconds for a uniform-ratio MoD baseline; on Lazy Diffusion for inpainting it achieves FID 13.42 at 13.89 seconds versus FID 18.34 at 13.02 seconds for the same baseline. The learned pattern concentrates compression in redundant middle layers and noisier early timesteps, while later layers and clearer timesteps stay mostly uncompressed.

Load-bearing premise

The paper assumes that, during training, the weighted average of the two nearest discrete compression-ratio outputs faithfully predicts what the model would produce at the actual learned ratio, so that optimizing this blend carries over to the hard nearest-bin choice used at inference.

Editorial extensions

If this is right

  • At a fixed latency budget, DiffCR should improve FID over uniform token compression because it moves pruning away from critical layers and clear-image timesteps.
  • DiffCR is compatible with orthogonal accelerations such as layer caching and few-step distillation, so its savings should stack with those methods.
  • One fine-tuning run produces intermediate checkpoints at a range of average compression ratios, yielding a family of deployable models along a quality-latency frontier.
  • Because the learned pattern assigns higher compression to noisier timesteps, the efficiency gain should grow as the number of sampling steps increases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same differentiable-ratio mechanism should transfer to other hard discrete choices in dynamic networks, such as selecting attention heads or channels, whenever a selection can be relaxed into a weighted blend of two neighboring options.
  • The learned noise-level pattern suggests that a simple hand-designed schedule that reduces compression as denoising proceeds could capture much of the gain; DiffCR's distinctive contribution would then be discovering that schedule automatically, layer by layer, rather than specifying it.
  • The routers' importance maps resemble attention maps, which suggests the approach could extend to other content-adaptive generation tasks such as video or super-resolution, where token redundancy varies spatially and temporally.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces DiffCR, a method to make mixture-of-depth (MoD) token routing in diffusion transformers differentiable with respect to compression ratios. Alongside a token-level router, DiffCR learns per-layer and per-timestep compression ratios by interpolating between two discrete MoD branches during training and snapping to the nearest 10% bin at inference. The method is evaluated on text-to-image with PixArt-Σ and on inpainting with Lazy Diffusion, reporting lower FID than ToMe, AT-EDM, and uniform MoD at roughly comparable latency and memory. The authors also analyze learned ratio patterns, report human-preference scores, and include supplementary comparisons with caching baselines.

Significance. If the results hold, DiffCR would be a practically relevant extension of MoD to vision, and the idea of learning compression ratios via a differentiable surrogate is attractive because it avoids per-layer manual tuning. The paper is also commendable for the breadth of its evaluation: two tasks, comparisons against training-free and learned baselines, router visualizations, ratio-trajectory analysis, human-preference scores, and supplementary caching-baseline comparisons. However, the central mechanism is currently under-validated: the paper does not show that optimizing the blended training surrogate transfers to the hard nearest-bin selection used at deployment, and the reported efficiency gains over uniform MoD are small and lack error bars. These gaps are fixable and do not invalidate the potential of the approach, but they need to be addressed before the trade-off claim can be accepted.

major comments (3)
  1. [Sec. 3.3 and Fig. 1(b)] The load-bearing step is the claim that the differentiable-ratio surrogate is a faithful proxy for the hard MoD selection. During training, a layer with learned ratio r computes a convex combination of the outputs at the lower and upper 10% bins; at inference the ratio snaps to the nearest bin. The paper provides no experiment showing that the blended objective's optimum matches the FID-optimal discrete bin assignment, nor that the loss landscape is convex or smooth enough for the transfer to hold. The trajectory plots (Figs. 4 and 8) only show convergence of the average ratio to the target, not that per-layer or per-timestep ratios are well-calibrated on the deployed path. I request a validation experiment, e.g., after training, evaluate FID for every discrete bin assignment (or a substantial random subset), and compare the surrogate's predicted quality with actual FID at the snapped ratios; this would test whether the learned ratios are optimal rather than merely MSE-converged.
  2. [Tables 1 and 2] The efficiency advantage of DiffCR-LT over uniform MoD is not actually demonstrated. For T2I, DiffCR-LT has 179.71 s latency and 1.664 GB memory versus 178.89 s and 1.659 GB for MoD; for inpainting at BS=128, it has 13.89 s and 34.88 GB versus 13.02 s and 32.58 GB for MoD. The entire case for DiffCR-LT therefore rests on the FID differences (10.68 vs 22.78 and 13.42 vs 18.34), but no error bars, number of seeds, or statistical significance tests are reported. Since fine-tuning alone changes the T2I FID from 151.0 to 11.93 (Table 1), the reader cannot exclude that the FID gap is partly due to training noise or to a different effective fine-tuning budget. Please report mean and std over at least three seeds for the key comparisons, and state the fine-tuning budget (steps/epochs and FLOPs) for each method.
  3. [Sec. 4.3 and Tables 1-2] The ablation DiffCR-L versus MoD changes two things at once: the per-layer/timestep ratio mechanism and the final fine-tuned weights. To attribute the FID gain (10.5 on T2I, 4.81 on inpainting) to the differentiable-ratio mechanism, the paper should include a control that isolates the ratio mechanism—for example, a uniform-ratio model fine-tuned for the same number of steps with the same router architecture, or a DiffCR variant whose routers are frozen while only the ratios are learned. As written, the improvement over the MoD baseline could in principle come from the extra training dynamics or from the different ratio schedule rather than from the surrogate itself.
minor comments (6)
  1. [Sec. 4.4] The sentence 'DiffCR achieves a higher human preference score of 4.685/0.847 compared to previous compression methods, ToMe and vanilla MoD' is ambiguous; the numbers are the differences relative to ToMe and MoD, respectively, and should be stated as such.
  2. [Tables 3 and 5] The HPS tables are not discussed in detail; please clarify how HPSv2 scores are computed for the inpainting task, since HPSv2 was originally designed for text-to-image evaluation.
  3. [Fig. 3] The figure would benefit from stating the model and resolution used and whether the x-axis is the actual compression ratio achieved or the target ratio; the current axis labels 'Compression Ratios (%)' are too terse.
  4. [Sec. 3.3] The phrase 'learning a compression ratio from a zero initialization' is imprecise because the scalar is initialized so that the resulting ratio is 0%, while the MSE target is nonzero; please clarify the initialization and the parameterization.
  5. [Sec. 4.1] The internal 220M-image dataset is described in one sentence; please provide more detail on filtering, mask generation, and availability, since the inpainting results cannot be reproduced otherwise.
  6. [Supplementary Sec. H] The comparison with caching baselines reports 'approximately 25% latency savings' but does not specify how the savings are matched across methods; please state the exact latency and FID for each method.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DiffCR's learned ratios are optimized against the diffusion objective and evaluated on held-out FID, so the central efficiency-quality claims are not predetermined by construction.

full rationale

The paper's central derivation is the differentiable compression-ratio scheme: per-layer and per-timestep scalars are optimized by the diffusion objective plus an explicit MSE term that pulls the batch-averaged ratio to a target. The MSE target is openly stated in Sec. 3.3 ('we incorporate an additional MSE loss between the current learned average ratios across all layers in the batch and the target ratio, which is a hyperparameter'), so convergence of the average ratio to the target is by design, not a disguised prediction. The claims that matter, namely that DiffCR-L and DiffCR-LT improve FID over uniform MoD at comparable latency, are empirical comparisons on held-out LAION sets (Tables 1 and 2), and the learned per-layer and per-timestep ratio patterns are post-hoc observations of trained parameters rather than quantities derived from the target. Self-citations to LazyDiffusion (the base model) and SpeeD (a supporting prior observation) are not load-bearing in a circular sense: the base model is an external architecture, and the SpeeD citation only corroborates the observed noisy-timestep pattern. The differentiable surrogate's train/inference mismatch (blended branches during training versus nearest-bin snap at inference) is a genuine correctness risk, but it is not circular because the held-out FID evaluation is independent of the surrogate's training loss. Overall, the paper's reported trade-offs are measured outcomes, not consequences of how the method is defined.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. Its central claim rests on hand-set target ratios, an MSE coefficient, timestep-region counts, and an unvalidated interpolation proxy for differentiable ratios. These are fitted or chosen parameters rather than derived quantities.

free parameters (5)
  • Target compression ratio (T2I) = 20%
    Chosen empirically; the paper says 20% or 30% offers an optimal trade-off (Sec. 3.2, Tab. 1).
  • Target compression ratio (inpainting) = 50% ViT encoder, 30% DiT decoder
    Used for all DiffCR inpainting runs (Tab. 2); no sensitivity analysis for these targets.
  • MSE loss coefficient for ratio convergence = 0.3 initial, dynamically adjusted
    Balances convergence speed and generation quality (Sec. 3.3, Suppl. B).
  • Number of timestep regions = 10 for inpainting, 4 for T2I
    Hand-picked to balance granularity and training stability (Sec. 3.4, Suppl. D).
  • Discrete ratio bin width = 10% intervals from 0% to 100%
    Defines the lower and upper branches used in differentiable training (Sec. 3.3).
assumptions (4)
  • domain assumption A single linear layer with sigmoid can predict token importance well enough to route compute.
    Used for all routers in Sec. 3.2; no comparison with larger routers or learned importance from attention.
  • ad hoc to paper Linearly interpolating outputs from two hard top-k MoD branches approximates the output at any continuous compression ratio.
    Core to the differentiable ratio scheme (Sec. 3.3, Fig. 1b); never directly validated against evaluating at the exact learned ratio.
  • domain assumption Fine-tuning pretrained DiTs with added routers is a fair basis for comparing compression methods.
    All methods are fine-tuned from the same pretrained models, but the uncontrolled FID change from fine-tuning (151.0 to 11.93 in Tab. 1) suggests sensitivity to this choice.
  • domain assumption FID, CLIP, and HPS values computed on 10K samples are stable enough for the reported comparisons.
    No confidence intervals, seeds, or repeated evaluations are reported (Tabs. 1, 2, 3, 5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers." pith.science (2026). https://pith.science/paper/3LWRQL5D

@misc{pith2026241216822,
  author       = {Pith},
  title        = {Pith review of: Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3LWRQL5D}},
  note         = {Machine review of arXiv:2412.16822}
}
read the original abstract

Diffusion Transformers (DiTs) have achieved state-of-the-art (SOTA) image generation quality but suffer from high latency and memory inefficiency, making them difficult to deploy on resource-constrained devices. One major efficiency bottleneck is that existing DiTs apply equal computation across all regions of an image. However, not all image tokens are equally important, and certain localized areas require more computation, such as objects. To address this, we propose DiffCR, a dynamic DiT inference framework with differentiable compression ratios, which automatically learns to dynamically route computation across layers and timesteps for each image token, resulting in efficient DiTs. Specifically, DiffCR integrates three features: (1) A token-level routing scheme where each DiT layer includes a router that is fine-tuned jointly with model weights to predict token importance scores. In this way, unimportant tokens bypass the entire layer's computation; (2) A layer-wise differentiable ratio mechanism where different DiT layers automatically learn varying compression ratios from a zero initialization, resulting in large compression ratios in redundant layers while others remain less compressed or even uncompressed; (3) A timestep-wise differentiable ratio mechanism where each denoising timestep learns its own compression ratio. The resulting pattern shows higher ratios for noisier timesteps and lower ratios as the image becomes clearer. Extensive experiments on text-to-image and inpainting tasks show that DiffCR effectively captures dynamism across token, layer, and timestep axes, achieving superior trade-offs between generation quality and efficiency compared to prior works. The project website is available at https://www.haoranyou.com/diffcr.

Figures

Figures reproduced from arXiv: 2412.16822 by the authors.

Figure 1
Figure 1. Overview of the proposed DiffCR framework: (a) token-level routing scheme and (b) differentiable compression ratios. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Visualization of the router’s predictions: (a) For inpainting tasks, where inputs are masked images with text prompts, we follow [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of latency and memory savings between our [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Visualization of the learned ratio patterns across both [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 4
Figure 4. Figure 4: Visualization of the compression ratio trajectory during [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 7
Figure 7. Figure 7: Model trajectories. Also, a key benefit of our DiffCR is that during fine-tuning, the average compression ratios across all layers gradually con￾verge to the target ratio, producing a series of “by￾product” models with a range of compression ra￾tios. As shown in [PITH…
Figure 6
Figure 6. Figure 6: Visual comparisons of our DiffCR with previous uncompressed models and SOTA compression methods: (a) Inpainting tasks, [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Visualization of the compression ratio trajectory during [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: More visualizations of the router’s predictions: (a) For inpainting tasks, where inputs are masked images with text prompts, we [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Visualization and analysis of the correlation between the learned compression ratios and the DiffCR router’s predictions. [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Overall comparison of DiffCR with baselines in terms of latency, FID, and TFLOPS for both T2I and inpainting tasks. [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Additional visual comparisons of our DiffCR with previous uncompressed models and SOTA compression methods: (a) [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Model trajectories of DiffCR. and Latent Diffusion (LD) models [28] are more sensitive to pruning and require longer fine-tuning to improve gener￾ation quality effectively, compared to T2I tasks. Moreover, for T2I tasks, DiffCR-LT demonstrates slightly greater sta￾bil…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 27 canonical work pages

  1. [1]

    All are worth words: A vit backbone for diffusion models

    Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 22669–22679, 2023. 1, 2

    work page 2023

  2. [2]

    Token merging for fast sta- ble diffusion

    Daniel Bolya and Judy Hoffman. Token merging for fast sta- ble diffusion. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4599–4603,

  3. [3]

    Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 1, 2, 5

    arXiv

  4. [4]

    Pixart-sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.04692, 2024. 1, 2, 4, 5, 7, 8

    arXiv 2024

  5. [5]

    Diffrate: Differentiable compression rate for efficient vision transformers

    Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffrate: Differentiable compression rate for efficient vision transformers. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 17164– 17174, 2023. 4

    work page 2023

  6. [6]

    Model compression and hardware acceleration for neural networks: A comprehensive survey

    Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4):485–532, 2020. 2

    work page 2020

  7. [7]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informa- tion processing systems, 34:8780–8794, 2021. 2

    2021

  8. [8]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. InInternational Con- ference on Learning Representations, 2020. 4

    work page 2020

Show all 60 references
  1. [9]

    Structural pruning for diffusion models

    Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. In Advances in Neural Infor- mation Processing Systems, 2023. 1, 2

    2023

  2. [10]

    Clipscore: A reference-free evaluation met- ric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation met- ric for image captioning. arXiv preprint arXiv:2104.08718,

    arXiv

  3. [11]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 6, 7

    2017

  4. [12]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 2

    2022 arXiv

  5. [13]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1, 2

    2020

  6. [14]

    Multi-scale dense networks for resource efficient image classification

    Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens Van Der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efficient image classification. arXiv preprint arXiv:1703.09844, 2017. 2

    2017 arXiv

  7. [15]

    Distilling diffusion models into condi- tional gans

    Minguk Kang, Richard Zhang, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, Eli Shechtman, Jun-Yan Zhu, and Taesung Park. Distilling diffusion models into condi- tional gans. ECCV 2024, 2024. 2

    2024

  8. [16]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565–26577, 2022. 2

    2022

  9. [17]

    Bk-sdm: A lightweight, fast, and cheap ver- sion of stable diffusion

    Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi. Bk-sdm: A lightweight, fast, and cheap ver- sion of stable diffusion. arXiv preprint arXiv:2305.15798 ,

    arXiv

  10. [18]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceeding...

    2014

  11. [19]

    Faster diffu- sion via temporal attention decomposition

    Haozhe Liu, Wentian Zhang, Jinheng Xie, Francesco Fac- cio, Mengmeng Xu, Tao Xiang, Mike Zheng Shou, Juan- Manuel Perez-Rua, and J ¨urgen Schmidhuber. Faster diffu- sion via temporal attention decomposition. arXiv preprint arXiv:2404.02747, 2024. 5

    2024 arXiv

  12. [20]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR (Poster), 2019. 6

    2019

  13. [21]

    Dpm-solver: a fast ode solver for diffusion probabilistic model sampling in around 10 steps

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: a fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Pro- ceedings of the 36th International Conference on Neural In- formation Processing Systems, Red Hook, NY ,...

    2024

  14. [22]

    Learning-to-cache: Accelerating diffusion trans- former via layer caching

    Xinyin Ma, Gongfan Fang, Michael Bi Mi, and Xinchao Wang. Learning-to-cache: Accelerating diffusion trans- former via layer caching. arXiv preprint arXiv:2406.01733,

    arXiv

  15. [23]

    On distillation of guided diffusion models

    Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14297–14306, 2023. 1, 2

    2023

  16. [24]

    Early exiting for acceler- ated inference in diffusion models

    Taehong Moon, Moonseok Choi, EungGu Yun, Jongmin Yoon, Gayoung Lee, and Juho Lee. Early exiting for acceler- ated inference in diffusion models. In ICML 2023 Workshop on Structured Probabilistic Inference & Generative Model- ing, 2023. 2

    2023

  17. [25]

    Cache me if you can: Effects of dns time-to-live

    Giovane CM Moura, John Heidemann, Ricardo de O Schmidt, and Wes Hardaker. Cache me if you can: Effects of dns time-to-live. InProceedings of the Internet Measurement Conference, pages 101–115, 2019. 2, 5

    2019

  18. [26]

    Hydranets: Specialized dynamic archi- tectures for efficient inference

    Ravi Teja Mullapudi, William R Mark, Noam Shazeer, and Kayvon Fatahalian. Hydranets: Specialized dynamic archi- tectures for efficient inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8080–8089, 2018. 2

    2018

  19. [27]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of the 38th International Conference on Machine Learning , pages 8162–8171. PMLR, 2021. 6

    2021

  20. [28]

    Lazy diffusion transformer for interactive image editing

    Yotam Nitzan, Zongze Wu, Richard Zhang, Eli Shechtman, Daniel Cohen-Or, Taesung Park, and Micha¨el Gharbi. Lazy diffusion transformer for interactive image editing. arXiv preprint arXiv:2404.12382, 2024. 1, 2, 4, 5, 6, 7, 8

    2024 arXiv

  21. [29]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  22. [30]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 1, 2, 6, 7

    2023 arXiv

  23. [31]

    Ef- ficient diffusion transformer with step-wise dynamic atten- tion mediators

    Yifan Pu, Zhuofan Xia, Jiayi Guo, Dongchen Han, Qixiu Li, Duo Li, Yuhui Yuan, Ji Li, Yizeng Han, Shiji Song, et al. Ef- ficient diffusion transformer with step-wise dynamic atten- tion mediators. arXiv preprint arXiv:2408.05710, 2024. 2

    2024 arXiv

  24. [32]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937–13949,

  25. [33]

    Mixture-of-depths: Dynamically allocating com- pute in transformer-based language models

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam San- toro. Mixture-of-depths: Dynamically allocating com- pute in transformer-based language models. arXiv preprint arXiv:2404.02258, 2024. 2, 3, 8, 1, 5

    2024 arXiv

  26. [34]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 2, 6

    2022

  27. [35]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022. 1, 2

    2022 arXiv

  28. [36]

    Adversarial diffusion distillation

    Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In European Conference on Computer Vision , pages 87–103. Springer,

  29. [37]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 6, 7

    2021 arXiv

  30. [38]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  31. [39]

    Todo: Token downsampling for efficient generation of high-resolution im- ages

    Ethan Smith, Nayan Saxena, and Aninda Saha. Todo: Token downsampling for efficient generation of high-resolution im- ages. arXiv preprint arXiv:2402.13573, 2024. 1, 2

    2024 arXiv

  32. [40]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265. PMLR, 2015. 2

    2015

  33. [41]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 1, 2

    2010 arXiv

  34. [42]

    Branchynet: Fast inference via early exiting from deep neural networks

    Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd international con- ference on pattern recognition (ICPR) , pages 2464–2469. IEEE, 2016. 2

    2016

  35. [43]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 2

    2017

  36. [44]

    Diffusers: State-of-the-art diffu- sion models

    Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffu- sion models. https://github.com/huggingface/ diffusers...

    2022

  37. [45]

    Attention-driven training-free efficiency enhancement of diffusion models

    Hongjie Wang, Difan Liu, Yan Kang, Yijun Li, Zhe Lin, Ni- raj K Jha, and Yuchen Liu. Attention-driven training-free efficiency enhancement of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 16080–16089, 2024. 1, 2...

    2024

  38. [46]

    A closer look at time steps is worthy of triple speed-up for diffusion model training

    Kai Wang, Yukun Zhou, Mingjia Shi, Zhihang Yuan, Yuzhang Shang, Xiaojiang Peng, Hanwang Zhang, and Yang You. A closer look at time steps is worthy of triple speed-up for diffusion model training. arXiv preprint arXiv:2405.17403, 2024. 1, 2, 6, 3

    2024 arXiv

  39. [47]

    Imagen editor and editbench: Advancing and evaluating text-guided im- age inpainting

    Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont- Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating text-guided im- age inpainting. In Proceedings of the IEEE/CVF c...

    2023

  40. [48]

    Skipnet: Learning dynamic routing in convolutional networks

    Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In Proceedings of the Euro- pean conference on computer vision (ECCV) , pages 409– 424, 2018. 2

    2018

  41. [49]

    Dual dynamic inference: Enabling more efficient, adap- tive, and controllable deep inference

    Yue Wang, Jianghao Shen, Ting-Kuei Hu, Pengfei Xu, Tan Nguyen, Richard Baraniuk, Zhangyang Wang, and Yingyan Lin. Dual dynamic inference: Enabling more efficient, adap- tive, and controllable deep inference. IEEE Journal of Se- lected Topics in Signal Processing, 14(4):623–633...

    2020

  42. [50]

    https : / / github

    Stable Diffusion WebUI. https : / / github . com / AUTOMATIC1111 / stable - diffusion - webui ,

  43. [51]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341 ,

    arXiv

  44. [52]

    Blockdrop: Dynamic inference paths in residual networks

    Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8817–8826, 2018. 2

    2018

  45. [53]

    Smartbrush: Text and shape guided object inpainting with diffusion model

    Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22428–22437, 2023. 6

    2023

  46. [54]

    Deepcache: Principled cache for mo- bile deep vision

    Mengwei Xu, Mengze Zhu, Yunxin Liu, Felix Xiaozhu Lin, and Xuanzhe Liu. Deepcache: Principled cache for mo- bile deep vision. In Proceedings of the 24th annual inter- national conference on mobile computing and networking , pages 129–144, 2018. 2, 5

    2018

  47. [55]

    Im- proved distribution matching distillation for fast image syn- thesis

    Tianwei Yin, Micha ¨el Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Im- proved distribution matching distillation for fast image syn- thesis. arXiv preprint arXiv:2405.14867, 2024. 2

    2024 arXiv

  48. [56]

    One-step diffusion with distribution matching distillation

    Tianwei Yin, Micha ¨el Gharbi, Richard Zhang, Eli Shecht- man, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition , pages 6613–662...

    2024

  49. [57]

    Not all tokens are equal: Human-centric visual analysis via token clustering transformer

    Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Ping Luo, Wanli Ouyang, and Xiaogang Wang. Not all tokens are equal: Human-centric visual analysis via token clustering transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11101– ...

    2022

  50. [58]

    Dynamic diffusion transformer

    Wangbo Zhao, Yizeng Han, Jiasheng Tang, Kai Wang, Yib- ing Song, Gao Huang, Fan Wang, and Yang You. Dynamic diffusion transformer. arXiv preprint arXiv:2410.03456 ,

    arXiv

  51. [2023]

    Accessed: 2024-11-10. 6

    2024

  52. [2024]

    eel sushi roll

    2 Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers Supplementary Material A. More Visualization of Token Routers In Sec. 3.2, we provided an example visualization of the router predictions to evaluate the effectiveness o...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.