REVIEW 6 cited by
Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
One of the main drawback of diffusion models is the slow inference time for image generation. Among the most successful approaches to addressing this problem are distillation methods. However, these methods require considerable computational resources. In this paper, we take another approach to diffusion model acceleration. We conduct a comprehensive study of the UNet encoder and empirically analyze the encoder features. This provides insights regarding their changes during the inference process. In particular, we find that encoder features change minimally, whereas the decoder features exhibit substantial variations across different time-steps. This insight motivates us to omit encoder computation at certain adjacent time-steps and reuse encoder features of previous time-steps as input to the decoder in multiple time-steps. Importantly, this allows us to perform decoder computation in parallel, further accelerating the denoising process. Additionally, we introduce a prior noise injection method to improve the texture details in the generated image. Besides the standard text-to-image task, we also validate our approach on other tasks: text-to-video, personalized generation and reference-guided generation. Without utilizing any knowledge distillation technique, our approach accelerates both the Stable Diffusion (SD) and DeepFloyd-IF model sampling by 41$\%$ and 24$\%$ respectively, and DiT model sampling by 34$\%$, while maintaining high-quality generation performance.
Forward citations
Cited by 6 Pith papers
-
EchoCache: Energy-Guided Cross-Modal Caching for Efficient Audio-Driven Video Generation
Audio time-frequency energy guides which video latents get recomputed during diffusion denoising, yielding up to 2.46x faster audio-driven video generation with competitive quality.
-
DiffSparse: Accelerating Diffusion Transformers with Learned Token Sparsity
A learnable cost predictor plus dynamic programming allocates layer-wise token sparsity for diffusion transformers, removing forced full steps and cutting ~54% compute on PixArt-α without quality loss.
-
LESA: Learnable Stage-Aware Predictors for Diffusion Model Acceleration
A learned, stage-segmented KAN predictor for feature caching accelerates diffusion transformers by 5-6.25x while preserving more image/video fidelity than prior training-free forecasters.
-
Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
Morse accelerates pretrained diffusion models by interleaving jump-sampling steps of the original model with a faster learned residual-correction model, reporting lossless average speedups of 1.78x to 3.31x.
-
Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression
ScaleKV cuts KV cache memory for Visual Autoregressive text-to-image generation to 10% by classifying layers as drafters or refiners per scale and pruning low-attention tokens while keeping benchmark scores nearly unchanged.
-
Predictive Feature Caching for Training-free Acceleration of Molecular Geometry Generation
Predictive feature caching, borrowed from image diffusion, speeds up molecular flow-matching generation by 2-3x at near-matched quality by forecasting hidden features instead of recomputing them.
Discussion (0). Continue with ORCID to comment.