REVIEW 19 cited by
Faster Diffusion via Temporal Attention Decomposition
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We explore the role of attention mechanism during inference in text-conditional diffusion models. Empirical observations suggest that cross-attention outputs converge to a fixed point after several inference steps. The convergence time naturally divides the entire inference process into two phases: an initial phase for planning text-oriented visual semantics, which are then translated into images in a subsequent fidelity-improving phase. Cross-attention is essential in the initial phase but almost irrelevant thereafter. However, self-attention initially plays a minor role but becomes crucial in the second phase. These findings yield a simple and training-free method known as temporally gating the attention (TGATE), which efficiently generates images by caching and reusing attention outputs at scheduled time steps. Experimental results show when widely applied to various existing text-conditional diffusion models, TGATE accelerates these models by 10%-50%. The code of TGATE is available at https://github.com/HaozheLiu-ST/T-GATE.
Forward citations
Cited by 19 Pith papers
-
SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editing
By rectifying editing instructions with GPT-4o and adding a triplet loss on positive and negative instructions, a standard InstructPix2Pix model trained on 40K samples outperforms the 14.1B-parameter SmartEdit on Real-Edit.
-
Token Radius Attention for Efficient Video Generation
Video diffusion transformers can run ~1.5-2x faster with competitive quality by converting each query's attention entropy into a spatially decayed retention radius instead of dense attention.
-
Steering Optimisation Trajectories in Diffusion Representation Learning
SteeringDRL identifies two optimization regimes in diffusion autoencoders and uses gated residual U-Nets with a log SNR curriculum to steer training toward disentangled representations, improving performance across mu...
-
Transition Matching Distillation for Fast Video Generation
Splitting a video diffusion model into a fixed feature extractor and a small recurrent flow head lets TMD generate videos in one to two effective steps with better VBench scores than prior distilled models.
-
OmniCache: A Trajectory-Oriented Global Perspective on Training-Free Cache Reuse for Diffusion Transformer Models
A training-free cache-reuse scheme that spreads computation across the full diffusion trajectory and subtracts estimated noise, accelerating DiT sampling with claimed competitive quality.
-
DDiT: Dynamic Resource Allocation for Diffusion Transformer Model Serving
DDiT cuts text-to-video serving latency by up to 1.44x via DiT-VAE phase decoupling and step-level, starvation-aware GPU reassignment.
-
dKV-Cache: The Cache for Diffusion Language Models
dKV-Cache reuses cached key and value states of decoded tokens during diffusion LM denoising, delivering 2-10x faster inference with near-lossless quality on several benchmarks.
-
Can We Achieve Efficient Diffusion without Self-Attention? Distilling Self-Attention into Convolutions
Diffusion models can keep most of their image quality after swapping global self-attention for a distilled multi-scale convolutional block, with FLOPs claimed to fall by up to 6929 times at 16K resolution.
-
CAT Pruning: Cluster-Aware Token Pruning For Text-to-Image Diffusion Models
A token-pruning cache method cuts diffusion model computation by roughly half while keeping image quality, using noise magnitude, spatial clustering, and selection balance.
-
Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers
DiffCR learns per-layer and per-timestep token compression ratios for diffusion transformers, improving FID at similar latency relative to uniform token pruning.
-
AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration
AsymRnR selectively reduces query and key/value tokens in video DiT attention to cut FLOPs and latency by 10 to 30 percent with minor or no VBench score change.
-
Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model
TeaCache speeds up video diffusion models by caching model outputs when the difference between timestep-embedding-modulated inputs is small, after polynomial rescaling calibrated on a small prompt set.
-
Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient
CoDe speeds up Visual Auto-Regressive image generation by using a 2B model for early coarse scales and a 0.3B model for later fine scales, with 1.7x-2.9x speedup and only a small FID increase.
-
Latent Space Disentanglement in Diffusion Transformers Enables Precise Zero-shot Semantic Editing
Diffusion transformer latent spaces are shown to be semantically disentangled, and prompt-difference directions plus a score-distillation step enable zero-shot fine-grained image editing.
-
EC-Diff: Fast and High-Quality Edge-Cloud Collaborative Inference for Diffusion Models
EC-Diff accelerates edge-cloud diffusion inference with a k-step noise approximation strategy and a two-stage greedy search for the cloud-edge handoff point, claiming about 2x speedup with preserved quality.
-
Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression
ScaleKV cuts KV cache memory for Visual Autoregressive text-to-image generation to 10% by classifying layers as drafters or refiners per scale and pruning low-attention tokens while keeping benchmark scores nearly unchanged.
-
Cached Adaptive Token Merging: Dynamic Token Reduction and Redundant Computation Elimination in Diffusion Model
CA-ToMe speeds up Stable Diffusion inference by adaptively merging similar tokens and caching merge pairs, reaching 6.09s average generation time with FID 34.05 on ImageNet.
-
Unveiling Redundancy in Diffusion Transformers (DiTs): A Systematic Study
Redundancy between consecutive diffusion steps varies widely across DiT models but is stable within each model across prompts, step counts, and schedulers, so caching strategies must be model-specific.
-
SkipVAR: Accelerating Visual Autoregressive Modeling via Adaptive Frequency-Aware Skipping
SkipVAR selects, per sample, between step skipping and unconditional branch replacement using handcrafted frequency features and a trained logistic regression, to accelerate visual autoregressive generation.
Discussion (0). Continue with ORCID to comment.