Pith. sign in

REVIEW 8 cited by

Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.01733 v2 pith:FPQMF4VH submitted 2024-06-03 cs.LG cs.CV

Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching

Xinyin Ma , Gongfan Fang , Michael Bi Mi , Xinchao Wang This is my paper
classification cs.LG cs.CV
keywords diffusioncachingcomputationinferencelayerslearning-to-cachetransformertransformers
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Diffusion Transformers have recently demonstrated unprecedented generative capabilities for various tasks. The encouraging results, however, come with the cost of slow inference, since each denoising step requires inference on a transformer model with a large scale of parameters. In this study, we make an interesting and somehow surprising observation: the computation of a large proportion of layers in the diffusion transformer, through introducing a caching mechanism, can be readily removed even without updating the model parameters. In the case of U-ViT-H/2, for example, we may remove up to 93.68% of the computation in the cache steps (46.84% for all steps), with less than 0.01 drop in FID. To achieve this, we introduce a novel scheme, named Learning-to-Cache (L2C), that learns to conduct caching in a dynamic manner for diffusion transformers. Specifically, by leveraging the identical structure of layers in transformers and the sequential nature of diffusion, we explore redundant computations between timesteps by treating each layer as the fundamental unit for caching. To address the challenge of the exponential search space in deep models for identifying layers to cache and remove, we propose a novel differentiable optimization objective. An input-invariant yet timestep-variant router is then optimized, which can finally produce a static computation graph. Experimental results show that L2C largely outperforms samplers such as DDIM and DPM-Solver, alongside prior cache-based methods at the same inference speed. Code is available at https://github.com/horseee/learning-to-cache

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DisCa: Accelerating Video Diffusion Transformers with Distillation-Compatible Learnable Feature Caching

    cs.CV 2026-02 unverdicted novelty 7.0

    DisCa replaces heuristic feature caching with a lightweight learnable neural predictor compatible with distillation, achieving 11.8× acceleration on video diffusion transformers with preserved generation quality.

  2. Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models

    cs.LG 2026-02 unverdicted novelty 7.0

    Early and late denoising steps in masked diffusion LMs are robust to smaller-model replacement, enabling 17% FLOPs reduction with modest generative quality loss.

  3. DiTango: Cost-Effective Parallel Diffusion Generation with Selective Attention State Reuse

    cs.CV 2026-07 conditional novelty 6.0

    Selective reuse of composed attention states across denoising steps lets DiTango skip both remote KV communication and attention compute for low-contribution sequence partitions, cutting multi-GPU diffusion latency by...

  4. CoCoDiff: Optimizing Collective Communications for Distributed Diffusion Transformer Inference Under Ulysses Sequence Parallelism

    cs.DC 2026-04 unverdicted novelty 6.0

    CoCoDiff achieves 3.6x average and 8.4x peak speedup for distributed DiT inference on up to 96 GPU tiles via tile-aware all-to-all, V-first scheduling, and selective V communication.

  5. S2O: Early Stopping for Sparse Attention via Online Permutation

    cs.LG 2026-02 unverdicted novelty 6.0

    S2O uses online permutation and importance-based early stopping to increase effective sparsity in attention, delivering 7.51x attention and 3.81x end-to-end speedups on Llama-3.1-8B at 128K context with preserved accuracy.

  6. LESA: Learnable Stage-Aware Predictors for Diffusion Model Acceleration

    cs.CV 2026-02 conditional novelty 6.0

    A learned, stage-segmented KAN predictor for feature caching accelerates diffusion transformers by 5-6.25x while preserving more image/video fidelity than prior training-free forecasters.

  7. PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformers Inference

    cs.CV 2024-05 unverdicted novelty 6.0

    PipeFusion applies patch partitioning and pipeline parallelism with one-step stale feature reuse to reduce communication overhead in DiT inference, reporting SOTA results on 8x L40 GPUs for Pixart, SD3, and Flux.1.

  8. AdaCorrection: Adaptive Offset Cache Correction for Accurate Diffusion Transformers

    cs.CV 2026-02 unverdicted novelty 5.0

    AdaCorrection adaptively corrects offset caches in DiT inference via on-the-fly spatio-temporal validity checks to maintain near-original FID with moderate acceleration.