Pith. sign in

REVIEW 7 cited by

CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.16112 v1 pith:NB4ESE4V submitted 2024-12-20 cs.CV

Songhua Liu , Zhenxiong Tan , Xinchao Wang This is my paper · ORCID
classification cs.CV
keywords attentioncomplexitylinearpre-trainedcleardiffusionditsfeature
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Diffusion Transformers (DiT) have become a leading architecture in image generation. However, the quadratic complexity of attention mechanisms, which are responsible for modeling token-wise relationships, results in significant latency when generating high-resolution images. To address this issue, we aim at a linear attention mechanism in this paper that reduces the complexity of pre-trained DiTs to linear. We begin our exploration with a comprehensive summary of existing efficient attention mechanisms and identify four key factors crucial for successful linearization of pre-trained DiTs: locality, formulation consistency, high-rank attention maps, and feature integrity. Based on these insights, we introduce a convolution-like local attention strategy termed CLEAR, which limits feature interactions to a local window around each query token, and thus achieves linear complexity. Our experiments indicate that, by fine-tuning the attention layer on merely 10K self-generated samples for 10K iterations, we can effectively transfer knowledge from a pre-trained DiT to a student model with linear complexity, yielding results comparable to the teacher model. Simultaneously, it reduces attention computations by 99.5% and accelerates generation by 6.3 times for generating 8K-resolution images. Furthermore, we investigate favorable properties in the distilled attention layers, such as zero-shot generalization cross various models and plugins, and improved support for multi-GPU parallel inference. Models and codes are available here: https://github.com/Huage001/CLEAR.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MatchAttention: Embedding Explicit Matching Constraints into Attention for Efficient Stereo Matching

    cs.CV 2025-10 conditional novelty 6.0 of 10

    MatchAttention replaces global cross-attention with a linearly-scaling, learnable-offset windowed attention whose relative position field is the predicted disparity/flow, yielding state-of-the-art stereo/flow accuracy...

  2. LSSGen: Leveraging Latent Space Scaling in Flow and Diffusion for Efficient Text to Image Generation

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A latent-space scaling framework that replaces pixel-space upscaling with a trainable latent upsampler and noise compensation, yielding faster high-resolution text-to-image generation.

  3. Exploring Diffusion Transformer Designs via Grafting

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Grafting uses activation distillation and lightweight fine-tuning to edit pretrained diffusion transformers into hybrid architectures with near-baseline quality at under 2% pretraining compute.

  4. Long-Context State-Space Video World Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A hybrid state-space and local-attention architecture gives autoregressive video diffusion models long-term spatial memory with constant per-frame inference cost, demonstrated on Maze and Minecraft.

  5. Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformers

    cs.CV 2025-05 conditional novelty 6.0 of 10

    GRAT accelerates pretrained diffusion transformers by grouping tokens and restricting each group's attention to neighboring blocks or criss-cross rows and columns, achieving large speedups with near-full-attention quality.

  6. UltraFlux: Data-Model Co-Design for High-quality Native 4K Text-to-Image Generation across Diverse Aspect Ratios

    cs.CV 2025-11 conditional novelty 5.0 of 10

    A data-model co-designed Flux DiT with resonance RoPE, post-trained F16 VAE, SNR-aware Huber wavelet loss, and aesthetic curriculum claims state-of-the-art open-source 4K multi-AR text-to-image generation.

  7. Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Sparse-vDiT replaces dense attention with fixed per-head sparse patterns chosen offline, achieving 1.58-1.85x end-to-end speedups on CogVideoX1.5, HunyuanVideo, and Wan2.1 with minimal quality loss.

Pith tools