Pith. sign in

REVIEW 3 major objections 6 minor 10 cited by

CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A pre-trained diffusion transformer can be made linear by replacing every attention layer with a circular local-window mechanism and fine-tuning only those layers for 10K iterations, yielding output comparable to the original.

desk verdict CLEAR is neighborhood attention with a circular window plus a smart distillation recipe; the aggregate parity claim holds, but the closest baseline is missing from the quantitative table and the paper's own symmetry caveat should temper how the result is sold. read the letter →

arxiv 2412.16112 v1 pith:NB4ESE4V submitted 2024-12-20 cs.CV

Songhua Liu , Zhenxiong Tan , Xinchao Wang This is my paper · ORCID
classification cs.CV
keywords diffusiontransformerslinearattentionlocalwindowknowledgedistillationhigh-resolutionimagegenerationefficientinferencetext-to-imageFLUX
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a pre-trained Diffusion Transformer can be converted from quadratic to linear attention without losing generative quality, and answers yes on the FLUX-1.dev text-to-image model. The recipe is CLEAR, a convolution-like local attention in which each image query attends to all text tokens plus image tokens inside a circular window of fixed radius, so per-query work is constant and total cost is linear in the number of image tokens. Only the attention layers are fine-tuned, for 10K iterations on 10K images generated by the teacher itself, using a distillation objective that also aligns predictions and per-layer attention output with the frozen teacher. The student then scores comparably to the teacher (against-real FID 34.88 versus 34.93, CLIP-T 31.00 versus 31.06 at r=32), cuts attention computation by 99.5 percent at 8K resolution, and generates 8K images 6.3 times faster. If the claim holds, expensive pre-trained DiTs can be made linear for about a day of fine-tuning, with the distilled local layers transferring zero-shot to sibling models and plugins.

What carries the argument

The load-bearing object is CLEAR's circular-window attention mask: a fixed-radius Euclidean neighborhood around each query that keeps the per-query key-value count constant as resolution grows, making overall complexity $\mathcal{O}(n)$ in the number of image tokens and costing about $\pi/4$ the compute of a square window. It is the only scheme among the surveyed families (formulation variation, key-value compression, key-value sampling) that satisfies all four factors the paper identifies as essential: locality, formulation consistency with softmax attention, high-rank attention maps, and preservation of raw query/key/value features. The second mechanism is the distillation objective: the flow-matching loss $\mathcal{L}_{\text{fm}}$ together with prediction consistency $\mathcal{L}_{\text{pred}}$ and per-layer attention-output consistency $\mathcal{L}_{\text{attn}}$ against the frozen teacher, trained on 10K self-generated samples chosen for distributional closeness to the teacher's training data. The third is the patch-wise text-token approximation $O_{\text{text}}\approx \frac{1}{N}\sum_{p=1}^{N}\mathrm{softmax}(Q_{\text{text}}K_p^{\top}/\sqrt{c})V_p$, which removes the need to synchronize all key-value tokens for text queries and thereby makes multi-GPU patch-parallel inference communication-light.

What would settle it

Compute, for FLUX-1.dev, the fraction of image-to-image attention mass that falls outside the r=32 circular window, aggregated over all attention layers and denoising timesteps; if that fraction is substantial and a linearized student trained with the same 10K-iteration recipe on any DiT with such long-range heads fails to reach teacher parity, the locality premise is refuted. A complementary check supplied by the paper itself is wall-clock speed: at 1024x1024 and below, CLEAR is no faster than the original model, so the linear-time advantage is an empirical claim that must be re-measured at each target resolution.

Watch

Extended reading notes

Core claim

The central claim is that pre-trained DiTs such as FLUX-1.dev rely predominantly on local token interactions, so long-range image-to-image attention can be discarded and re-learned locally without hurting generation quality. The paper supports this with attention-map visualizations and with rotary-position-embedding perturbation experiments: clipping remote feature distances to a small radius leaves output largely intact, while perturbing local distances collapses it. On this basis it replaces every attention layer with CLEAR, whose mask admits a text-image pair whenever either token is a text token or the two image tokens lie within Euclidean radius $r$ ($M_{ij}=1$ if $i\le n_{\text{text}}$ or $j\le n_{\text{text}}$ or ${d^{(x)}_{ij}}^2+{d^{(y)}_{ij}}^2<r^2$). After fine-tuning attention parameters only, with the flow-matching loss plus prediction and attention-consistency losses against the frozen teacher, the student matches teacher quality at 1024x1024 and extrapolates to 4K and 8K, where attention compute falls by 99.5 percent and generation speeds up 6.3 times. The distilled layers also transfer without adaptation to FLUX-1.schnell and to pre-trained ControlNet, and enable patch-parallel multi-GPU inference with constant communication overhead for text tokens.

Load-bearing premise

The method rests on the premise that a pre-trained DiT like FLUX-1.dev conducts its image-to-image attention almost entirely locally, so deleting all long-range image attention and re-learning with a circular window of radius at most 32 loses nothing essential; that premise is supported by attention-map visualizations and rotary-position perturbation tests on one model, not by a quantitative accounting of attention mass across layers and timesteps.

Editorial extensions

If this is right

  • Ultra-high-resolution generation becomes practical on a single GPU: at 8K the attention computation drops 99.5 percent and wall-clock generation accelerates 6.3 times relative to the original model.
  • One distillation serves a model family: the CLEAR layers trained on FLUX-1.dev run zero-shot on FLUX-1.schnell and with pre-trained ControlNet without any adaptation.
  • Patch-parallel multi-GPU inference scales with resolution, requiring only boundary communication of size $O(r \times W)$ between adjacent patches, where full-attention DiTs run out of memory.
  • The conversion cost is small: only attention parameters are trained, on 10K self-generated samples for 10K iterations, so a strong linearized DiT is obtained in roughly one day of fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The locality premise is demonstrated on a single model, so the same recipe should be stress-tested on other DiT families; if attention there carries more long-range mass, larger radii or a few global tokens would be the obvious repair, and the paper's fully specified mask, loss, and data recipe make that test immediate.
  • The superiority of 10K self-generated images over a larger real-data corpus suggests the distillation succeeds by distribution matching between teacher and student, implying dataset size matters far less than distributional overlap for architectural distillation of this kind.
  • The patch-wise text-attention approximation shows that even text conditioning can be computed almost independently per patch, hinting that the text branch could itself be pushed toward sub-linear cost at some quality margin.
  • The authors' own caveats locate the method's operating range: the appendix concedes that absent explicit long-range interaction the model may miss global structure such as symmetry, and the conclusion concedes that wall-clock speedup disappears below 1024x1024 because sparse attention lacks fused kernels; both point to hybrid designs with a few global tokens and to custom sparse-attention kernels
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes CLEAR, a convolution-like local circular-window attention mechanism that replaces all attention layers in a pre-trained diffusion transformer while keeping text-token attention global. Only attention-layer parameters are fine-tuned, using 10K self-generated 1024x1024 samples and 10K iterations with a distillation objective combining flow matching, prediction matching, and attention-output matching. The authors report aggregate quality comparable to the original FLUX-1.dev on 5,000 COCO validation images (e.g., against-real FID 34.88 vs 34.93 at r=32), a 99.5% reduction in attention computation, and a 6.3x wall-clock speedup at 8K resolution. They also report zero-shot transfer to FLUX-1.schnell and ControlNet, and multi-GPU patch-parallel inference.

Significance. If the parity and efficiency results hold, CLEAR is a simple and practical way to convert a pre-trained DiT into a linear-complexity model, which is directly relevant for ultra-high-resolution generation. The paper's strengths include independent against-real metrics at 1024x1024 (FID, LPIPS, CLIP-T), arithmetically consistent efficiency numbers in Tab. 7, cross-model and plugin generalization experiments, and public code and models. The main caveat is that the headline 'comparable results' claim rests on aggregate COCO statistics and on a locality premise that is supported by qualitative evidence only; the paper's own Appendix D concedes a possible weakness on global structural properties such as symmetry.

major comments (3)
  1. [Appendix D; Sec. 3.3; Tab. 2] Appendix D concedes that 'due to the absence of explicit long-distance token interactions, our method may underperform in capturing overall structural properties, such as potential symmetry.' This is load-bearing for the central parity claim, because with only attention parameters trainable (Sec. 3.3), the student cannot recover image-wide layout except by routing information through many local hops. Aggregate against-real FID/LPIPS/CLIP-T numbers in Tab. 2 cannot detect systematic failures on prompts requiring mirror symmetry, reflection, or multi-object spatial relations. Please add a targeted stress-test suite or per-category breakdown (e.g., symmetry, object counting, relative positions), and if any category degrades, qualify the 'comparable results' claim accordingly.
  2. [Tab. 2; Sec. 4.2] The main parity comparison is reported without any error bars or repeated-run statistics. Since the headline differences are small (against-real FID 34.88 vs 34.93; CLIP-T 31.00 vs 31.06 for r=32), it is important to know the run-to-run variability of the evaluation metrics. Please report multiple evaluation seeds or bootstrap confidence intervals on the 5,000-image sample, so that 'comparable to the teacher' can be distinguished from noise.
  3. [Sec. 3.1; Fig. 4; Fig. 5] The premise that pre-trained DiTs rely primarily on local interactions is supported by a single attention-map visualization and one RoPE perturbation experiment at one scale and on one model. Fig. 4 shows selected heads at an intermediate denoising step, and clipping RoPE relative distances is not equivalent to zeroing the attention weights. Please quantify locality across layers and timesteps (e.g., fraction of attention mass within radius r), and report the same locality check for SD3.5-L to substantiate the claimed cross-architecture generality of the locality principle.
minor comments (6)
  1. [Sec. 3.2] The phrase 'Motivated on this' should read 'Motivated by this.'
  2. [Sec. 3.3] The sentence 'where z0 is denotes the feature' contains a grammatical error; it should read 'where z0 denotes the feature.'
  3. [Sec. 4.1] The sentence 'Lattn is applied on single transformer blocks of FLUX, whose layer indices are 20 ~ 57' is ambiguous; please specify whether Lattn is applied to all blocks 20-57 or to a selected subset.
  4. [Sec. 4.3; Tab. 10] The caption of Tab. 10 explains why r=16 at 1024x1024 is unavailable, but the main text does not mention this limitation of the multi-GPU scheme at lower resolutions; please add a sentence.
  5. [Tab. 1; Sec. 2.3] The 'Maybe' entry for Agent Attention in the Locality column is never explained; a one-sentence clarification would help the reader interpret the table.
  6. [Eq. (7)] In Eq. (7), the notation Vp is used without specifying whether it contains only image tokens of patch p or includes text tokens as well; please clarify.

Circularity Check

1 steps flagged · score 4.0 of 10

Against-original metrics mirror the distillation objective; independent against-real benchmarks keep the core claim standing.

  1. fitted input called prediction [Section 3.3 Eq. (5) vs Section 4.2 Table 2 discussion]
    "With the knowledge distillation loss terms defined in Eq. 5, the differences between outputs from the linearized models and the original model are further minimized. For instance, the CLIP Image score exceeds 90 when r = 32."

    Eq. (5) defines L_pred and L_attn as per-sample L2 distances between the student's denoising predictions/attention outputs and the teacher's. Because only attention parameters are trainable (Sec. 3.3) and this exact distance is minimized during fine-tuning, the 'Against Original' columns of Tab. 2/Tab. 3 (e.g., CLIP-I 92.18, DINO 85.44, FID 8.85 for r=32 with distill) are measurements of the optimized objective, not independent evidence that CLEAR is teacher-comparable. The paper's own sentence makes this explicit by attributing the high CLIP-I score directly to the distillation loss terms. Presenting these numbers as support for the claim of 'comparable or even superior performance' is therefore circular for that metric.

full rationale

The paper's derivation is largely empirical and self-contained. The four design factors (locality, formulation consistency, high-rank attention maps, feature integrity) are motivated by controlled fine-tuning comparisons and by evidence on the frozen teacher (attention-map visualization in Fig. 4 and RoPE clipping in Fig. 5), not by a self-citation chain. The only self-citation to LinFusion [38] is used for hyper-parameter choices and as a comparison baseline, so it is not load-bearing. The main circular element is the 'Against Original' evaluation: Eq. (5) explicitly trains the student to minimize L2 distance to teacher denoising predictions and attention outputs, and Tables 2/3 then report student-teacher FID/LPIPS/CLIP-I/DINO as evidence of success. Those numbers are measurements of the optimized objective and hence are not independent confirmation. This is why the teacher-comparable claim is only partially circular: the paper also reports against-real FID and CLIP-T on held-out COCO (34.88 vs 34.93; 31.00 vs 31.06) and zero-shot transfer to FLUX-1.schnell and ControlNet with no adaptation, which are external and not reduced by Eq. (5). Appendix D's concession that long-range structure such as symmetry may suffer is a correctness/generalization caveat, not a circularity. Overall score 4 reflects one real but non-central circular evaluation mode.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on one user-chosen architectural parameter (window radius r), two distillation weights taken from prior work, and several empirical domain assumptions about locality, training data, and the text-attention approximation. No fundamentally new entities (e.g., new forces or particles) are introduced; the circular window is a masking scheme.

free parameters (4)
  • local window radius r = 8, 16, 32; r=16 used as default
    Main architectural knob. The paper evaluates three radii and selects r=16 as the default trade-off for SD3.5 and multi-GPU experiments. The linear-complexity claim holds for any fixed r, but the quality numbers depend on the chosen value.
  • distillation weights alpha and beta = 0.5 each
    Set following prior architectural distillation works (BK-SDM [31], LinFusion [38], the latter by the same authors); not tuned in this paper.
  • NTK-aware RoPE scale factor = 10
    Adopted from [44] for high-resolution extrapolation; used for the 4K/8K results in Fig. 9 and Tab. 3.
  • SDEdit editing strength = 0.5 and 0.7
    Controls the coarse-to-fine high-resolution generation in Fig. 9; chosen by the user per example.
assumptions (5)
  • domain assumption Pre-trained DiTs rely primarily on local token interactions (locality)
    Loaded in Sec. 3.1-3.2 to justify replacing global attention with a local window; evidence is attention-map visualization (Fig. 4) and RoPE perturbation experiments (Fig. 5) on FLUX-1.dev only.
  • domain assumption 10K self-generated samples from the teacher suffice for effective distillation
    The whole cheap-fine-tuning claim rests on this; Sec. 4.3 compares to LAION but does not scale the dataset size.
  • ad hoc to paper The patch-wise average approximation for text-token attention (Eq. 7) preserves output quality
    Used for multi-GPU inference; validated only empirically in Tab. 6, no derivation.
  • domain assumption Flow matching and distillation losses (Eqs. 4-6) are sufficient to transfer teacher behavior to a local-attention student
    Adopted from prior distillation literature; no proof that these losses guarantee parity, only empirical results.
  • standard math Softmax attention as defined in Eq. 1 is the correct reference and its quadratic complexity is the bottleneck
    Standard definition from Vaswani et al.; the paper's complexity analysis is based on it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up." pith.science (2026). https://pith.science/paper/NB4ESE4V

@misc{pith2026241216112,
  author       = {Pith},
  title        = {Pith review of: CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NB4ESE4V}},
  note         = {Machine review of arXiv:2412.16112}
}
read the original abstract

Diffusion Transformers (DiT) have become a leading architecture in image generation. However, the quadratic complexity of attention mechanisms, which are responsible for modeling token-wise relationships, results in significant latency when generating high-resolution images. To address this issue, we aim at a linear attention mechanism in this paper that reduces the complexity of pre-trained DiTs to linear. We begin our exploration with a comprehensive summary of existing efficient attention mechanisms and identify four key factors crucial for successful linearization of pre-trained DiTs: locality, formulation consistency, high-rank attention maps, and feature integrity. Based on these insights, we introduce a convolution-like local attention strategy termed CLEAR, which limits feature interactions to a local window around each query token, and thus achieves linear complexity. Our experiments indicate that, by fine-tuning the attention layer on merely 10K self-generated samples for 10K iterations, we can effectively transfer knowledge from a pre-trained DiT to a student model with linear complexity, yielding results comparable to the teacher model. Simultaneously, it reduces attention computations by 99.5% and accelerates generation by 6.3 times for generating 8K-resolution images. Furthermore, we investigate favorable properties in the distilled attention layers, such as zero-shot generalization cross various models and plugins, and improved support for multi-GPU parallel inference. Models and codes are available here: https://github.com/Huage001/CLEAR.

Figures

Figures reproduced from arXiv: 2412.16112 by the authors.

Figure 1
Figure 1. Ultra-resolution results generated by the linearized FLUX.1-dev model with our approach CLEAR. Resolution is marked on the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of speed and GFLOPS between the pro [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Preliminary results of various efficient attention methods on FLUX-1.dev. The prompt is “ [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: We try perturbing remote and local features respectively [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 7
Figure 7. Figure 7: To enhance multi-GPU parallel inference, each text [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: Qualitative examples by the linearized FLUX-1.dev models with CLEAR and the original model. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Qualitative examples of using CLEAR with SDEdit [ [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Fine-tuning on real data results in inferior performance [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 12
Figure 12. Figure 12: The linearized DiTs by CLEAR are compatible with [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Qualitative comparisons on FLUX-1.dev (top) and SD3.5-Large (bottom). The left subplots are results by the original models [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: More 4K examples by the CLEAR linearized FLUX-1.dev. Prompts are listed in Fig. [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: GPT-generated prompts used in the main manuscript. [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: GPT-generated prompts used in the appendix. [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Fast Video Generation with Sliding Tile Attention

    cs.CV 2025-02 conditional novelty 7.0 of 10

    Sliding tile attention (STA) replaces full 3D attention in video diffusion transformers with dense tile-local windows, achieving 1.89x training-free and up to 3.53x fine-tuned end-to-end speedups on HunyuanVideo with ...

  2. MatchAttention: Embedding Explicit Matching Constraints into Attention for Efficient Stereo Matching

    cs.CV 2025-10 conditional novelty 6.0 of 10

    MatchAttention replaces global cross-attention with a linearly-scaling, learnable-offset windowed attention whose relative position field is the predicted disparity/flow, yielding state-of-the-art stereo/flow accuracy...

  3. LSSGen: Leveraging Latent Space Scaling in Flow and Diffusion for Efficient Text to Image Generation

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A latent-space scaling framework that replaces pixel-space upscaling with a trainable latent upsampler and noise compensation, yielding faster high-resolution text-to-image generation.

  4. Exploring Diffusion Transformer Designs via Grafting

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Grafting uses activation distillation and lightweight fine-tuning to edit pretrained diffusion transformers into hybrid architectures with near-baseline quality at under 2% pretraining compute.

  5. Long-Context State-Space Video World Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A hybrid state-space and local-attention architecture gives autoregressive video diffusion models long-term spatial memory with constant per-frame inference cost, demonstrated on Maze and Minecraft.

  6. Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformers

    cs.CV 2025-05 conditional novelty 6.0 of 10

    GRAT accelerates pretrained diffusion transformers by grouping tokens and restricting each group's attention to neighboring blocks or criss-cross rows and columns, achieving large speedups with near-full-attention quality.

  7. Turbo2K: Towards Ultra-Efficient and High-Quality 2K Video Synthesis

    cs.CV 2025-04 conditional novelty 6.0 of 10

    An efficient text-to-video system produces 2K, 24 fps, 5-second videos with a 4B-parameter model by distilling a 13B teacher and guiding high-resolution generation with low-resolution features.

  8. HRDiT: Training-Free High-Resolution Image Generation with Off-the-Shelf Diffusion Transformer Models

    cs.CV 2026-08 conditional novelty 5.0 of 10

    HRDiT enables off-the-shelf diffusion transformer text-to-image models to produce coherent high-resolution images with far less computation, without any retraining.

  9. UltraFlux: Data-Model Co-Design for High-quality Native 4K Text-to-Image Generation across Diverse Aspect Ratios

    cs.CV 2025-11 conditional novelty 5.0 of 10

    A data-model co-designed Flux DiT with resonance RoPE, post-trained F16 VAE, SNR-aware Huber wavelet loss, and aesthetic curriculum claims state-of-the-art open-source 4K multi-AR text-to-image generation.

  10. Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Sparse-vDiT replaces dense attention with fixed per-head sparse patterns chosen offline, achieving 1.58-1.85x end-to-end speedups on CogVideoX1.5, HunyuanVideo, and Wan2.1 with minimal quality loss.

Reference graph

Works this paper leans on

72 extracted references · 27 canonical work pages · cited by 10 Pith papers

  1. [1]

    All are worth words: A vit backbone for diffusion models

    Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 22669–22679, 2023. 1

    work page 2023

  2. [2]

    Multidiffusion: Fusing diffusion paths for controlled image generation

    Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. In International Conference on Machine Learn- ing, pages 1737–1752. PMLR, 2023. 2, 7

    work page 2023

  3. [3]

    Long- former: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Long- former: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. 2, 3

    arXiv 2004

  4. [4]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the International Conference on Computer Vi- sion (ICCV), 2021. 6

    work page 2021

  5. [5]

    Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 1

    arXiv

  6. [6]

    Pixart- σ: Weak-to-strong training of diffu- sion transformer for 4k text-to-image generation, 2024

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- σ: Weak-to-strong training of diffu- sion transformer for 4k text-to-image generation, 2024. 1, 3, 6, 12, 13

    work page 2024

  7. [7]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. 3, 6, 12

    arXiv 1904

  8. [8]

    Rethinking attention with performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Do- han, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020. 2

    arXiv 2009

Show all 72 references
  1. [9]

    Fast and accurate deep network learn- ing by exponential linear units (elus)

    Djork-Arn ´e Clevert. Fast and accurate deep network learn- ing by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015. 12

    2015 arXiv

  2. [10]

    Diffusion models in vision: A survey

    Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2023. 1

    2023

  3. [11]

    Flashattention-2: Faster attention with bet- ter parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023. 2

    2023 arXiv

  4. [12]

    Transformers are ssms: General- ized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: General- ized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024. 2, 3, 6

    2024 arXiv

  5. [13]

    Flashattention: Fast and memory-efficient exact at- tention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christo- pher R´e. Flashattention: Fast and memory-efficient exact at- tention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022. 2

    2022

  6. [14]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informa- tion processing systems, 34:8780–8794, 2021. 1

    2021

  7. [15]

    Demofusion: Democratising high- resolution image generation with no $$$

    Ruoyi Du, Dongliang Chang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. Demofusion: Democratising high- resolution image generation with no $$$. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6159–6168, 2024. 2, 7

    2024

  8. [16]

    I-max: Maximize the resolu- tion potential of pre-trained rectified flow transformers with projected flow

    Ruoyi Du, Dongyang Liu, Le Zhuo, Qin Qi, Hongsheng Li, Zhanyu Ma, and Peng Gao. I-max: Maximize the resolu- tion potential of pre-trained rectified flow transformers with projected flow. arXiv preprint arXiv:2410.07536, 2024. 7, 13

    2024 arXiv

  9. [17]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  10. [18]

    Lumina-t2x: Transforming text into any modality, reso- lution, and duration via flow-based large diffusion transform- ers

    Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Transforming text into any modality, reso- lution, and duration via flow-based large diffusion transform- ers. arXiv preprint arXiv:2405.05945, 2024. 1

    2024 arXiv

  11. [19]

    Demystify mamba in vision: A linear attention perspective

    Dongchen Han, Ziyi Wang, Zhuofan Xia, Yizeng Han, Yi- fan Pu, Chunjiang Ge, Jun Song, Shiji Song, Bo Zheng, and Gao Huang. Demystify mamba in vision: A linear attention perspective. arXiv preprint arXiv:2405.16605, 2024. 12

    2024 arXiv

  12. [20]

    Agent attention: On the integration of softmax and linear attention

    Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Siyuan Pan, Pengfei Wan, Shiji Song, and Gao Huang. Agent attention: On the integration of softmax and linear attention. In European Conference on Computer Vision , pages 124–

  13. [21]

    Neighborhood attention transformer

    Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 3

    2023

  14. [22]

    Scalecrafter: Tuning-free higher- resolution visual generation with diffusion models

    Yingqing He, Shaoshu Yang, Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang, Ran He, Qifeng Chen, and Ying Shan. Scalecrafter: Tuning-free higher- resolution visual generation with diffusion models. In The Twelfth International Conference on Learning Representa-...

    2024

  15. [23]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 6

    2017

  16. [24]

    Distilling the knowledge in a neural net- work

    Geoffrey Hinton. Distilling the knowledge in a neural net- work. arXiv preprint arXiv:1503.02531, 2015. 2

    2015 arXiv

  17. [25]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1 9

    2020

  18. [26]

    Image quality metrics: Psnr vs

    Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pages 2366–2369. IEEE, 2010. 6

    2010

  19. [27]

    Trans- former quality in linear time

    Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Trans- former quality in linear time. In International conference on machine learning, pages 9099–9117. PMLR, 2022. 2, 3

    2022

  20. [28]

    Fouriscale: A frequency perspective on training-free high-resolution image synthesis

    Linjiang Huang, Rongyao Fang, Aiping Zhang, Guanglu Song, Si Liu, Yu Liu, and Hongsheng Li. Fouriscale: A frequency perspective on training-free high-resolution image synthesis. arXiv preprint arXiv:2403.12963, 2024. 2

    2024 arXiv

  21. [29]

    Training- free diffusion model adaptation for variable-sized text-to- image synthesis

    Zhiyu Jin, Xuli Shen, Bin Li, and Xiangyang Xue. Training- free diffusion model adaptation for variable-sized text-to- image synthesis. Advances in Neural Information Processing Systems, 36:70847–70860, 2023. 7

    2023

  22. [30]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International confer- ence on machine learning, pages 5156–5165. PMLR, 2020. 2, 3, 6, 12

    2020

  23. [31]

    Bk-sdm: Architecturally compressed stable diffusion for efficient text-to-image generation

    Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi. Bk-sdm: Architecturally compressed stable diffusion for efficient text-to-image generation. In Workshop on Efficient Systems for Foundation Models@ ICML2023 ,

  24. [32]

    Multi-concept customization of text-to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1931–1941, 2023. 6

    1931

  25. [33]

    Flux: Official inference repository for flux.1 models, 2024

    Black Forest Labs. Flux: Official inference repository for flux.1 models, 2024. Accessed: 2024-11-12. 2, 7

    2024

  26. [34]

    Distri- fusion: Distributed parallel inference for high-resolution dif- fusion models

    Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Kai Li, and Song Han. Distri- fusion: Distributed parallel inference for high-resolution dif- fusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2024

  27. [35]

    Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding. arXiv preprint arXiv:2405.087...

    2024 arXiv

  28. [36]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  29. [37]

    Flow matching for generative mod- eling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling. arXiv preprint arXiv:2210.02747, 2022. 5

    2022 arXiv

  30. [38]

    Linfusion: 1 gpu, 1 minute, 16k image.arXiv preprint arXiv:2409.02097, 2024

    Songhua Liu, Weihao Yu, Zhenxiong Tan, and Xinchao Wang. Linfusion: 1 gpu, 1 minute, 16k image.arXiv preprint arXiv:2409.02097, 2024. 2, 3, 4, 6, 7, 12

    2024 arXiv

  31. [39]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 3, 6, 12

    2021

  32. [40]

    Sdedit: Guided image synthesis and editing with stochastic differential equa- tions

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jia- jun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equa- tions. arXiv preprint arXiv:2108.01073, 2021. 7, 8, 13

    2021 arXiv

  33. [41]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162–8171. PMLR,

  34. [42]

    Pytorch: An im- perative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Ad- vances in neural information processing systems ...

    2019

  35. [43]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  36. [44]

    Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation,

    Bowen Peng and Jeffrey Quesnelle. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation,

  37. [45]

    Abc: Attention with bounded-memory control

    Hao Peng, Jungo Kasai, Nikolaos Pappas, Dani Yogatama, Zhaofeng Wu, Lingpeng Kong, Roy Schwartz, and Noah A Smith. Abc: Attention with bounded-memory control. arXiv preprint arXiv:2110.02488, 2021. 12

    2021 arXiv

  38. [46]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  39. [47]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Confer- ence for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. 6

    2020

  40. [48]

    Theory, analysis, and best practices for sigmoid self- attention

    Jason Ramapuram, Federico Danieli, Eeshan Dhekane, Floris Weers, Dan Busbridge, Pierre Ablin, Tatiana Likhomanenko, Jagrit Digani, Zijin Gu, Amitis Shidani, et al. Theory, analysis, and best practices for sigmoid self- attention. arXiv preprint arXiv:2409.04431, 2024. 2, 3, 6

    2024 arXiv

  41. [49]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1

    2022

  42. [50]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  43. [51]

    Efficient content-based sparse attention with rout- 10 ing transformers

    Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with rout- 10 ing transformers. Transactions of the Association for Com- putational Linguistics, 9:53–68, 2021. 2, 3

    2021

  44. [52]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500...

    2023

  45. [53]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016. 6

    2016

  46. [54]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  47. [55]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,

  48. [56]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 2

    2017

  49. [57]

    Diffusers: State-of-the-art diffu- sion models

    Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffu- sion models. https://github.com/huggingface/ diffusers...

    2022

  50. [58]

    Linformer: Self-attention with linear complexity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020. 2, 3

    2006 arXiv

  51. [59]

    Mul- tiscale structural similarity for image quality assessment

    Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Mul- tiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, pages 1398–1402. Ieee, 2003. 6

    2003

  52. [60]

    Elite: Encoding visual con- cepts into textual embeddings for customized text-to-image generation

    Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual con- cepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302.13848, 2023. 6

    2023 arXiv

  53. [61]

    Megafusion: Extend dif- fusion models towards higher-resolution image generation without further tuning

    Haoning Wu, Shaocheng Shen, Qiang Hu, Xiaoyun Zhang, Ya Zhang, and Yanfeng Wang. Megafusion: Extend dif- fusion models towards higher-resolution image generation without further tuning. arXiv preprint arXiv:2408.11001 ,

    arXiv

  54. [62]

    Sana: Ef- ficient high-resolution image synthesis with linear diffusion transformers

    Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Yujun Lin, Zhekai Zhang, Muyang Li, Yao Lu, and Song Han. Sana: Ef- ficient high-resolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410.10629, 2024. 2

    arXiv

  55. [63]

    Nystr¨omformer: A nystr¨om-based algorithm for approximat- ing self-attention

    Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nystr¨omformer: A nystr¨om-based algorithm for approximat- ing self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 14138–14148, 2021. 3

    2021

  56. [64]

    Diffusion models: A comprehensive survey of methods and applications

    Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Run- sheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming- Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4): 1–39, 2023. 1

    2023

  57. [65]

    Gated linear attention trans- formers with hardware-efficient training

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention trans- formers with hardware-efficient training. arXiv preprint arXiv:2312.06635, 2023. 2, 3, 6

    2023 arXiv

  58. [66]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. 2023. 6

    2023

  59. [67]

    When linear attention meets autoregressive decoding: Towards more effective and efficient linearized large lan- guage models

    Haoran You, Yichao Fu, Zheng Wang, Amir Yazdanbakhsh, et al. When linear attention meets autoregressive decoding: Towards more effective and efficient linearized large lan- guage models. arXiv preprint arXiv:2406.07368, 2024. 2, 3

    2024 arXiv

  60. [68]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neu- ral information processing systems, 33:17283–17297, 2020. 2, 3

    2020

  61. [69]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2, 6, 7, 8, 13

    2023

  62. [70]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 6

    2018

  63. [71]

    , // 3 "A tiger is kissing a rabbit

    Yu Zhang, Songlin Yang, Ruijie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, et al. Gated slot attention for efficient linear-time sequence modeling. arXiv preprint arXiv:2409.07146, 2024. 2, 3, 12 11 A. Details of Efficient Attention Alt...

    2024 arXiv

  64. [140]

    2, 3, 6, 12

    Springer, 2025. 2, 3, 6, 12

    2025

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.