Pith. sign in

REVIEW 11 cited by

U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.02730 v3 pith:G5EM5IWW submitted 2024-05-04 cs.CV

Yuchuan Tian , Zhijun Tu , Hanting Chen , Jie Hu , Chao Xu , Yunhe Wang This is my paper · ORCID
classification cs.CV
keywords u-netditsarchitecturediffusionu-ditcomputationconductdemonstrate
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Diffusion Transformers (DiTs) introduce the transformer architecture to diffusion tasks for latent-space image generation. With an isotropic architecture that chains a series of transformer blocks, DiTs demonstrate competitive performance and good scalability; but meanwhile, the abandonment of U-Net by DiTs and their following improvements is worth rethinking. To this end, we conduct a simple toy experiment by comparing a U-Net architectured DiT with an isotropic one. It turns out that the U-Net architecture only gain a slight advantage amid the U-Net inductive bias, indicating potential redundancies within the U-Net-style DiT. Inspired by the discovery that U-Net backbone features are low-frequency-dominated, we perform token downsampling on the query-key-value tuple for self-attention that bring further improvements despite a considerable amount of reduction in computation. Based on self-attention with downsampled tokens, we propose a series of U-shaped DiTs (U-DiTs) in the paper and conduct extensive experiments to demonstrate the extraordinary performance of U-DiT models. The proposed U-DiT could outperform DiT-XL/2 with only 1/6 of its computation cost. Codes are available at https://github.com/YuchuanTian/U-DiT.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UniMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation

    cs.CV 2025-07 conditional novelty 7.0 of 10

    UniMC uses tokenized instance conditions (class, box, keypoints) and a timestep-aware modulator in a DiT backbone to control multi-class human and animal image generation, trained and evaluated on the new HAIG-2.9M dataset.

  2. TinyFusion: Diffusion Transformers Learned Shallow

    cs.CV 2024-12 conditional novelty 7.0 of 10

    A learnable depth-pruning method that optimizes post-fine-tuning recoverability produces a 14-layer DiT-XL with FID 2.86 and a 2x speedup at 7% of the original training cost.

  3. Unveiling the Secret of AdaLN-Zero in Diffusion Transformer

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Zero-initializing the modulation weights is the dominant reason adaLN-Zero outperforms adaLN, and replacing it with a Gaussian initialization of std 0.001 improves FID at the same training steps.

  4. Amortized Moment Matching for Visual Generation

    cs.LG 2026-07 accept novelty 6.0 of 10

    Amortized Fréchet Distance uses neural nets to match conditional means and covariances, yielding stronger one-step visual generators than explicit FD-loss or multi-step teachers.

  5. DiffSparse: Accelerating Diffusion Transformers with Learned Token Sparsity

    cs.CV 2026-04 conditional novelty 6.0 of 10

    A learnable cost predictor plus dynamic programming allocates layer-wise token sparsity for diffusion transformers, removing forced full steps and cutting ~54% compute on PixArt-α without quality loss.

  6. PiT: Progressive Diffusion Transformer

    cs.CV 2025-05 conditional novelty 6.0 of 10

    PiT replaces global attention in diffusion transformers with window attention plus a depthwise-conv bridging branch and a progressive channel schedule, achieving better FID than DiT and U-DiT at lower or similar compute.

  7. Ingredients: Blending Custom Photos with Video Diffusion Transformers

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Ingredients adds a mask-supervised identity router to a video diffusion transformer, enabling multi-person videos from a few reference photos without per-identity fine-tuning.

  8. Identity-Preserving Text-to-Video Generation by Frequency Decomposition

    cs.CV 2024-11 conditional novelty 6.0 of 10

    ConsisID generates identity-preserving videos by injecting low-frequency facial features into shallow layers and high-frequency identity features into attention blocks of a DiT video model.

  9. Align Your Flow: Scaling Continuous-Time Flow Map Distillation

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Align Your Flow distills flow maps with new continuous-time objectives and autoguidance, achieving state-of-the-art few-step FID on ImageNet and strong text-to-image results.

  10. RainFusion: Adaptive Video Generation Acceleration via Multi-Dimensional Visual Redundancy

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Training-free sparse attention that classifies each head as spatial, temporal, or textural and applies a matched mask or token reduction, giving about 1.9x attention speedup with small VBench losses.

  11. LaVin-DiT: Large Vision Diffusion Transformer

    cs.CV 2024-11 conditional novelty 5.0 of 10

    A single diffusion transformer with a spatial-temporal VAE and in-context example pairs unifies over 20 vision tasks, with strong scores on several image benchmarks and uneven evidence on video tasks.

Pith tools