PiT: Progressive Diffusion Transformer

Chengjie Wang; Jiafu Wu; Jiangning Zhang; Jian Li; Jinlong Peng; Yabiao Wang; Yun Cao

REVIEW 4 major objections 5 minor 3 references

PiT: Progressive Diffusion Transformer

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that diffusion transformers barely rely on long-range attention, and that a windowed attention plus convolutional bridge architecture (PiT) outperforms full-attention DiT and U-DiT models at the same training budget.

desk verdict The empirical case for replacing global attention with window attention plus a conv bridge in DiTs is credible and consistently shown, but the paper oversells the Kth-order attention story and includes a wrong theorem in the supplement. read the letter →

arxiv 2505.13219 v5 pith:KHXN5KWO submitted 2025-05-19 cs.CV

Jiafu Wu , Yabiao Wang , Jian Li , Jinlong Peng , Yun Cao , Chengjie Wang , Jiangning Zhang This is my paper · ORCID

classification cs.CV

keywords diffusiontransformerwindowattentionimagegenerationPseudoShiftedKth-orderlow-frequencybiaschannelallocation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Diffusion Transformers spend most of their compute on full global self-attention, but this paper argues that most of that global computation is redundant. The authors measure attention maps from a pretrained DiT and find that 99% of tokens attend mainly to neighbors within a few pixels, so they replace global attention with a cheaper static-window attention plus a convolutional high-frequency bridging branch (PSWA). They add a channel-allocation schedule (PCCA) that shifts channels from the convolutional branch to the attention branch as layers deepen, which they present as capturing higher-order similarity at no extra cost. The resulting PiT models beat full-attention DiT and U-DiT baselines at the same training budget; PiT-L reaches FID 9.18 versus 19.47 for DiT-XL/2 while using fewer FLOPs. If the paper is right, window attention plus convolution is not just a cheaper approximation but a better inductive bias for diffusion image generation.

What carries the argument

The load-bearing machinery is Pseudo Shifted Window Attention (PSWA), a channel-split attention block that replaces global multi-head self-attention in diffusion transformers. PSWA runs static window attention on one set of channels and a depthwise separable convolution on another, then concatenates the two branches; the convolution both captures high-frequency information and links adjacent windows without the cost of shifted-window overlaps. The second component, Progressive Coverage Channel Allocation (PCCA), is a per-layer schedule that grows the window-attention channel fraction and shrinks the convolution fraction as depth increases, so tokens that were locally aggregated by large convolution kernels in one layer feed the attention of the next. Because a $5\times5$ convolution covers a third-order neighborhood, attention computed on those transferred channels becomes $K$-th order attention, defined as similarity between neighborhood aggregates rather than between raw tokens. This machinery converts the empirical locality observation into a cheaper architecture, and the supplement's Fourier argument is meant to justify why the convolution branch is needed.

What would settle it

Measure the row and column attention distance of Equation 4 on attention maps from a larger pretrained DiT, or from DiT-B/2 operating at a 32×32 latent resolution; if the 99th-percentile distance approaches the full map size, the locality premise behind PiT fails and window-attention would be expected to lose to global attention on those scales.

Watch

Extended reading notes

Core claim

The paper's central claim is that diffusion transformers do not actually need the global receptive field full self-attention provides. Using row and column attention distances averaged over one million attention maps from a pretrained DiT-B/2 at 16×16 latent resolution, the authors report that 99% of tokens attend within a row/column distance of 2–6, so local interactions dominate. On that basis they assert that static window attention is sufficient, and they design PSWA to split channels between window attention and a depthwise convolution branch; the convolution supplies high-frequency detail and bridges window boundaries. PCCA then progressively increases the attention-branch channel share across layers, so features aggregated over neighborhoods in earlier layers enter later attention computations, yielding what the paper calls $K$-th order attention at no extra cost. The authors also argue, with a Fourier-domain analysis in the supplement, that attention mechanisms carry a low-frequency bias that overlap-based window connections inherit, and that PSWA's parallel convolution branch is a robust way around it. Empirically, PiT-L obtains FID 9.18 at 400K iterations, a 54% improvement over DiT-XL/2's 19.47 with fewer FLOPs.

Load-bearing premise

The entire efficiency argument rests on the measured claim that 99% of DiT attention stays within a 2–6 token distance at 16×16 latent resolution; if that locality is an artifact of one model size and one resolution, the window-attention advantage may not transfer.

Editorial extensions

If this is right

At a fixed 400K-iteration budget on ImageNet at 256×256, PiT-L reaches FID 9.18 versus 19.47 for DiT-XL/2 while using 100.80 G FLOPs instead of 114.46 G.
Simply replacing MHSA with PSWA in standard isotropic DiT models improves FID at every tested size (PiT-S/2, PiT-B/2, PiT-L/2) while lowering FLOPs.
PiT-B also outperforms U-DiT-B at 512×512 resolution under the same training settings, so the gain is not limited to the 256×256 benchmark.
Because the attention is windowed, throughput rises sharply: PiT-B/2 processes about 618 images per second versus 272 for DiT-B/2 on the same GPUs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's experiments, the locality measurement comes from a single pretrained DiT-B/2 at a 16×16 latent resolution; if larger models or higher latent resolutions develop longer-range dependencies, the window size and PCCA schedule would need to scale and the efficiency gap could close.
The DWConv bridge is effectively a cheap convolutional inductive bias, so the same channel-split design could be tried in text-to-image or video diffusion transformers; the paper does not test those settings.
The supplementary low-frequency theorem is only a sketch: the proof drops the nonlinear Softmax term and the theorem name appears unfinished, so the theoretical case for the DWConv branch is weaker than the empirical evidence.
The reported 54% improvement suggests training compute, not model capacity, was the bottleneck for the DiT-XL/2 baseline; a direct test would be to give DiT-XL/2 the same FLOPs-reduced training recipe and see whether the gap persists.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

I read PiT with interest. The headline result is solid and useful: swapping MHSA for a static window attention plus a depthwise-conv bridge gives a consistent FID improvement over both DiT and U-DiT across model sizes and training budgets. The controlled replacement in Table 4 is a clean experiment, and the step-by-step training curves in Table 3 add confidence that the gains are not a fluke. The attention-distance measurement on a pretrained DiT is a nice empirical observation, and the throughput numbers in Table 6 are concrete.

What is genuinely new is the PCCA schedule: to my knowledge, no one else has proposed progressively shifting channels from a conv branch to a window-attention branch as a way to increase receptive-field coverage in deeper layers. The PSWA block itself is a recombination of known parts, but the combination is effective and the ablations in Tables 7 and 8 support the design choices.

The soft spots are all in the interpretation and the supplementary material. The abstract and Section 4 claim that PCCA captures Kth-order attention 'without any additional computational cost.' But Table 7 reports only FID, not FLOPs, for the five allocation schedules. Window attention and 5x5 depthwise conv have different per-channel costs; shifting channels from conv to attention changes the layer's compute. If attention is the more expensive branch, the best schedule (Cwin increasing) may just be a favorable compute reallocation, not proof of a Kth-order mechanism. The paper needs to report FLOPs per schedule, or directly implement the Kth-order aggregation and compare. This does not kill the empirical result, but it undercuts the semantic story.

More serious is the supplementary theorem. The proof claims that (1-C)^t does not decay to zero when C<1, which is plainly false for 00 collapses, so the theorem as written is not proved. It should be corrected or removed. The theorem is not load-bearing for the main experiments, but leaving a false equation in the supplement is a credibility problem.

Minor notes: no code or error bars, and the U-DiT comparisons may use published numbers rather than reruns under identical settings. The DiT* reruns in Table 4 show some control, but the direct U-DiT apples-to-apples comparison is unclear.

Overall, the paper is for researchers working on efficient diffusion backbones. The empirical claim is likely true and worth engaging with. I would send it to a serious venue, but with clear instructions to fix the FLOPs confound in the PCCA ablation and correct or delete the theorem. If those are addressed, it is a publishable efficiency contribution.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes PiT, a family of diffusion transformers that replace global multi-head attention with static window attention (PSWA) plus a depthwise-convolution high-frequency bridging branch, and introduce PCCA, a channel-allocation schedule that progressively shifts channels from the convolution branch to the window-attention branch across layers. The authors claim PCCA realizes 'Kth-order attention' at no extra cost. They report ImageNet 256×256 FID improvements over DiT and U-DiT at lower or comparable FLOPs for S/B/L sizes, a 512×512 comparison, training-curve tables, MHSA-replacement experiments, channel-allocation and K-order ablations, and a supplementary theorem on the low-frequency bias of attention.

Significance. If the empirical FID results are reproducible, the paper would strengthen the case that local window attention plus a cheap convolutional bridge can outperform full global attention in diffusion transformers under equal training budgets, which is of practical significance for scaling image generation. The paper's strengths include the systematic size sweep (S/B/L), the iteration-by-iteration comparison in Table 3, the direct MHSA replacement in Table 4, and the channel-allocation/K ablations. However, the interpretive layer around PCCA and Kth-order attention is not yet supported by the reported equations and ablations, and the supplementary theorem's proof is invalid. The central empirical claim is plausible but needs the confounds below addressed.

major comments (4)

[§PCCA and Table 7] The statement in §PCCA that PCCA captures Kth-order attention 'without any additional computational cost' is not substantiated and is likely false as stated. A channel reassigned from the 5×5 DWConv branch (25 multiply-adds per token per channel for the convolution itself) to the window-attention branch incurs the attention branch's per-channel interaction cost, which for typical window sizes is larger than 25. Table 7 compares five allocation schedules but reports only FID, never FLOPs, so row 5's best FID is confounded with a shift of compute toward the attention branch in deeper layers. Please report per-schedule FLOPs (and ideally per-layer channel budgets) and either verify the no-cost claim or rewrite it as a limited-cost claim.
[§PCCA, Eq. (11)] Equation (11) is not a derivation of the implemented computation. The channels in the red dashed box were convolved by 5×5 DWConv at layer k, but at layer k+1 they pass through LayerNorm, QKV projections, and window attention before any similarity is computed; the expression S^3_ij = Σ_c Conv5(X_i^c)·Conv5(X_j^c) omits these operations and therefore does not show that the network literally computes third-order similarity. The Kth-order interpretation should be presented as an analogy or supported by a derived identity that accounts for the actual operations.
[Supplementary Theorem 1] The proof is invalid. In Eqs. (22)–(23), the paper derives E_L(t) ≥ (1-C)^t E_L(0) and, for C<1, concludes this remains bounded below by c>0; but if 0<C<1, (1-C)^t decays to zero, so c=0. The assumptions C<1 in Eq. (21) and the bound in Eq. (24) are also introduced without proof. The theorem therefore does not establish the claimed low-frequency propagation. This does not affect the empirical comparisons, but the supplementary claim should be corrected or removed.
[§Revisiting, Figure 1] The locality analysis uses attention maps from one pre-trained DiT-B/2 at a single 16×16 latent resolution. The paper generalizes this to 'DiTs' as a class and uses it to motivate replacing global attention at all scales. Since the transfer of the locality conclusion to larger latent resolutions (e.g., 32×32 in the 512×512 setting), larger models, and earlier training stages is not shown, the claim should be explicitly scoped or supported by additional measurements; the current sentence 'DiT does not rely on long-distance attention modeling' is stronger than the evidence.

minor comments (5)

[Table 3 text] The sentence 'PiT-L with a smaller model size surpasses the metrics of DiT-XL/2 at 7M iterations when it reaches 1M iterations' should be rewritten; the intended comparison is PiT-L at 1M steps versus DiT-XL/2 at 7M steps.
[Abstract and Table 2] The abstract and text report '54% FID improvement' and '10.68' absolute FID gain; from Table 2 these values depend on whether the baseline is DiT-XL/2 (19.47) or DiT-XL/2* (19.86). Please state the baseline explicitly.
[Figure 1 caption] Figure 1 and the text differ on whether the empirical sample is 'one million samples' or 'one million attention maps'; please reconcile and specify the number of images, tokens, heads, and layers.
[Supplementary section title] The supplementary section title contains an unfinished placeholder ('Propagation of Low-Frequency Theorem??'); please complete it.
[Tables 2 and 3] The main FID tables do not report error bars or multiple seeds; given the small gaps in Table 2 (PiT-B vs U-DiT-B: 15.56 vs 16.64), a note on run-to-run variance would be helpful.

Circularity Check

1 steps flagged · score 2.0 of 10

PCCA's 'Kth-order attention' is defined by the same equation it is said to capture, but the main FID benchmarks are external and self-contained.

self definitional [Method, PCCA section, Equations 8-11]
"The Kth-order similarity between positions i and j is then defined as: S_K(i,j) ≜ ⟨Φ(N_K(i)), Φ(N_K(j))⟩ ... When these channels are passed to the static window attention branch in layer k+1 to naturally compute first-order similarity, the result is no longer a simple first-order similarity, but rather third-order similarity. The mathematical expression is as follows: S3_ij = Σ_c Conv5(X_i^c)·Conv5(X_j^c)."

Equation 11 is exactly Equation 9 instantiated with Φ = Conv5: both are inner products of convolutionally aggregated neighborhood features. Therefore, PCCA's 'capture of Kth-order attention' is guaranteed by construction rather than derived from independent principles. The paper then uses this self-defined concept to explain why the Cwin-increasing channel schedule performs best in Table 7. The FID values themselves come from external training comparisons with DiT and U-DiT and are not fitted to this definition, so the circularity is confined to the interpretive/mechanistic layer.

full rationale

The load-bearing empirical claim—PiT-L reaching 9.18 FID versus 19.47 for DiT-XL/2 and 10.08 for U-DiT-L at 400K iterations—is measured against external baselines under shared training settings, so the benchmark itself is self-contained. No parameter is fitted to the reported FID and then renamed a prediction. The paper's self-citations (EMO/EMOv2 etc.) are background literature and are not load-bearing for the main results. The supplementary 'Propagation of Low-Frequency' theorem relies on unproved spectral assumptions (e.g., Equations 18 and 24), and the 'without additional computational cost' PCCA claim ignores unequal per-channel FLOPs between DWConv and window attention; these are correctness or accounting risks, not circularity. The only identifiable circularity is definitional: PCCA's Kth-order similarity is defined as precisely the operation PCCA performs, so the claim that PCCA 'captures' Kth-order attention is a relabeling of the mechanism. Because this loop does not determine the external FID comparisons, the circularity score is 2.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central empirical result (FID comparisons) depends on a handful of design choices (DWConv kernel size, channel schedule, window size) that are selected via ablations rather than derived. The theoretical interpretation additionally rests on an unproved Fourier-dynamics theorem and on an informal identification between conv-transformed channels and Kth-order similarity. These choices are not fitted to the benchmark numbers in a hidden way, but they do leave the method underspecified for reproduction.

free parameters (3)

DWConv kernel size K = 5 (ablated; K=2 also optimal in Table 8)
The kernel size for the high-frequency bridging branch is ablated on FID (Table 8) and K=5 chosen for main results; this selects the receptive field that defines the claimed Kth-order attention.
Channel allocation schedule C_attn^k = not reported numerically
The progressive proportion of static window attention channels over layers is a hand-designed schedule that is varied in Table 7; the best schedule (C_win increasing) is chosen based on FID, and the exact values are not given in the paper.
Window size and stride = not reported
The static window attention requires a window size and stride; the paper specifies no window size, padding, or number of windows, making the architecture underspecified.

assumptions (3)

domain assumption Attention distance measured on pre-trained DiT-B/2 at 16x16 latent map is representative of DiT behavior across scales and training budgets
Used in Section 2 'Exploring the Intrinsic Dependence Distance' to justify replacing global attention with window attention; if this locality does not hold at larger scales, the main efficiency claim is weakened.
ad hoc to paper The Fourier dynamics equation (18) with damping e^{-gamma(rho)||omega||^2} and the bound C<1 in Eq. (21) hold for Softmax attention
Supplementary Theorem 1 relies on these unproved operator bounds and an undefined gamma(rho); the theorem is used to claim all window attention suffers low-frequency bias.
ad hoc to paper A channel that has been processed by a 5x5 DWConv at layer k, when passed to window attention at layer k+1, computes third-order similarity as in Eq. (11)
This is the load-bearing interpretation of PCCA as Kth-order attention; the equation only sums per-channel conv outputs and does not correspond to the Kth-order attention defined in Eq. (10).

invented entities (1)

Kth-order attention
purpose: To reinterpret the channel-transfer schedule (PCCA) as a higher-order attention mechanism and justify the design
No module computes S_K directly; the paper's Eq. (11) is an approximation and the actual attention remains first-order similarity on conv-transformed features, so there is no falsifiable handle beyond the paper's own narrative.

how reviews work

0 comments

Cite this review

Pith. "Pith review of PiT: Progressive Diffusion Transformer." pith.science (2026). https://pith.science/paper/KHXN5KWO

@misc{pith2026250513219,
  author       = {Pith},
  title        = {Pith review of: PiT: Progressive Diffusion Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KHXN5KWO}},
  note         = {Machine review of arXiv:2505.13219}
}

read the original abstract

Diffusion Transformers (DiTs) achieve remarkable performance within image generation via the transformer architecture. Conventionally, DiTs are constructed by stacking serial isotropic global modeling transformers, which face significant quadratic computational cost. However, through empirical analysis, we find that DiTs do not rely as heavily on global information as previously believed. In fact, most layers exhibit significant redundancy in global computation. Additionally, conventional attention mechanisms suffer from low-frequency inertia, limiting their efficiency. To address these issues, we propose Pseudo Shifted Window Attention (PSWA), which fundamentally mitigates global attention redundancy. PSWA achieves moderate global-local information through window attention. It further utilizes a high-frequency bridging branch to simulate shifted window operations, which both enrich the high-frequency information and strengthen inter-window connections. Furthermore, we propose the Progressive Coverage Channel Allocation (PCCA) strategy that captures high-order attention without additional computational cost. Based on these innovations, we propose a series of Pseudo Progressive Diffusion Transformer (PiT). Our extensive experiments show their superior performance; for example, our proposed PiT-L achieves 54% FID improvement over DiT-XL/2 while using less computation.

Figures

Figures reproduced from arXiv: 2505.13219 by the authors.

**Figure 3.** The Fourier transform frequency maps from DiT [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗

**Figure 4.** The overall framework of PiT. The static window attention and DWConv are the core operations of the PSWA [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗

Figures from the paper (3 more)

**Figure 5.** Figure 5: The first-order similarity measure tends to prioritize low-level features such as color and texture. As illustrated by the [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: The comparison of image generation results [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: As training iterations increase, the generated im [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 2 linked inside Pith

[2]

InEuropean Conference on Computer Vision, 37–55

Diffit: Diffusion vision transformers for image gener- ation. InEuropean Conference on Computer Vision, 37–55. Springer. He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep resid- ual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recogni- tion, 770–778. Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising d...

arXiv 2016
[3]

Touvron, H.; Cord, M.; Douze, M.; Massa, F.; Sablayrolles, A.; and J´egou, H

U-dits: Downsample tokens in u-shaped diffusion transformers.arXiv preprint arXiv:2405.02730. Touvron, H.; Cord, M.; Douze, M.; Massa, F.; Sablayrolles, A.; and J´egou, H. 2021. Training data-efficient image trans- formers & distillation through attention. InInternational conference on machine learning, 10347–10357. PMLR. Vaswani, A.; Shazeer, N.; Parmar,...

arXiv 2021
[2024]

InForty-first international conference on machine learning

Scaling rectified flow transformers for high-resolution image synthesis. InForty-first international conference on machine learning. Gao, S.; Zhou, P.; Cheng, M.-M.; and Yan, S. 2023a. Masked diffusion transformer is a strong image synthesizer. InProceedings of the IEEE/CVF international conference on computer vision, 23164–23173. Gao, S.; Zhou, P.; Cheng...

arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [2]

InEuropean Conference on Computer Vision, 37–55

Diffit: Diffusion vision transformers for image gener- ation. InEuropean Conference on Computer Vision, 37–55. Springer. He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep resid- ual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recogni- tion, 770–778. Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising d...

arXiv 2016

[2] [3]

Touvron, H.; Cord, M.; Douze, M.; Massa, F.; Sablayrolles, A.; and J´egou, H

U-dits: Downsample tokens in u-shaped diffusion transformers.arXiv preprint arXiv:2405.02730. Touvron, H.; Cord, M.; Douze, M.; Massa, F.; Sablayrolles, A.; and J´egou, H. 2021. Training data-efficient image trans- formers & distillation through attention. InInternational conference on machine learning, 10347–10357. PMLR. Vaswani, A.; Shazeer, N.; Parmar,...

arXiv 2021

[3] [2024]

InForty-first international conference on machine learning

Scaling rectified flow transformers for high-resolution image synthesis. InForty-first international conference on machine learning. Gao, S.; Zhou, P.; Cheng, M.-M.; and Yan, S. 2023a. Masked diffusion transformer is a strong image synthesizer. InProceedings of the IEEE/CVF international conference on computer vision, 23164–23173. Gao, S.; Zhou, P.; Cheng...

arXiv