Pith. sign in

REVIEW 2 major objections 2 minor

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

Register tokens improve convergence and generation quality of pixel-space DiTs by producing cleaner feature maps at high noise levels.

2026-05-20 19:03 UTC pith:674KEV4R

load-bearing objection Registers improve pixel-space DiTs without the usual ViT outliers, but the link from cleaner high-noise features to gains stays observational. the 2 major comments →

arxiv 2605.16147 v2 pith:674KEV4R submitted 2026-05-15 cs.CV

Registers Matter for Pixel-Space Diffusion Transformers

Nikita Starodubcev , Ilia Sudakov , Ilya Drobyshevskiy , Artem Babenko , Dmitry Baranchuk This is my paper
classification cs.CV
keywords register tokensdiffusion transformerspixel-space generationfeature mapshigh noise levelsdual-stream architecturevision transformers
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines whether register tokens, which mitigate issues in vision transformers, also help diffusion transformers trained directly in pixel space. Although pixel-space DiTs lack the high-norm patch-token outliers common in ViTs, adding register tokens still leads to faster training convergence and better final image quality. Analysis of intermediate layers shows that register tokens keep feature maps cleaner especially during the high-noise phases of the diffusion process. This observation also accounts for the strong results of recent DiT designs that already embed similar mechanisms implicitly. The work introduces a parameter-efficient dual-stream architecture that processes register tokens separately to gain further quality with almost no added runtime cost.

Core claim

DiTs operating in pixel space do not display the patch-token outliers that plague ViTs, yet register tokens still deliver clear gains in convergence speed and generation quality. Representation analysis links these gains to cleaner feature maps at high noise levels. Recent strong pixel-space DiT models already contain implicit register-like behaviors. A dual-stream architecture that dedicates separate processing streams to register tokens yields additional quality improvements while adding negligible overhead.

What carries the argument

register tokens, which produce cleaner feature maps at high noise levels without participating in the main patch-token processing

Load-bearing premise

The performance gains come specifically from cleaner feature maps at high noise levels rather than from changes in optimization dynamics or other unmeasured factors.

What would settle it

An experiment that measures feature-map cleanliness at high noise levels and finds no correlation with the observed quality gains from register tokens would disprove the proposed mechanism.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Faster convergence during training of pixel-space DiTs
  • Higher quality in final generated images
  • Cleaner intermediate representations during high-noise diffusion steps
  • Implicit register-like mechanisms explain the success of recent DiT architectures
  • A dual-stream design achieves further gains with almost no extra compute

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same register mechanism may help other transformer-based generative models beyond diffusion
  • Specialized token streams could become a standard design choice in future diffusion transformers
  • The benefit might be tested at different noise schedules or resolutions to isolate the effect
  • Register tokens could reduce the need for heavy regularization techniques in pixel-space training

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript examines the role of register tokens in pixel-space Diffusion Transformers (DiTs). Unlike Vision Transformers, the authors find that DiTs lack high-norm patch-token outliers. Nevertheless, they report that adding register tokens improves convergence speed and generation quality. Intermediate representation analysis indicates that register tokens yield cleaner feature maps at high noise levels, which the authors suggest may explain the gains. They further note that recent pixel-space DiT designs appear to incorporate implicit register-like mechanisms and introduce a parameter-efficient dual-stream architecture that dedicates separate processing streams to register tokens, achieving quality improvements with negligible runtime cost.

Significance. If the empirical gains and mechanistic interpretation hold under rigorous controls, the work would offer actionable guidance for designing pixel-space diffusion transformers and clarify why register tokens remain useful even without the outlier problem that motivated them in ViTs. The dual-stream proposal is practically attractive because of its low overhead. The observation that state-of-the-art pixel-space models already embed register-like behavior is a useful retrospective insight. These contributions could influence architectural choices in future diffusion models, provided the causal attribution is strengthened.

major comments (2)
  1. [Representation analysis section] The central attribution in the representation analysis—that cleaner feature maps at high noise levels are responsible for the observed convergence and quality gains—is purely observational. No intervention, ablation, or controlled experiment isolates this mechanism from alternative explanations such as changes in optimization dynamics, gradient flow, or implicit regularization induced by the extra tokens. Because the motivating ViT outlier-suppression benefit is explicitly absent, this untested causal premise is load-bearing for the headline claim.
  2. [Experimental results] The experimental results reporting improved convergence and generation quality do not include statistical significance tests across multiple random seeds, detailed baseline comparisons that hold all other hyperparameters fixed, or ablations on register-token count and placement. Without these controls it is difficult to quantify the reliability and magnitude of the claimed benefits.
minor comments (2)
  1. The abstract and introduction would benefit from explicit statements of the datasets, metrics (e.g., FID, precision/recall), and training budgets used to measure generation quality.
  2. [Dual-stream architecture description] Notation for the dual-stream architecture (e.g., how the two streams interact at each layer) should be defined more formally, perhaps with a diagram or pseudocode, to aid reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We address each major comment below and describe the revisions we will make to strengthen the causal interpretation and experimental rigor.

read point-by-point responses

  1. Referee: [Representation analysis section] The central attribution in the representation analysis—that cleaner feature maps at high noise levels are responsible for the observed convergence and quality gains—is purely observational. No intervention, ablation, or controlled experiment isolates this mechanism from alternative explanations such as changes in optimization dynamics, gradient flow, or implicit regularization induced by the extra tokens. Because the motivating ViT outlier-suppression benefit is explicitly absent, this untested causal premise is load-bearing for the headline claim.

    Authors: We agree that the current analysis is observational and does not include interventions that would isolate the proposed mechanism from alternatives such as optimization dynamics or regularization effects. In the revision we will add controlled ablations that compare feature-map statistics and performance when register tokens are present versus absent while monitoring gradient norms and loss landscapes. We will also revise the language in the representation section to more clearly frame the cleaner feature maps as a consistent correlate rather than a proven causal driver, while retaining the empirical observation that register tokens improve results even in the absence of patch-token outliers. revision: yes

  2. Referee: [Experimental results] The experimental results reporting improved convergence and generation quality do not include statistical significance tests across multiple random seeds, detailed baseline comparisons that hold all other hyperparameters fixed, or ablations on register-token count and placement. Without these controls it is difficult to quantify the reliability and magnitude of the claimed benefits.

    Authors: We acknowledge these gaps in statistical rigor and controls. In the revised manuscript we will rerun the main experiments with at least three random seeds, reporting means and standard deviations for convergence curves and FID scores. We will also present baseline comparisons in which all other hyperparameters remain fixed and add ablations that vary both the number of register tokens and their placement within the transformer blocks. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical observations remain independent of fitted inputs

full rationale

The paper is an empirical study demonstrating that register tokens improve convergence and generation quality in pixel-space DiTs despite the absence of ViT-style patch-token outliers. It supports this via direct experiments and observational representation analysis showing cleaner feature maps at high noise levels. No equations, derivations, or first-principles results are presented that reduce the reported performance gains to quantities defined, fitted, or predicted from within the same experiment. Claims rest on external benchmarks and measurements rather than self-referential definitions or self-citation chains that would force the outcome by construction. Any citations to prior register-token work are standard background and not load-bearing for the DiT-specific findings.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper rests on standard deep-learning assumptions about the validity of FID or similar metrics, the representativeness of the training distribution, and that feature-map statistics at selected noise levels are causally linked to final generation quality; no explicit free parameters or invented entities are described in the abstract.

axioms (1)
  • domain assumption Standard assumptions about optimization dynamics and evaluation metrics in diffusion model training hold for the tested architectures.
    Invoked implicitly when attributing performance gains to register tokens and cleaner feature maps.

pith-pipeline@v0.9.0 · 5717 in / 1283 out tokens · 51401 ms · 2026-05-20T19:03:01.995560+00:00 · methodology

0 comments
read the original abstract

Vision Transformers (ViTs) are known to exhibit high-norm patch-token outliers that degrade feature map quality, a problem effectively mitigated by register tokens. As diffusion models increasingly adopt transformer architectures and move toward pixel-space training, they become closer in form to ViTs, raising the question of whether register tokens are also useful for Diffusion Transformers (DiTs). In this work, we show that DiTs differ from ViTs in a key respect: they do not exhibit patch-token outliers but still benefit from registers. Interestingly, registers are more effective in pixel-space DiTs than in latent-space DiTs. By analyzing intermediate representations, we find that register tokens produce cleaner feature maps at high noise levels, which may contribute to their effectiveness in pixel-space generation. We further observe that recent pixel-space DiT architectures implicitly incorporate register-like mechanisms, which may partially account for their strong empirical performance. Motivated by these observations, we propose Register Guidance, a technique that amplifies the contribution of register tokens responsible for improving visual structure and coherence.

Figures

Figures reproduced from arXiv: 2605.16147 by Artem Babenko, Dmitry Baranchuk, Ilia Sudakov, Ilya Drobyshevskiy, Nikita Starodubcev.

Figure 1
Figure 1. Figure 1: Diffusion transformers do not exhibit attention-map outliers. Unlike ViTs, where attention-map anomalies typically appear in low-information regions (e.g., background), DiT attention remains focused on the main objects. Contributions. We find that, unlike ViTs, diffusion transformers in both latent and pixel spaces do not exhibit noticeable high-norm outliers among patch tokens. Instead, patch-token norms … view at source ↗
Figure 2
Figure 2. Figure 2: Without Registers. (a) In DINOv2, anomalies are localized to few image tokens, which exhibit significantly higher norms than others. (b) In contrast, no outliers are observed for pDiTs, suggesting that registers may be unnecessary in this case. (a) (b) 0 50 100 150 200 250 300 token index 0 1 2 3 4 5 6 Token norm ×104 pDiT-B/16, With Registers block 5 block 10 0 50 100 150 200 250 token index 0.0 0.6 1.2 1… view at source ↗
Figure 3
Figure 3. Figure 3: With Registers. (a) As expected, introducing register tokens in DINOv2 shifts high-norm outliers into these tokens. (b) Interestingly, pDiTs also exhibit high-norm tokens in the added registers, even though such outliers are absent without registers. pDiT-B/16, 131M pDiT-L/16, 459M pDiT-H/16, 953M Epoch w/o reg. w/ reg. w/ in-context w/o reg. w/ reg. w/ in-context w/o reg. w/ reg. w/ in-context 200 7.39 5.… view at source ↗
Figure 4
Figure 4. Figure 4: Register tokens consistently reduce feature norms across patch tokens. We measure feature norms for image tokens only (excluding registers) at three diffusion timesteps and observe a consistent reduction across all tokens when registers are used. Original image pDiT-B/16 with registers pDiT-B/16 without registers (a) (b) 0 1 2 3 4 5 6 7 8 9 10 avg Block 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Timestep … view at source ↗
Figure 5
Figure 5. Figure 5: Register tokens improve intermediate representations. (a) We compute the Total Variation (TV) of intermediate features for models with and without register tokens. We report the ratio (with / without registers), where lower values indicate that the model with registers produces smoother features. We find that registers improve feature smoothness at high noise levels (t ∈ [0, 0.2]). (b) We visualize feature… view at source ↗
Figure 6
Figure 6. Figure 6: Register tokens act as both global information carriers and norm sinks. Linear probing reveals that low-norm register tokens encode meaningful global semantics and achieve strong classification accuracy, whereas low-accuracy registers exhibit extremely large norms, suggesting that they primarily function as norm sinks that absorb magnitude from patch tokens. Input image Attention maps for register tokens (… view at source ↗
Figure 7
Figure 7. Figure 7: (a) Registers with high probing accuracy encode diverse semantic information about an image, whereas (b) low-accuracy norm sinks do not. We visualize attention maps for register tokens and observe that some attend to distinct semantic regions, such as foreground objects and background areas. In contrast, norm sinks with low probing accuracy do not exhibit meaningful semantic structure. First, we find that … view at source ↗
Figure 8
Figure 8. Figure 8: In-context class tokens act as registers. (a) Certain tokens acquire disproportionately high feature norms, functioning as norm sinks. (b) Some tokens encode broad global information, rather than purely class-specific features as originally intended. 2.5 Registers Are Effective in Deeper Layers Next, we ablate both the number of register tokens and the transformer blocks in which they are introduced. We co… view at source ↗
Figure 19
Figure 19. Figure 19: 15 [PITH_FULL_IMAGE:figures/full_fig_p015_19.png] view at source ↗
Figure 9
Figure 9. Figure 9: High-norm outliers consistently emerge within register tokens across timesteps. We visualize token-wise feature norms of pDiT-B/16 with registers for t = 0.0, 0.3, and 0.7, and observe the same behavior in all cases. 0 50 100 150 200 250 token index 1 2 3 4 5 Token norm ×103 DT-B/16, Without Registers block 2 block 5 block 8 block 10 0 50 100 150 200 250 300 token index 0 1 2 3 4 5 Token norm ×104 DT-B/16,… view at source ↗
Figure 10
Figure 10. Figure 10: Token-wise feature norms for pDiTs of varying scales on ImageNet 256 × 256, with and without registers. Without registers, patch-token norms remain uniform across scales. Introducing registers leads to the emergence of high-norm outliers within the register tokens. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Token-wise feature norms for VAE-space SiTs of varying scales on ImageNet 256 × 256, with and without registers. SiTs without registers exhibit uniform patch-token norms across scales, while adding registers produces high-norm register tokens. 0 50 100 150 200 250 300 token index 0.5 1.0 1.5 2.0 Token norm ×102 RAE-S, With Registers block 0 block 2 block 5 block 8 block 10 0 50 100 150 200 250 token index… view at source ↗
Figure 12
Figure 12. Figure 12: Token-wise feature norms for DINOv2-space RAEs of varying scales on ImageNet 256 × 256, with and without registers. RAEs without registers exhibit uniform patch-token norms across scales, while adding registers produces high-norm register tokens. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Register tokens consistently reduce feature norms across patch tokens. We measure feature norms for image tokens only (excluding register tokens) at three diffusion timesteps for pDiT models of different scales, and observe a consistent reduction in feature norms across nearly all tokens when register tokens are used. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Register tokens make intermediate representations cleaner by reducing noise. We compute the Total Variation of intermediate features for models with and without register tokens. We report the ratio (with registers / without registers), where lower values indicate that models with registers produce smoother feature representations. We find that register tokens improve feature smoothness at high noise level… view at source ↗
Figure 15
Figure 15. Figure 15: Registers improve spatial organization at high noise levels. In addition to the TV ratio (left), we also analyze the correlation decay slope [45] (right), where lower values indicate stronger spatial organization. Both metrics show the same trend: register tokens improve internal representations at high noise levels, starting from block 4, where the registers are introduced. 21 [PITH_FULL_IMAGE:figures/f… view at source ↗
Figure 16
Figure 16. Figure 16: Linear probing of register tokens under different configurations. (Left) Standard register tokens introduced from the 4th layer; (Middle) Register tokens used as in-context class embeddings introduced from the 4th layer; (Right) Standard register tokens introduced from the 0th layer. Across different timesteps, we find that introducing registers from the earliest layers produces substantially less informa… view at source ↗
Figure 17
Figure 17. Figure 17: Pixel-space pDiTs have the highest feature norms across all tokens for different timesteps compared to latent-space counterparts. We compare token-wise feature-map norms for pDiT, SiT, and RAE models, all without register tokens. 0 4 9 13 17 21 26 avg Block 0.0 0.1 0.2 0.4 0.6 0.7 1.0 Timestep 0.12 0.59 2.05 1.19 0.53 0.18 0.40 0.72 0.23 0.83 1.32 1.19 0.47 0.10 0.18 0.62 0.33 1.10 1.20 0.96 0.44 0.11 0.2… view at source ↗
Figure 18
Figure 18. Figure 18: Pixel-space pDiTs exhibit substantially higher Total Variation (TV) values than latent-space counterparts. We compare the TV ratio of intermediate feature maps across timesteps and transformer blocks for pixel-space pDiTs (pDiT-H) and latent-space models (SiT-XL and RAE￾XL) without registers. Pixel-space pDiTs consistently produce noisier intermediate representations. 22 [PITH_FULL_IMAGE:figures/full_fig… view at source ↗
Figure 19
Figure 19. Figure 19: For SSL ViTs such as DINOv2, register tokens do not reduce patch-token feature norms, unlike in DiTs. We measure feature norms across all tokens for different blocks and model sizes of DINOv2, and observe that register tokens do not consistently reduce feature norms of patch tokens. 0 1000 2000 3000 4000 token index 0.0 0.5 1.0 1.5 2.0 token norm ×104 SD3.5. Norms of text (left) and image tokens (right) b… view at source ↗
Figure 20
Figure 20. Figure 20: Text sequences in text-to-image diffusion models exhibit behavior similar to register tokens in ImageNet-based DiTs: some tokens become high-norm outliers and potentially act as registers. We measure token-wise feature norms in SD3.5 (left) and FLUX (right) for both text and image tokens. We observe that the outliers primarily emerge within the text sequence. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_20.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.