pith. machine review for the scientific record. sign in

arxiv: 2604.11890 · v3 · submitted 2026-04-13 · 💻 cs.LG · stat.ML

Recognition: unknown

Subcritical Signal Propagation at Initialization in Normalization-Free Transformers

Sergey Alekseev
Authors on Pith no claims yet

Pith reviewed 2026-05-10 15:33 UTC · model grok-4.3

classification 💻 cs.LG stat.ML
keywords transformerssignal propagationinitializationLayerNormsubcriticalAPJNnormalization-freedynamic tanh
0
0 comments X

The pith

Transformers without LayerNorm exhibit subcritical signal propagation, with stretched-exponential APJN growth instead of power-law.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines how activations and gradients propagate through deep transformers at random initialization by tracking the averaged partial Jacobian norm, a quantity that measures average gradient amplification from one layer to the next. Recurrence relations are derived for both activation statistics and this norm under bidirectional attention and permutation-symmetric token inputs, yielding asymptotic predictions for large depth. The central result is that the familiar criticality distinction carries over from residual networks: pre-LayerNorm transformers display power-law growth in the norm, while architectures that replace LayerNorm with elementwise tanh-like nonlinearities display stretched-exponential growth and are therefore classified as subcritical. This classification directly accounts for the greater sensitivity to initialization scale and optimizer settings observed in Dynamic Tanh and Dynamic erf transformers.

Core claim

The criticality picture known from residual networks carries over to transformers: the pre-LayerNorm architecture exhibits power-law APJN growth, whereas transformers with LayerNorm replaced by elementwise tanh-like nonlinearities have stretched-exponential APJN growth, indicating that the latter are subcritical. The theory, obtained from recurrence relations for activation statistics and APJNs, also predicts how attention modifies the large-depth asymptotics and matches measured values in deep vision transformers.

What carries the argument

The averaged partial Jacobian norm (APJN), a scalar that quantifies the average amplification of back-propagated gradients across layers and thereby diagnoses whether signal propagation remains critical or becomes subcritical as depth increases.

Load-bearing premise

The recurrence relations for activation statistics and APJNs remain accurate for real deep vision transformers when bidirectional attention and permutation-symmetric token configurations are assumed.

What would settle it

Measure the depth scaling of APJN in a deep normalization-free transformer that uses tanh-like activations; observing power-law rather than stretched-exponential growth would falsify the subcritical classification.

Figures

Figures reproduced from arXiv: 2604.11890 by Sergey Alekseev.

Figure 1
Figure 1. Figure 1: (a) Backward APJN J B,b: theory vs. ViT on synthetic permutation-symmetric inputs. Circles denote backward APJN values measured in the ViT model on several permutation-symmetric input token configurations, for both the pre-LN variant and Derf variants with several values of α. Solid lines denote the backward APJN values predicted by the theoretical model in Eqs. (8)–(12). (b) Per-sample GMFE values, interp… view at source ↗
Figure 2
Figure 2. Figure 2: Backward APJNs for Derf and pre-LN at several values of [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Theoretical values of the scale parameter [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training stability comparison between Derf and pre-LN ViTs. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: (a) Effect of the number of warmup epochs on the training stability of Derf ViT for different α. (b) Effect of the learning rate on the training stability of Derf for different α. Panels (a) and (b) show test accuracy on CIFAR-100 after 10 training epochs. (c) Test accuracy of Derf models with small α over 90 training epochs. (4) Choosing an overly small initial value of α in Derf models can slow convergen… view at source ↗
Figure 2
Figure 2. Figure 2: APJNs: ViT (CIFAR-100 inputs) vs. theory vs. asymptotic theory. [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 6
Figure 6. Figure 6: The activation covariance components (a) Qb and (b) P b , as well as their ratio (c) P b/Qb . Upper row: theoretical values predicted by the recurrence relations (8) and (9); lower row: values measured in a ViT model for a permutation-symmetric input token configuration with (q 0 , p0 ) = (1.0, 0.2). 0 50 100 100 101 102 103 104 105 106 107 J B,b (σ21,σOV ) = (0.6, 1.2) theory ViT 0 50 100 100 101 102 103 … view at source ↗
Figure 7
Figure 7. Figure 7: Backward APJN J B,b: ViT on CIFAR-100 inputs versus theory comparison. (a) and (b) show two different CIFAR-100 input samples. Circles denote APJN values measured in ViT models (the pre-LN variant and Derf variants with multiple values of α). Solid lines denote the backward APJN values predicted by the theoretical model in Eqs. (8)–(12). 24 [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Backward APJN: ViT with CIFAR-100 inputs vs. theory at additional values of [PITH_FULL_IMAGE:figures/full_fig_p025_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: (a) Forward APJN J b,0 : theory vs. ViT on synthetic permutation-symmetric inputs. Circles denote forward APJN values measured in the ViT model on several permutation-symmetric inputs, for both the pre-LN variant and Derf variants with several values of α. Solid lines denote the forward APJN values predicted by the theoretical model in Eqs. (8)–(12). (b) Per-sample GMFE values between the theoretical value… view at source ↗
Figure 10
Figure 10. Figure 10: Backward APJN J B,b for ViT with Derf, together with the full-theory prediction and the asymptotic prediction from Eq. (18). (a) α = 0.3; (b) α = 1.9. Teal circles indicate backward APJN values measured in ViT on 100 random training samples from CIFAR-100. Black solid lines show the full-theory predictions obtained from the recurrence relations (8), (9), and (12) with (q 0 , p0 ) = (0.5, 0.25), while oran… view at source ↗
Figure 11
Figure 11. Figure 11: Ratio of the cross-positional Jacobian correlation to the APJN, computed from theory for [PITH_FULL_IMAGE:figures/full_fig_p026_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Histograms of normalized activation vector norms and normalized dot products between [PITH_FULL_IMAGE:figures/full_fig_p027_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Pre-LN: Histograms of within-sample relative fluctuations of self-dot products ( [PITH_FULL_IMAGE:figures/full_fig_p027_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Pre-LN: Sample-to-sample relative fluctuations of mean self-dot products ( [PITH_FULL_IMAGE:figures/full_fig_p027_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Derf, α = 1: Histograms of within-sample relative fluctuations of self-dot products (upper row) and cross-positional dot products between activation vectors (lower row) at multiple transformer blocks. 0 25 50 75 100 125 block 0.0 0.2 0.4 0.6 δ ¯Q /h ¯Qi δQ¯/hQ¯i 0 25 50 75 100 125 block 0.0 0.5 1.0 δ ¯P /h ¯Pi δP¯/hP¯i Sample-to-sample relative fluctuations of mean dot products (Derf, α = 1) [PITH_FULL_I… view at source ↗
Figure 16
Figure 16. Figure 16: Derf, α = 1: Sample-to-sample relative fluctuations of mean self-dot products (left) and mean cross-positional dot products between activation vectors (right) across transformer blocks. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Gradient amplification in the ViT model from [PITH_FULL_IMAGE:figures/full_fig_p030_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Upper: test performance curves; lower: training curves for the training-stability comparison between Derf and pre-LN when varying depth, shown in [PITH_FULL_IMAGE:figures/full_fig_p031_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Upper: test performance curves; lower: training curves for the training-stability compari￾son between Derf and pre-LN when varying the weight standard deviation, shown in [PITH_FULL_IMAGE:figures/full_fig_p031_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Training stability comparison between Derf and pre-LN as depth varies, using [PITH_FULL_IMAGE:figures/full_fig_p032_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Training stability comparison between Derf and pre-LN as depth varies, using [PITH_FULL_IMAGE:figures/full_fig_p032_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: (a) Effect of the number of warmup epochs and (b) effect of the learning rate on the training stability of Derf for different α, without weight decay. Comparison to [PITH_FULL_IMAGE:figures/full_fig_p033_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: (a) Effect of the number of warmup epochs and (b) effect of the learning rate on the training stability of Derf for different α, with α frozen during training. Comparison to [PITH_FULL_IMAGE:figures/full_fig_p033_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Comparison of a Derf α sweep and a pre-LN γ sweep at depth 12, showing test accuracy and training loss over 20 training epochs. 0 20 40 epoch 10 20 30 test accuracy (a) Smaller LR (LR = 1e-4, warmup = 10) α 0.1 0.2 0.3 0.5 0 20 40 epoch 10 20 30 test accuracy (b) Larger LR (LR = 1e-3, warmup = 10) α 0.1 0.2 0.3 0 20 40 epoch 10 20 30 test accuracy (c) Final LN not replaced with Derf (LR = 3e-4, warmup = 1… view at source ↗
Figure 25
Figure 25. Figure 25: Effect of initializing Derf with an overly small [PITH_FULL_IMAGE:figures/full_fig_p034_25.png] view at source ↗
read the original abstract

We study signal propagation at initialization in transformers through the averaged partial Jacobian norm (APJN), a measure of gradient amplification across layers. We extend APJN analysis to transformers with bidirectional attention and permutation-symmetric input token configurations by deriving recurrence relations for activation statistics and APJNs across layers. Our theory predicts how attention modifies the asymptotic behavior of the APJN at large depth and matches APJNs measured in deep vision transformers. The criticality picture known from residual networks carries over to transformers: the pre-LayerNorm architecture exhibits power-law APJN growth, whereas transformers with LayerNorm replaced by elementwise $\tanh$-like nonlinearities have stretched-exponential APJN growth, indicating that the latter are subcritical. Applied to Dynamic Tanh (DyT) and Dynamic erf (Derf) transformers, the theory explains why these architectures can be more sensitive to initialization and optimization choices and require careful tuning for stable training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims to extend APJN analysis to transformers by deriving recurrence relations for activation statistics and APJNs under bidirectional attention and permutation-symmetric token configurations. The theory predicts that pre-LayerNorm transformers exhibit power-law APJN growth with depth, while replacing LayerNorm with elementwise tanh-like nonlinearities yields stretched-exponential growth (indicating subcritical propagation). These asymptotics are asserted to match empirical APJN measurements in deep vision transformers and to explain the initialization sensitivity of DyT and Derf architectures.

Significance. If the recurrence derivations and empirical matches hold, the work provides a useful extension of the criticality framework from residual networks to transformers, offering a theoretical basis for understanding signal propagation without normalization and guiding the design of stable normalization-free models. The explicit distinction in asymptotic regimes and application to specific architectures like DyT/Derf add practical value for initialization and optimization in deep ViTs.

major comments (2)
  1. [Theory section on recurrence relations] The recurrence relations for activation statistics and APJNs (central to predicting the power-law vs. stretched-exponential distinction) are derived under bidirectional attention and permutation-symmetric inputs. The manuscript should explicitly demonstrate closure of these recurrences and test robustness when permutation symmetry is broken by learned positional embeddings, as standard ViTs include such embeddings that introduce token correlations not present in the mean-field setup; without this, the claimed distinction for real vision transformers does not necessarily follow.
  2. [Empirical validation section] The abstract states that the theory matches measured APJNs in deep vision transformers, but the load-bearing empirical validation requires details on the specific depths, architectures, and quantitative agreement (e.g., fitted exponents for power-law vs. stretched-exponential regimes) to confirm that finite-depth behavior aligns with the asymptotic predictions without unstated approximations.
minor comments (2)
  1. [Introduction and notation] Clarify the precise definition of APJN early in the manuscript and ensure all notation for attention and nonlinearity parameters is consistent between the recurrence derivations and the asymptotic analysis.
  2. [Applications to DyT/Derf] The discussion of DyT and Derf transformers would benefit from a brief table comparing their measured APJN growth rates to the theoretical predictions for tanh-like replacements.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each point below and have revised the manuscript to strengthen the presentation of recurrence closure and to expand the empirical details with quantitative metrics.

read point-by-point responses

  1. Referee: [Theory section on recurrence relations] The recurrence relations for activation statistics and APJNs (central to predicting the power-law vs. stretched-exponential distinction) are derived under bidirectional attention and permutation-symmetric inputs. The manuscript should explicitly demonstrate closure of these recurrences and test robustness when permutation symmetry is broken by learned positional embeddings, as standard ViTs include such embeddings that introduce token correlations not present in the mean-field setup; without this, the claimed distinction for real vision transformers does not necessarily follow.

    Authors: We appreciate the referee's emphasis on the assumptions in the mean-field derivation. In the revised manuscript we have added an explicit proof of closure in a new appendix subsection, verifying that the coupled recurrences for activation moments and APJNs form a closed system under bidirectional attention and permutation symmetry. Regarding positional embeddings, the empirical APJN measurements reported in Section 4 were obtained on standard Vision Transformers that employ learned positional embeddings, which already break strict permutation symmetry. The observed growth rates remain consistent with the predicted power-law and stretched-exponential regimes. We have further included an appendix robustness experiment comparing APJN trajectories with and without positional embeddings; the qualitative distinction between architectures is preserved, with only small quantitative shifts in the growth constants. revision: yes

  2. Referee: [Empirical validation section] The abstract states that the theory matches measured APJNs in deep vision transformers, but the load-bearing empirical validation requires details on the specific depths, architectures, and quantitative agreement (e.g., fitted exponents for power-law vs. stretched-exponential regimes) to confirm that finite-depth behavior aligns with the asymptotic predictions without unstated approximations.

    Authors: We agree that more granular reporting strengthens the empirical claims. The revised Section 4 now contains a table specifying the tested depths (ViT-Base at 6–48 layers and custom deep configurations up to 100 layers), the exact architectures (pre-LayerNorm, DyT, Derf), and quantitative fit statistics. Pre-LayerNorm models exhibit power-law APJN growth with fitted exponent 0.87 (R² = 0.96) over the measured range; the elementwise tanh-like replacements follow stretched-exponential growth with decay parameter 0.12 (R² = 0.94). These fits confirm that finite-depth observations align with the asymptotic predictions without additional approximations. revision: yes

Circularity Check

0 steps flagged

No circularity: recurrence derivations are independent and predictive

full rationale

The paper derives recurrence relations for activation statistics and APJNs from the bidirectional attention and permutation-symmetric token assumptions, then uses those closed-form recurrences to predict distinct asymptotic regimes (power-law vs. stretched-exponential APJN growth) at large depth. These predictions are compared to, rather than fitted from, measured APJNs in vision transformers. No self-citation chain, fitted-input-as-prediction, or self-definitional step is present; the derivation chain remains self-contained against external empirical benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no explicit free parameters, axioms, or invented entities; the analysis relies on standard mathematical derivations of activation statistics and Jacobian norms without introducing new postulated quantities.

pith-pipeline@v0.9.0 · 5444 in / 1047 out tokens · 61404 ms · 2026-05-10T15:33:21.304213+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. When Does Removing LayerNorm Help? Activation Bounding as a Regime-Dependent Implicit Regularizer

    cs.LG 2026-04 unverdicted novelty 5.0

    DyT improves validation loss 27% at 64M params/1M tokens but worsens it 19% at 118M tokens, with saturation levels predicting the sign of the effect.

Reference graph

Works this paper leans on

9 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [1]

    Stronger normalization-free transformers

    URLhttps://arxiv.org/abs/2512.10938. Aditya Cowsik, Tamra Nebabu, Xiao-Liang Qi, and Surya Ganguli. Geometric dynamics of signal propagation predict trainability of transformers, 2024. URLhttps://arxiv.org/abs/2403.02579. Darshil Doshi, Tianyu He, and Andrey Gromov. Critical initialization of wide and deep neural networks through partial jacobians: Genera...

    work page arXiv 2024

  2. [2]

    Deep networks with stochastic depth

    URLhttps://arxiv.org/abs/1603.09382. Akhil Kedia, Mohd Abbas Zaidi, Sushil Khyalia, Jungho Jung, Harshith Goka, and Haejun Lee. Transformers get stable: An end-to-end signal propagation theory for language models, 2024. URL https://arxiv.org/ abs/2403.09635. Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty ...

    work page arXiv 2024

  3. [3]

    Deep Information Propagation

    URLhttps://arxiv.org/abs/1611.01232. Sam Shleifer, Jason Weston, and Myle Ott. Normformer: Improved transformer pretraining with extra normal- ization, 2021. URLhttps://arxiv.org/abs/2110.09456. 11 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. ...

    work page Pith review arXiv 2021

  4. [4]

    Transformers without normalization

    URLhttps://arxiv.org/abs/2503.10622. Appendix The appendix is organized as follows. Sec. A provides experimental details for the results presented in the main text. Sec. B contains derivations of the covariance propagation equations. Sec. C derives the APJN recurrence relations, and Sec. D presents derivations of the asymptotic APJN behavior. Sec. E provi...

    work page arXiv 2025

  5. [5]

    entries N(0, σ 2 1/d)

    Covariance propagation through W1ϕ(·) or W1LayerNorm(·).Here W1 ∈R dff×d has i.i.d. entries N(0, σ 2 1/d). Assume the input activations have covariance matrix Σab, a, b∈ {1,2} , with Σ11 = Σ22 =qandΣ 12 = Σ21 =p. The propagated covariance is Σ′ ab = 1 dff Eθ h (W1˜ha)·(W 1˜hb) i = σ2 1 d Eθ h ˜ha · ˜hb i .(24) For LayerNorm, evaluating the dot product giv...

  6. [6]

    entries N(0, σ 2 2/dff)

    Covariance propagation through W2ReLU(·).Here W2 ∈R d×dff has i.i.d. entries N(0, σ 2 2/dff). Similarly, given an input covariance matrixΣ ab, the propagated covariance is Σ′ ab =σ 2 2 E(h1,h2)∼N(0,Σ) [ReLU(ha)ReLU(hb)].(26) These Gaussian integrals can be evaluated explicitly, giving: q′ = 1 2 σ2 2q, p ′ = 1 2 σ2 2κ p q q,(27) where κ(ρ) = 1 π p 1−ρ 2 +ρ...

  7. [7]

    Combining the results above gives the MLP contributions in Eqs

    Residual addition.The residual addition adds the covariance contributions from the residual stream and the residual branch, because they are independent. Combining the results above gives the MLP contributions in Eqs. (8) and (9). B.2 Covariance Propagation Through Attention Given the attention-layer activation update in Eq. (1) hl+1 a =h l a +W l OW l V ...

  8. [8]

    Assume the input activations have covariance matrixΣab, a, b= 1,

    Covariance propagation through attention.Define h′ a =W OWV Pn b=1 Aab ˜hb. Assume the input activations have covariance matrixΣab, a, b= 1, . . . , n, with Σaa =q and Σab =p for a̸=b . In the uniform-attention approximation [Noci et al., 2022], the propagated covariance is Σ′ ab = 1 d Eθ [h′ a ·h ′ b] =σ 2 OV nX t,s=1 1 n2 Eθ " ˜ht · ˜hs d # .(30) Let ˜Σ...

    2022

  9. [9]

    ˜ht · ˜hs d # =σ 2 OV nX t,s=1 1 n2 Eθ

    Residual addition.The residual addition adds the covariance contributions from the residual stream and the residual branch, because they are independent: ql+1 ≈q l +σ 2 OV ˜p, p l+1 ≈p l +σ 2 OV ˜p.(34) B.3 Covariance Propagation Through Multi-Head Attention In the case of multiple attention heads, Eq. (34) remains unchanged under the standard weight init...

    2024