Pith. sign in

REVIEW 3 major objections 4 minor 28 references

This paper argues that Transformer feedforward design — skip connections, normalization placement, the two-matrix branch, width expansion — is best understood as preserving gradient and representation rank across depth, governed by a single

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 03:00 UTC pith:HEUR4GDA

load-bearing objection A serious, honest paper with two tight parameter-free laws and one central √L unification that is empirically calibrated rather than derived; worth refereeing despite the need for multi-seed replication. the 3 major comments →

arxiv 2607.14018 v1 pith:HEUR4GDA submitted 2026-07-15 cs.LG cs.AI

Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth

Katie Everett This is my paper
classification cs.LG cs.AI
keywords rank collapseeffective ranktransformer feedforward blocknormalization placementskip connectionsresidual networksMarchenko–Pastur lawdepth scaling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the components of a Transformer feedforward block — skip connections, normalization placement, the two-matrix branch, the width expansion — are best understood as mechanisms for preserving the effective rank of gradients and representations across depth, where effective rank measures how many independent directions a matrix's spectrum spans. Its central claim is that one number, the branch-to-skip ratio (how much of the gradient flows through the rank-reducing residual branch versus the identity skip), determines how much Jacobian rank survives: Post-Norm holds the ratio constant and rank collapses, while Pre-Norm lets the residual stream grow so the ratio decays and rank plateaus. Quantitatively, the initialization rank depends only on the product of that ratio and a √L depth factor, a law that unifies depth-scaling prescriptions such as the 1/√L branch downscaling. The paper also derives exact thresholds: the second branch matrix decorrelates a coherent mean spike (cross-layer alignment E[σ]²/E[σ²]) that would collapse the forward representation, and the width-expansion ratio must reach 1/p(σ) — 2× for ReLU — for the branch Jacobian to be full rank. If right, a cheap measurement at initialization predicts whether a configuration will train (demonstrated on CIFAR-10), and architecture design becomes a three-way tradeoff among rank collapse, ensemble-like behavior, and parameter count.

Core claim

At initialization, the input–output Jacobian's effective rank depends on the branch-to-skip ratio βα^M times a √L depth factor. Decomposing the Jacobian into paths through k branches, near-independent branch Jacobians make each order a function of βα^M√L alone; iso-rank contours scale as 1/√L. Normalization placement sets how the ratio evolves: a post-residual rescaling holds it constant and collapses rank; Pre-Norm or branch-output placement lets the stream grow, the ratio decay as 1/√ℓ, and rank plateau. The second matrix decorrelates branch means (alignment E[σ]²/E[σ²]), preventing a rank-1 mean spike; width expansion must reach m/d = 1/p(σ) to make the branch Jacobian full rank. This ran

What carries the argument

Load-bearing object: the branch-to-skip ratio r_ℓ = ‖βJ_ℓ‖_F / ‖I‖_F — the size of a block's branch Jacobian relative to the identity skip — with nominal form βα^M (β branch scale, α per-matrix init scale, M branch matrices). Central tool: the path decomposition of the Jacobian into sums over paths visiting k branches; near-independence of branch Jacobians across blocks at initialization gives each order-k term norm (βα^M√L)^k/√k!, collapsing all depth dependence into βα^M√L. Secondary mechanisms: the cross-layer branch-mean coherence c_ℓ = E[σ]²/E[σ²], which the second matrix decorrelates to zero, preventing a rank-1 mean spike; and the Marchenko–Pastur threshold γ = p(m/d) = 1, yielding wi

Load-bearing premise

The √L law — and the whole unification of rank around the single variable βα^M√L — assumes that at initialization branch Jacobians of different blocks point in nearly uncorrelated directions, so path contributions add like random walks (Appendix B.3); the paper's own trained-GPT-2 measurements show this assumption already fails after training (block cosine 0.15–0.18, accumulated branch 1.8–2.2× the uncorrelated prediction), making the law an initialization-only statement, and

What would settle it

Measure iso-rank contours at depth beyond 384 in a residual MLP at initialization: the law predicts βα^M ∝ L^{−1/2}; a fitted slope drifting toward −1 would mean path correlations are material and the single-variable law fails. A direct microscopic check measures the order-1 accumulated branch norm, which the random-walk assumption fixes at βα^M√L — the paper finds exactly this at initialization (1.00×) and 1.78–2.23× in trained GPT-2, so a clean experiment would track the accumulated-branch ratio while progressively correlating blocks, to find the deviation threshold where the √L law breaks.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single measurement at initialization — the input–output Jacobian's effective rank — predicts whether a configuration trains: on CIFAR-10, networks whose rank collapses to 1–2% of the width fail (accuracy falls to chance), while moderate collapse costs a few points and Pre-Norm/Output-Norm configurations train across the whole grid.
  • The 1/√L branch-downscaling prescription coincides exactly with holding βα^M√L constant, making it the scaling at which initialization rank is depth-invariant — a candidate explanation for why that scaling recurs across the depth-scaling literature.
  • Only the post-residual normalization placement collapses rank, because it is the one that holds the branch-to-skip ratio constant; this unifies the empirical move from Post-Norm to Pre-Norm and explains why branch-input and branch-output placements keep a rank floor.
  • The width-expansion threshold m/d = 1/p(σ) puts the standard 4× feedforward expansion comfortably above the full-rank boundary (2× for ReLU) and predicts the needed expansion for smooth activations: about 2.2× for GELU and 2.6× for SiLU.
  • Pre-Norm's rank plateau and its deep layers becoming near-identity perturbations are two sides of the same decaying ratio — consistent with evidence that deep layers of trained models can be pruned with little loss and that language models behave like ensembles of shallower subnetworks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Testable extension: sweep the branch-to-skip ratio on tasks that differ in how much they reward layer composition. If the paper's framing is right, accuracy on multi-step reasoning should peak at a larger ratio than next-token prediction, whose ensemble-like behavior the paper shows is consistent with a decaying ratio — a prediction the paper gestures at but does not test.
  • The √L law is an initialization statement; the paper's trained-GPT-2 measurements show training induces block correlations that inflate the accumulated branch by 1.8–2.2×. An untested extension is a trained-time analogue with a correlation-corrected depth exponent, which would connect the rank law to the training dynamics the paper leaves open.
  • The coherence identity c_ℓ = E[σ]²/E[σ²] suggests a general design criterion the paper does not state: any branch whose elementwise nonlinearity has nonzero output mean needs either a second matrix or a mean-subtracting normalization after the activation; this could be probed for gated activations, which the paper names as an open case.
  • The Marchenko–Pastur threshold implies a conditioning margin γ − 1 above 1/p(σ); a natural, untested design question is whether downstream performance tracks that margin — e.g., whether activations with low survival rates (SiLU) benefit from expansions beyond the standard 4×.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a unified account of how the components of a Transformer feedforward block control the effective rank of gradients and representations at initialization. It argues that skip connections route the Jacobian around rank-reducing branch operations and that the branch-to-skip ratio βα^M controls the tradeoff between rank collapse and ensemble-like behavior. It further claims that normalization placement sets this ratio across depth: Post-Norm holds the ratio constant and collapses, while Pre-Norm lets the ratio decay as the residual stream grows, so rank plateaus; quantitatively, the initialization rank is claimed to depend only on βα^M√L, with iso-rank contours scaling as 1/√L. The paper also analyzes the two-matrix branch: the second matrix decorrelates branch means, preventing a coherent mean-spike collapse whose per-layer constant is c_ℓ=E[σ]^2/E[σ^2]; and the width expansion preserves branch-Jacobian rank, with a Marchenko–Pastur threshold m/d=1/p(σ). Finally, initialization effective rank is shown to predict CIFAR-10 trainability across a depth/initialization grid.

Significance. If correct, this is a valuable unifying picture: it ties normalization placement, residual scaling, activation choice, and width expansion to one observable—the effective rank of the input–output Jacobian at initialization—and produces several parameter-free, closed-form predictions. The paper's strengths are its machine-checkable numerical checks (e.g., the path-decomposition reconstruction, exact Marchenko–Pastur spectra for ReLU/abs, and the c_ℓ correlation measured as 0.320 vs. 1/π≈0.318), the explicit equivalence βα^M symmetry, and the clear separation between measured laws and heuristic derivations. The CIFAR-10 trainability maps, while coarse, are a credible falsifiable demonstration. However, the central √L depth law currently rests on a single-sample empirical calibration and a heuristic random-walk argument whose validation is limited to order-1 path sums in GPT-2; these gaps need to be addressed before the unification claim is fully supported.

major comments (3)
  1. [Section 4, Figure 5, Table 4] The paper's central quantitative claim—that the effective rank at initialization depends only on βα^M√L, with iso-rank contours scaling as 1/√L—is supported empirically by Figure 5 and Table 4. The text explicitly states that 'the rank measurements in this section use a single sample of the weights and the input'. A single draw cannot bound the sampling variability of the fitted contour exponent s≈−0.50, which is a load-bearing number: if the collapse onto βα^M√L is a single-seed artifact, the unification of Pre-Norm and Post-Norm fails. The heuristic in Appendix B.3 does not fully substitute, since the paper states it does not attempt a closed form for the rank curve. Please provide multi-seed measurements with error bars on s, and show the βα^M√L collapse and the 1/√L contours either for individual seeds or with explicit quantiles.
  2. [Appendix B.3, Eq. (23)] The √L law rests on the assumption that branch Jacobians at different blocks are uncorrelated, so order-k path sums add like random walks. The validation in Figure 12 measures only order-1 accumulated-branch norms, and it does so on GPT-2 (GELU with attention sublayers), not on the ReLU residual MLP used in Figure 5. The authors themselves report that after training the accumulated branch reaches 1.78×–2.23× the uncorrelated prediction, so the assumption is not universal; within initialization it is verified only for a different architecture and only for k=1. Since positive correlations at higher path orders would change the depth scaling from βα^M√L toward something closer to βα^M L, the manuscript needs either an empirical test of Eq. (23) for k>1 in the exact Figure 5 setting, or a derivation from weight independence that does not rely on measured block decorrelation.
  3. [Section 5 and Appendix C] The derivation of the cross-layer identity cos(m_ℓ,m_ℓ′)=c_ℓ is presented in Appendix C as taking place 'in the regime where the mean spike dominates the residual stream', with the pre-activation collapsing to z≈1. That regime is exactly the collapse that the identity is then used to explain. The empirical measurement of 0.320 for ReLU and 0.188 for GELU is convincing as a correlation statement, and the identity may well hold more broadly, but the causal explanation has a circular flavor: the formula is derived after the spike has already formed. To make the explanatory claim load-bearing, the paper should either prove or directly measure that the correlation equals c_ℓ before the spike dominates, or explicitly reframe the derivation as an asymptotic characterization of the collapsed regime rather than the mechanism of its onset.
minor comments (4)
  1. [Section 4, Figure 6] Each CIFAR heatmap cell trains one network per learning rate, with no seed averaging. The paper already calls CIFAR-10 a 'coarse test of trainability', but the absence of error bars makes it hard to judge whether the boundary between trainable and untrainable regions is sharp. Reporting accuracy over a few seeds, or at least the standard deviation in the transition region, would strengthen the claim.
  2. [Section 2 and Figure 2 captions] Figure 2 states 'single seed' while Figure 1 reports 'mean over 12 seeds'. This inconsistency in reporting conventions is confusing, especially in a section that makes a symmetry-collapse claim. Please state the seed policy uniformly, or explain why the single-seed figure is justified by the exact βα^M symmetry.
  3. [Appendix B.3, Figure 12] The GPT-2 validation is performed on 'a fixed prompt'. The manuscript should state whether the conclusions are stable across prompt positions and sequence lengths, or at least add a sentence noting that the block-correlation and accumulated-norm measurements were checked for prompt dependence.
  4. [References] Several cited items carry 2026 dates (e.g., Chen and Wei, Emadi, Fahim and Karim, Wang et al., Liu et al.) and appear as arXiv preprints. Please ensure all references are complete with version identifiers and that any works still under review are clearly marked as preprints; if the journal has a policy on citing future-dated preprints, this should be followed consistently.

Circularity Check

1 steps flagged

Mild derivation-level circularity in the c_ℓ alignment law; central depth and threshold laws are self-contained.

specific steps
  1. other [Section 5 and Appendix C, 'The cross-layer correlation equals c_ℓ (spiked regime)']
    "We derive the main-text identity cos(mℓ, mℓ′) = cℓ for a deep network at initialization, in the regime where the mean spike dominates the residual stream. ... Once the spike dominates, the representations of all inputs have collapsed onto its direction, which is the all-ones direction 1 ... After the Pre-Norm every input then has z≈1, independent of x."

    The paper argues that cross-layer branch-mean alignment c_ℓ = E[σ]²/E[σ²] controls the linear growth of the mean spike and hence the residual-rank collapse. But the derivation offered for cos(mℓ,mℓ′)=c_ℓ assumes the target regime: 'once the spike dominates' and 'z≈1', i.e. it assumes the collapse that the alignment is supposed to explain. The causal chain 'alignment → growth → collapse' is therefore partly self-supporting at the derivation level. The empirical match (0.320 vs 1/π, 0.188 vs 0.187) provides independent support, so this is a mild, non-fatal circularity rather than a fitted result renamed as a prediction.

full rationale

Aside from the Appendix C spiked-regime derivation, the paper's central quantitative claims are not circular. The βα^M equivalence is proven by positive homogeneity (Appendix A.4), so Figure 2's collapse is a theorem rather than a fit. The √L depth law is presented as an empirical finding with a stated heuristic (Appendix B.3) whose assumptions are explicitly listed and tested on GPT-2; no equation reduces to a fitted value that is then called a prediction. The Marchenko–Pastur threshold m/d=1/p(σ) is derived from the Jacobian factorization and the half-space conditioning identity, and the CIFAR rank-trainability relation is a correlation, not a fitted parameter reused as a prediction. There are no load-bearing self-citations (the author does not cite prior work of her own). The single-seed rank measurements noted in Section 4 are a statistical robustness caveat, not circularity. The one mild circular step is the c_ℓ derivation in Appendix C, which assumes the spike-dominated collapse regime it is used to explain; because the identity is also measured independently and the main-text mechanism (shared all-ones component) does not logically require the spiked limit, this does not undermine the overall derivation chain.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

No new particles, forces, or conserved quantities are postulated. The 'coherent mean spike' and 'branch-to-skip ratio' are named mechanisms/quantities, not entities with independent falsifiable handles; the mean spike is inherited from Davis & Drusvyatskiy (2025). The closed-form laws (c_ℓ, 1/p threshold) have zero fitted inputs; the three listed fitted values are calibrations/validations supporting the empirical depth law and mechanism claims.

free parameters (3)
  • iso-rank contour exponent s = −0.50 (range −0.46 to −0.57 across conditions, Table 4)
    The central depth law 'rank depends on βα^M√L' is established by measuring iso-rank contours in log-log space; the √L factor is the empirical exponent, not a derived exact result. The paper states 'we measure the shape of each curve empirically' (B.3).
  • mean-growth exponents q (M=1, M=2) = 0.99 (M=1 ReLU), 0.56 (M=2 ReLU)
    Fitted power-law exponents used to support the claim of linear vs random-walk accumulation of branch means (Figure 7b); confirmatory of the c_ℓ mechanism rather than an input to it.
  • stream-growth coefficient κ = 0.51 (measured; ReLU theory E[σ²]=1/2)
    Validates the 1/√ℓ ratio-decay law (Eq. 22) underlying the Pre-Norm plateau; theory-derived value is 0.5, so this is a check, not a fitted input.
axioms (6)
  • domain assumption Per-block branch Jacobians are independent at initialization, so order-k path sums add like random walks (Eq. 23).
    Load-bearing for the √L depth law and the claim that rank depends only on βα^M√L; stated in Appendix B.3, validated at init and mildly violated after training on GPT-2 (cosine 0.15–0.18; accumulated norm 1.78–2.23× prediction).
  • domain assumption Activations are positively homogeneous (σ(αz)=ασ(z)) for the βα^M equivalence.
    Section 3 / A.4; exact for ReLU and abs; GELU and SiLU (used in Table 1 and Figure 8) satisfy it only approximately, so the 'rank depends only on βα^M' claim is approximate for them.
  • domain assumption W_up and W_down are asymptotically free; the masked submatrix is an i.i.d. Gaussian ensemble with second moment I.
    Section 6 / D.5: needed for the Marchenko–Pastur spectrum and Gaussian-product curve; validated on pretrained GPT-2 Medium (mean bulk coherence 0.96 vs Haar baseline 1), though GPT-2 uses GELU while the exact MP statement is for ReLU/abs.
  • domain assumption In the mean-spike regime the Pre-Norm input collapses to z≈1, making cross-layer pre-activations independent.
    Appendix C: the derivation of cos(m_ℓ,m_ℓ′)=c_ℓ assumes the spike dominates; stated as leading-order valid as the spike grows and d→∞, with anchor checks confirming (0.565 vs √(1/π)=0.564).
  • domain assumption Normalization projection terms contribute negligibly to rank loss.
    Section 4 / B.2: bounded at ≤0.4% of full rank via the detached-rescaling ablation and projection-product table; assumed throughout Section 4's branch-to-skip ratio argument.
  • standard math Marchenko–Pastur law and concentration in the proportional limit d, m→∞; binomial count L^k/k! for k≪L.
    Section 6 / D.3 and Appendix B.3; standard random-matrix background the threshold derivation and path count rest on.

pith-pipeline@v1.3.0-alltime-deepseek · 41514 in / 21506 out tokens · 192545 ms · 2026-08-02T03:00:27.140898+00:00 · methodology

0 comments
read the original abstract

We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization. We reinterpret skip connections and normalization, long understood as controlling magnitude, as mechanisms for preserving gradient rank across depth, since the very matrix multiplications and nonlinear activations that make the network expressive also reduce the rank. We show that skip connections trade off rank collapse against ensemble-like behavior, controlled by the relative scales of the branch and the skip: skip connections route the gradient around the residual branch, where rank is lost, rather than along the long gradient paths that encourage the layers to compose. The placement of the normalization layer controls this same tradeoff by setting the branch-to-skip ratio across depth, unifying much of the normalization placement and depth scaling literature, in particular why rank collapses for Post-Norm but plateaus for Pre-Norm. Other aspects of the architecture, like the two-matrix structure that expands and contracts the width, use additional parameters to preserve the representation or branch Jacobian rank. The second matrix decorrelates a coherent mean spike that would grow across blocks with a single matrix and uncentered activation, preventing the residual representation from collapsing. The width expansion between the two matrices keeps the branch Jacobian full rank: applying the rank-reducing activation in this expanded space leaves enough directions to span the original, at a width that follows a Marchenko--Pastur law. The initialization rank of the input--output Jacobian predicts which networks train on CIFAR-10. Taken together, we recast architecture design for deep networks as navigating an intrinsic tradeoff among rank collapse, ensemble-like behavior, and parameter count.

Figures

Figures reproduced from arXiv: 2607.14018 by Katie Everett.

Figure 1
Figure 1. Figure 1: Skip connections restore the effective rank that each feedforward block contracts; the residual scale β and the width expansion control how much survives with depth. Effective rank of the Jacobian ∂hℓ/∂h0 up to each layer ℓ, normalized by the width (model dimension) d, for a Pre-Norm ReLU residual MLP (L = 48, d = 256, β = 1, α = 1, and 4× width expansion unless otherwise stated; mean over 12 seeds). Left:… view at source ↗
Figure 2
Figure 2. Figure 2: The effective rank at initialization depends on the branch scale β and the initialization scale α only through the nominal branch-to-skip ratio βαM (M the number of branch matrices), for both normalization placements. Effective rank of the input–output Jacobian, normalized by the width, for a residual MLP at initialization (d = 256, L = 48, two square d × d branch matrices, single seed). (a) Pre-Norm: the … view at source ↗
Figure 3
Figure 3. Figure 3: Short paths (mostly skip connections) dominate the input–output Jacobian magnitude; a smaller branch scale β or initialization scale α shifts it toward even shorter paths. The path lengths depend only on the nominal branch-to-skip ratio βαM. Path decomposition of a Pre-Norm residual MLP at initialization (d = 256, L = 48, 4× width expansion; mean over 8 seeds). (a) Per-order Jacobian magnitude: the Frobeni… view at source ↗
Figure 4
Figure 4. Figure 4: The projection terms in the normalization Jacobian are not the cause of the rank collapse. All panels: residual networks at initialization with a two-matrix ReLU branch (both matrices d × d, d=256), β=1; effective rank of the input–output Jacobian, normalized by width. Panels (a) and (b) fix α=1. (a) Only the post-residual placement collapses rank. A normalization at either branch placement, the input (Pre… view at source ↗
Figure 5
Figure 5. Figure 5: The effective rank depends only on the product of the branch-to-skip ratio βαM (exact symmetry) and a √ L depth factor (empirical). Post-Norm collapses to zero, while Pre-Norm plateaus at a moderate level. All panels at initialization with a two-matrix ReLU branch (both matrices d × d, d=256), β=1; βαM is the nominal branch-to-skip ratio (M branch matrices). (a) The norm placement controls the branch-to-sk… view at source ↗
Figure 6
Figure 6. Figure 6: The initialization effective rank predicts CIFAR-10 trainability. Heatmaps over initialization scale α and depth L, for residual networks with a two-matrix ReLU branch (both matrices d × d, d=256) and β=1. (a,c) Effective rank of the input–output Jacobian at initialization. (b,d) Best CIFAR-10 test accuracy over a learning-rate sweep with Adam. For Pre-Norm (a,b), the rank stays above 0.2 and training succ… view at source ↗
Figure 7
Figure 7. Figure 7: A single uncentered branch matrix grows a mean spike that collapses the forward residual stream rank; a second matrix or a centered activation prevents the collapse. We track the residual stream representation Zℓ, the matrix whose rows are the representations of n probe inputs at layer ℓ, in a depth-L network at initialization (Pre-RMSNorm, β=1, d=256, L=512, 3 seeds). (a) The mean of Zℓ over inputs is the… view at source ↗
Figure 8
Figure 8. Figure 8: The branch Jacobian’s effective rank increases with the FFN expansion ratio, and a Gaussian￾product law predicts it. (a) The normalized effective rank erank(Jf )/d increases with the expansion ratio m/d for ReLU, GELU, SiLU, and abs, fastest for abs (p = 1); GELU and SiLU reach a higher effective rank than ReLU at small expansion ratios despite their smaller p. (b) Plotted against γ = p (m/d), ReLU and abs… view at source ↗
Figure 9
Figure 9. Figure 9: The masked up-projection follows the Marchenko–Pastur law, and its conditioning improves as m/d rises above 1/p. Eigenvalue histogram (filled) of the Gram matrix G = 1 neff (D Wup) ⊤(D Wup) of the masked up-projection D Wup against the Marchenko–Pastur density (line), for ReLU, GELU, and abs (rows) at width expansions m/d ∈ {2, 4, 8} (columns). The mask D = diag(σ ′ (Wupx)) keeps a fraction p of the m rows… view at source ↗
Figure 10
Figure 10. Figure 10: Without the Pre-RMSNorm control, the unnormalized configurations collapse and mask the operation comparison. Effective rank of the input–output Jacobian across depth at α=β=1, as in Figure 4b but with no Pre-RMSNorm. (a) With no operation or mean subtraction alone, there is no normalization on the branch: the branch-to-skip ratio stays constant and the rank collapses at nearly the rate of the rescaling op… view at source ↗
Figure 11
Figure 11. Figure 11: At initialization, rms(hℓ) 2 grows like 1 + κ(βαM) 2 ℓ with κ = E[σ 2 ] = 1 2 for ReLU; the measured κ ≈ 0.51 agrees at every initialization scale. (a) rms(hℓ) 2 versus depth for a Pre-Norm two-matrix ReLU branch (both matrices d × d, d=256; β=1), shown as the measured stream (solid) against the fit (dashed); the stream grows faster at larger α. (b) Plotted against (βαM) 2 ℓ, rms(hℓ) 2 − 1 collapses onto … view at source ↗
Figure 12
Figure 12. Figure 12: The assumptions behind the √ L depth factor hold exactly in GPT-2 at initialization; after training, the path decorrelation is mildly violated and the stream grows faster. The path-counting argument assumes that the FFN branch Jacobians of different blocks are uncorrelated, so that the path sums of Eq. 23 accumulate as random walks, and the Pre-Norm argument depends on the residual stream growing while th… view at source ↗
Figure 13
Figure 13. Figure 13: Output-Norm trains across the grid, like Pre-Norm. Initialization effective rank over width (left) and best CIFAR-10 test accuracy over an Adam learning-rate sweep (right) for a normalization at the branch output (β=1), over the same grid as [PITH_FULL_IMAGE:figures/full_fig_p034_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: The branch-mean correlation is the same for every pair of layers, not only neighbors. Pairwise cosine cos(mℓ, mℓ ′ ) of the branch means at initialization (Pre-RMSNorm, β=1, d=256, L=512, n=4096, 3 seeds; the matrix samples the depth-512 network every 8 layers). (a) A single uncentered matrix aligns every pair at the coherence constant cℓ = 1/π (off-diagonal mean 0.320); the field is uniform, so the align… view at source ↗
Figure 15
Figure 15. Figure 15: Pre-Norm cannot remove the mean spike; a branch-output normalization can, but only by subtracting the mean. Effective rank erank(Zℓ)/d for M=1 ReLU at initialization (β=1, d=256, 3 seeds). (a) In the Pre-Norm placement both normalizers collapse: Pre-LayerNorm, which centers the branch input, reaches 0.078 by depth 512 against 0.009 for Pre-RMSNorm — roughly eightfold slower, but still collapsed. (b) At th… view at source ↗
Figure 16
Figure 16. Figure 16: Without normalization the coherent mean spike halves the trainable depth; with normalization the rank collapse it causes does not impair CIFAR training. All ReLU, β=1, width 256. (a) Without normalization, the single-matrix architecture (M=1) fails by about depth 50 and the two-matrix architecture (M=2) by about depth 100 (best test accuracy over the learning-rate sweep; CIFAR chance accuracy 0.1). (b) Th… view at source ↗
Figure 17
Figure 17. Figure 17: Submatrix spectra for all four activations. Measured Gram eigenvalue histograms (filled) of D Wup against the Marchenko–Pastur density (line), for ReLU, GELU, SiLU, and abs (rows) at width expansions m/d ∈ {2, 4, 8} (columns). Exact (solid) for ReLU and abs, approximate (dashed) for GELU and SiLU. Companion to [PITH_FULL_IMAGE:figures/full_fig_p040_17.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 16 linked inside Pith

  1. [1]

    Only large weights (and not skip connections) can prevent the perils of rank collapse

    Josh Alman and Zhao Song. Only large weights (and not skip connections) can prevent the perils of rank collapse. arXiv preprint arXiv:2505.16284,

    Pith/arXiv arXiv

  2. [3]

    Baker, Barak A

    Bradley T. Baker, Barak A. Pearlmutter, Robyn Miller, Vince D. Calhoun, and Sergey M. Plis. Low-rank learning by design: the role of network architecture and activation linearity in gradient rank collapse.arXiv preprint arXiv:2402.06751,

    Pith/arXiv arXiv

  3. [5]

    Each Sk multiplies k branch Jacobians, so it inherits the rank deficiency of a k-fold branch composition, weighted byβ k

    The order k counts the feedforward blocks a path passes through, from S0 =I (the pure skip) to SL =J L · · ·J1 (every branch). Each Sk multiplies k branch Jacobians, so it inherits the rank deficiency of a k-fold branch composition, weighted byβ k. The order-k component βkSk is the part of the Jacobian built from paths through exactly k feedforward blocks...

    2021

  4. [6]

    A dash marks a configuration that does not reach0.4at any learning rate in the sweep (3×10 −5 to10 −2)

    Pre-Norm stays at the top of the sweep at every depth; Post-Norm’s stable range narrows with depth, faster at larger α. A dash marks a configuration that does not reach0.4at any learning rate in the sweep (3×10 −5 to10 −2). placementα L=12 24 48 96 192 384 Pre-Norm all−2−2−2−2−2−2 Post-Norm0.5−2−2−2.5−3−3.5−4.5 Post-Norm1−2.5−3−3.5−4— — Post-Norm2−2.5−3.5...

    2025

  5. [8]

    Exact attention sensitivity and the geometry of transformer stability.arXiv preprint arXiv:2602.18849,

    Seyed Morteza Emadi. Exact attention sensitivity and the geometry of transformer stability.arXiv preprint arXiv:2602.18849,

    arXiv

  6. [10]

    Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

    Gemma Team. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

    Pith/arXiv arXiv

  7. [11]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Gemma Team. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Pith/arXiv arXiv

  8. [13]

    Kimi K2: Open agentic intelligence.arXiv preprint arXiv:2507.20534,

    Kimi Team. Kimi K2: Open agentic intelligence.arXiv preprint arXiv:2507.20534,

    Pith/arXiv arXiv

  9. [14]

    BranchNorm: Robustly scaling extremely deep transformers

    Yijin Liu, Xianfeng Zeng, Fandong Meng, and Jie Zhou. BranchNorm: Robustly scaling extremely deep transformers. InFindings of the Association for Computational Linguistics: ACL 2024, pages 11675–11687,

    2024

  10. [15]

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Llama Team. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  11. [17]

    gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925,

    OpenAI. gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925,

    Pith/arXiv arXiv

  12. [18]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Qwen Team. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  13. [20]

    Normformer: Improved transformer pretraining with extra normalization

    21 Sam Shleifer, Jason Weston, and Myle Ott. Normformer: Improved transformer pretraining with extra normalization. arXiv preprint arXiv:2110.09456,

    Pith/arXiv arXiv

  14. [21]

    B2T connection: Serving stability and performance in deep transformers

    Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. B2T connection: Serving stability and performance in deep transformers. InFindings of the Association for Computational Linguistics: ACL 2023, pages 3078–3095,

    2023

  15. [23]

    Tensor programs IVb: Adaptive optimization in the infinite-width limit.arXiv preprint arXiv:2308.01814,

    Greg Yang and Etai Littwin. Tensor programs IVb: Adaptive optimization in the infinite-width limit.arXiv preprint arXiv:2308.01814,

    Pith/arXiv arXiv

  16. [25]

    and for the Pre-Norm residual stream in Blake et al. [2025]. The rescaling must extend to other optimizer hyperparameters, including ϵ in Adam, weight decay, and gradient clipping [Yang and Littwin, 2023]. This equivalence holds exactly only in infinite precision and infinite dynamic range. B Normalization Placement: Additional Results and Derivations B.1...

    2025

  17. [27]

    (c)A centered activation has no mean to align (mean−0.000)

    34 0 256 512 Layer 0 256 512Layer ′ (a) M = 1 ReLU 0 256 512 Layer 0 256 512 (b) M = 2 ReLU 0 256 512 Layer 0 256 512 (c) M = 1 tanh 0.35 0 cl = 1/ cos(m , m ′) Figure 14:The branch-mean correlation is the same for every pair of layers, not only neighbors.Pairwise cosine cos(mℓ, mℓ′) of the branch means at initialization (Pre-RMSNorm,β=1, d=256, L=512, n=...

    2025

  18. [28]

    We call the expected fraction of units with nonzero derivative the survival rate,p(σ) = Prz∼N(0,1) [σ′(z)̸= 0]

    37 D.1 Jacobian factorization and the survival rate The feedforward branchf(x) =W down σ(Wupx), withW up ∈R m×d andW down ∈R d×m, has Jacobian Jf (x) =W down D(x)W up, D(x) = diag σ′(Wupx) .(25) The mask D(x) is diagonal, so it can only lower rank: rankJ f ≤#{i:σ ′((Wupx)i)̸= 0} . We call the expected fraction of units with nonzero derivative the survival...

    1967

  19. [2014]

    GLU variants improve transformer.arXiv preprint arXiv:2002.05202,

    Noam Shazeer. GLU variants improve transformer.arXiv preprint arXiv:2002.05202,

    Pith/arXiv arXiv 2002

  20. [2016]

    SpanNorm: Reconciling training stability and performance in deep transformers.arXiv preprint arXiv:2601.22580,

    Chao Wang, Bei Li, Jiaqi Zhang, Xinyu Liu, Yuchun Fan, Linkun Lyu, Xin Chen, Jingang Wang, Tong Xiao, Peng Pei, and Xunliang Cai. SpanNorm: Reconciling training stability and performance in deep transformers.arXiv preprint arXiv:2601.22580,

    Pith/arXiv arXiv

  21. [2019]

    Layer normalization.arXiv preprint arXiv:1607.06450,

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization.arXiv preprint arXiv:1607.06450,

    Pith/arXiv arXiv

  22. [2020]

    DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437,

    DeepSeek-AI. DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437,

    Pith/arXiv arXiv

  23. [2021]

    MiniMax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313,

    MiniMax. MiniMax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313,

    Pith/arXiv arXiv

  24. [2022]

    When do spectral gradient updates help in deep learning?arXiv preprint arXiv:2512.04299,

    Damek Davis and Dmitriy Drusvyatskiy. When do spectral gradient updates help in deep learning?arXiv preprint arXiv:2512.04299,

    arXiv

  25. [2023]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016a. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630...

    Pith/arXiv arXiv

  26. [2024]

    Post-LayerNorm is back: Stable, ExpressivE, and deep.arXiv preprint arXiv:2601.19895,

    Chen Chen and Lai Wei. Post-LayerNorm is back: Stable, ExpressivE, and deep.arXiv preprint arXiv:2601.19895,

    arXiv

  27. [2025]

    On residual network depth.arXiv preprint arXiv:2510.03470,

    Benoit Dherin and Michael Munn. On residual network depth.arXiv preprint arXiv:2510.03470,

    arXiv

  28. [2026]

    The depth delusion: Why transformers should be wider, not deeper.arXiv preprint arXiv:2601.20994,

    Md Muhtasim Munif Fahim and Md Rezaul Karim. The depth delusion: Why transformers should be wider, not deeper.arXiv preprint arXiv:2601.20994,

    arXiv