Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

SpanNorm claims that rewiring one residual connection in a PostNorm Transformer block—making it skip the whole block—delivers PreNorm-style training stability and PostNorm-style performance at once.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:31 UTC pith:HTDI2FY7

load-bearing objection A simple, plausible residual redesign with strong reported pretraining gains, but the theory overclaims and the submission needs code, seeds, and a cleanup before it can be fully trusted. the 4 major comments →

arxiv 2601.22580 v2 pith:HTDI2FY7 submitted 2026-01-30 cs.CL cs.AIcs.LG

SpanNorm: Reconciling Training Stability and Performance in Deep Transformers

Chao Wang , Bei Li , Jiaqi Zhang , Xinyu Liu , Yuchun Fan , Linkun Lyu , Xin Chen , Jingang Wang
show 3 more authors
Tong Xiao Peng Pei Xunliang Cai
This is my paper
classification cs.CL cs.AIcs.LG
keywords Transformer normalizationPreNorm vs PostNormresidual connectionstraining stabilitygradient vanishingdepth scalingMixture-of-Expertsinitialization scaling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the PreNorm/PostNorm dilemma comes from where the residual path connects, not from normalization placement alone. Its SpanNorm block keeps PostNorm-style computation, normalizing aggregated sub-layer outputs, but changes the FFN residual so it adds the block input instead of the attention output. That spanning skip gives backpropagation a clean path through the entire block, so gradient decay becomes linear instead of quadratic per layer, while the final normalization keeps block output variance bounded. Paired with output-matrix initialization scaled as O(1/√L), SpanNorm is reported to train stably at 128 layers and 6.5B parameters and to outperform PreNorm and several hybrid normalizations in dense and MoE settings, with no extra compute.

Core claim

SpanNorm keeps PostNorm computation but rewires one residual: Y_l = LN(MHA(X'_l)+X'_l), X'_{l+1} = LN(FFN(Y_l)+X'_l), so the FFN residual adds the block input, not the attention output. The paper claims this gets both benefits at once: gradient magnitude decays as σ^{-L} per layer, the square root of PostNorm's σ^{-2L} (Eq. 11), and the block-end normalization resets variance to Θ(1), preventing the linear variance growth that sends deep PreNorm Jacobians to identity. A depth-scaling theorem requires Var(F(LN(x)))=O(1/L), achieved by initializing the FFN output and attention output projections at O(1/√L). Empirically, SpanNorm converges at 128 layers (6.5B) where PostNorm and HybridNorm fail

What carries the argument

The carrying mechanism is the block-spanning residual: the second sub-layer adds the pre-block input X'_l instead of the intermediate representation Y_l, while LayerNorm is applied after aggregation as in PostNorm. This single topological change does two jobs: it gives backpropagation a skip path through the entire block, so only one normalization factor sits on the macro gradient path, and it resets feature variance to unit scale at every block output. The auxiliary machinery is depth-scaled initialization: setting the standard deviation of the FFN output projection and the attention output projection to O(1/√L) makes the residual branch variance O(1/L), which the paper's Theorem 4.1 identi

Load-bearing premise

Everything hinges on Assumption 3.1's claim that pre-normalized sums in every sub-layer share a consistent standard deviation σ > 1 across depth; the square-root gradient advantage (Eq. 11) is derived by counting one normalization on the macro path per block, but the paper's own Appendix A bound (Eq. 18) has two LN factors per block, so if σ varies with depth or the inner normalization factor dominates, the claimed σ^{-L} vs σ^{-2L} separation and the ~500x comparison do not

What would settle it

Measure the per-layer standard deviations σ_Zl and σ_Al of pre-normalized sums during a 128-layer SpanNorm run and compare the gradient norm ratio to PostNorm under identical initialization and learning rate; if the ratio does not track √(∥G_Post∥), or if σ_Zl decays with depth while σ_Al stays large, Eq. 11's mechanism is not what produces stability. A cheaper check: train the 512-layer proxy without the O(1/√L) output-matrix scaling; the paper's own Figure 14 predicts catastrophic collapse, so stable convergence without Scale Init would contradict Theorem 4.1's necessity claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • PostNorm's instability can be removed without sacrificing its accuracy: SpanNorm trains at 128 layers/6.5B where PostNorm and HybridNorm diverge.
  • Deep PreNorm's representation collapse is alleviated: layer-pair cosine similarity drops from above 0.5 to about 0.25 with distance, and the Effective Dimension Ratio more than doubles (30.66 vs 12.99).
  • Gradient magnitudes follow a square-root law: for equal per-layer scaling σ, SpanNorm's gradient is σ^{-L} against PostNorm's σ^{-2L}, a roughly 500x advantage at 128 layers with σ≈1.05.
  • Scale Init enables deep training under a fixed learning rate: proxy models from 32 to 512 layers show monotonically decreasing loss without per-depth learning-rate tuning.
  • SpanNorm is a drop-in replacement: it reports identical throughput to PreNorm and adds no parameters or FLOPS.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: If the gradient-separation claim is right, the explanation is architectural skip count, not loss landscape; the same spanning-residual trick might stabilize other post-norm-style deep networks such as diffusion or recurrent stacks that suffer from quadratic per-block attenuation.
  • Inference: The paper leaves open whether the homogeneous-variance assumption holds during real training; directly measuring σ_Zl and σ_Al per layer would tell whether SpanNorm's advantage is the claimed square-root relation or an empirical side effect of the variance reset.
  • Inference: Because the block output is normalized while the identity path stays unnormalized, SpanNorm could combine naturally with normalization-free or learnable-scale alternatives; a testable variant replaces the final LayerNorm with a learnable scale and offset.
  • Inference: The monotonic 32→512-layer loss curve suggests SpanNorm might be pushed past 1,000 layers; a cheap falsifiable check is a 1k-layer proxy with the same fixed learning rate and Scale Init.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. SpanNorm is a Transformer block variant in which the FFN residual connection is taken from the block input rather than from the MHA output: X'_{l+1} = LN(FFN(Y_l) + X'_l) while Y_l = LN(MHA(X'_l) + X'_l) (Eqs. 5-6). The paper argues that this preserves the PostNorm computational style while creating a clean path through the block, and supports this with (i) a gradient-scaling analysis in Section 3.3 claiming that SpanNorm's gradient decay is the square root of PostNorm's, (ii) an initialization recipe in Section 4 that scales W2 and WO by O(1/sqrt(L)), and (iii) experiments on dense and MoE models up to 128 layers / 6.5B parameters plus a 512-layer proxy stress test. The reported results show consistent gains over PreNorm and over several recent normalization hybrids.

Significance. If the empirical claims are reproducible, the contribution is practically useful: SpanNorm is a simple change with no additional compute, and the reported stability at 128 layers and monotonic 'deeper is better' trend in the 512-layer proxy are striking. The paper's strengths are the breadth of the evaluation across dense and MoE architectures and the explicit analysis of representation collapse via cosine similarity and spectral metrics. However, the paper does not provide code, benchmark tables have no error bars or seeds, and the main theoretical result in Section 3.3 is a heuristic that is not fully consistent with the paper's own Jacobian bound in Appendix A. The empirical direction is believable, but the explanatory claims need to be either rigorously established or explicitly downgraded.

major comments (4)
  1. [Section 3.3, Eq. (11)] The central theoretical result, ||G_Span(L)|| ∝ sqrt(||G_Post(L)||), is derived by counting one LayerNorm factor per SpanNorm block versus two for PostNorm. But the Jacobian bound in Appendix A, Eq. (18), contains two LN factors per block: the bound is (1/σ_Zl)(1 + ||W2W1||_2(1 + ||W_O J_A'||_2)/σ_Al). Unless the second term in parentheses is shown to be O(1) at the operating point, the σ^{-L} scaling attributed to the topology is not established. Moreover, Assumption 3.1 posits a constant σ > 1, while Theorem 4.1's own requirement σ_Zl = 1 + O(1/L) implies σ ≈ 1 + O(1/L); the numerical illustration with σ ≈ 1.05 at L=128 (1.05^128 ≈ 500×) is therefore not supported by the scaling recipe. This does not directly disprove the empirical stability results, but it means the 'architectural topology, not initialization' conclusion is not actually proven. Please either prove a bound on the resid
  2. [Section 4 / Appendix A, Theorem 4.1] The proof of Theorem 4.1 relies on at least three unstated approximations: the LayerNorm Jacobian is replaced by the scalar 1/σ; the FFN Jacobian is approximated by ||W1W2||_2 under an identity-activation assumption; and Eq. (19) assumes that Var(F(Y_l)) and the identity path are uncorrelated so that variances add. None of these is listed as an assumption in the theorem statement. In addition, the condition Var(F(Y_l)) = O(1/L) is imposed to make σ_Zl = 1 + O(1/L), not derived from the forward dynamics. The resulting O(1/sqrt(L)) initialization is consistent with Megatron's practice, but calling the recipe 'theoretically justified' overstates the result. Please state the assumptions explicitly and add an empirical check at initialization that Var(F(Y_l)) indeed behaves as O(1/L).
  3. [Appendix B / Table 2] The baseline comparison in Table 2 is not equally tuned. According to Appendix B, LayerNorm Scale and Mix-LN are trained at the standard learning rate 2e-4, while HybridNorm and Peri-LN receive a tuned learning rate of 1e-4. LayerNorm Scale, whose depth-dependent scaling is exactly the mechanism the paper uses to explain its poor result, is not given the same tuning budget. In addition, Tables 1-3 report only single runs with no seed information; the only multi-seed study shown is the gradient-norm analysis in Figure 14. For the central claim that SpanNorm outperforms advanced hybrids by 1-4 average points, the lack of error bars or a sensitivity analysis is a significant gap. Please run the comparisons with multiple seeds (or at least report a hyperparameter sensitivity range) and give all baselines the same per-method tuning effort.
  4. [Tables and duplicated text] The manuscript contains duplicated table/figure blocks from a previous draft that use a different model name, 'FuseNorm', and report different numbers. For example, the duplicated 'Table 1' block after Figure 2 lists 'FuseNorm 5B' with Wiki. ppl 11.7 and Avg. 65.9, while the main Table 1 lists 'SpanNorm 5B' with Wiki. ppl 11.8 and Avg. 66.6. This is not merely a typo: a reader cannot tell which numbers are authoritative. All duplicated blocks must be removed and the tables reconciled before the paper can be considered publishable.
minor comments (5)
  1. [Throughout] The model name is inconsistent: 'SpanNorm' is used in the abstract and main text, but 'FuseNorm' appears in the duplicated figure captions (Figures 2-3) and the embedded Table 1. Please unify the terminology.
  2. [Algorithm 1 / Eqs. (7)-(8)] The pseudocode in Algorithm 1 does not show the first-layer special case with the additional LN(E) applied to the embedding before MHA. Please make the pseudocode consistent with Eqs. (7)-(8).
  3. [Figure 14 caption] The caption mentions '3 random seeds' and shows faint individual runs, but the main benchmark tables never state whether they are multi-seed or single-run. Please state this explicitly for each table.
  4. [Appendix B, Mix-LN description] The text refers to 'a configuration of 16 PostNorm layers and 48 PreNorm layers' and to 'PostNorm ratio α=25%' in Section 5. The quantity α is not defined in the main text; please define it and make the two statements consistent.
  5. [Table 4 / Section D.2] The width-scaling ablation shows that SpanNorm without width scaling outperforms SpanNorm with width scaling; the explanation attributes this to a coarse search grid. This is speculative; please either provide supporting evidence (e.g., a finer-grid result) or temper the interpretation.

Circularity Check

0 steps flagged

No load-bearing circularity: SpanNorm's stability and performance claims are judged against external baselines, and its theoretical gradient comparison follows from a stated heuristic assumption rather than from fitted data or a self-citation chain.

full rationale

The central claim — that SpanNorm reconciles PostNorm performance with PreNorm-like stability — is supported by experiments against external baselines (Table 1-3) under shared hyperparameters. No reported number is recovered by fitting a parameter to baseline data and then renaming it a prediction. The gradient comparison (Eq. 10-11) is a direct consequence of Assumption 3.1 (a homogeneous σ > 1, with one LN per SpanNorm macro-path vs. two for PostNorm); it is a stated modeling assumption, not an output derived from the empirical results. The initialization recommendation (Theorem 4.1, Eqs. 12 and 20) is derived from the Appendix A Jacobian bound and then applied uniformly to all models, including baselines, so it is not a fitted input disguised as a prediction. The paper cites prior works by its own authors (e.g., Wang et al. 2019; Li et al. 2020), but these are used only for background claims about PreNorm/PostNorm behavior and depth degradation; none of the citations forces the SpanNorm architecture or its evaluation. There is a genuine rigor concern: Assumption 3.1's σ > 1 is inconsistent with Theorem 4.1's own condition σ_Zl = 1 + O(1/L), and Appendix A Eq. 18 contains an additional inner-LN factor, so the illustrative 500× advantage (1.05^128) is not established. This is a correctness/assumption-validity weakness, not a circularity: the empirical stability findings are externally benchmarked and falsifiable. The score of 2 reflects only the minor, non-load-bearing presence of self-citations.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The architecture introduces no new learnable parameters or invented physical quantities; the load-bearing extras are the modeling assumptions above. The only hand-chosen design constants are the O(1/√L) initialization (borrowed from Megatron) and the special first-layer input LN; neither is fitted to the benchmark data.

axioms (5)
  • ad hoc to paper Pre-normalized sub-layer sums have homogeneous standard deviation σ > 1 across layers (Assumption 3.1).
    Used in §3.3 Eq. 10-11 to derive σ^{-2L} vs σ^{-L} gradient scaling; no empirical or theoretical justification that σ is constant and >1 across depth.
  • domain assumption LayerNorm's Jacobian spectral norm is approximately the inverse standard deviation of its input.
    Used throughout Appendix A (Eq. 15-18); a standard finite-width heuristic, not exact for attention/softmax and not proven here.
  • domain assumption At initialization the residual branch F(Y_l) and identity path X'_l are uncorrelated, so Var(Z_l)=1+Var(F(Y_l)).
    Used in Appendix A Eq. 19 to obtain Var(F(Y_l))=O(1/L); if correlated, the variance condition changes and Theorem 4.1's sufficiency is not established.
  • ad hoc to paper FFN Jacobian norm ≈ ||W1W2|| under identity activation.
    Appendix A, line after Eq. 17; an approximation that ignores nonlinear activations and attention/softmax contributions in the spectral bound.
  • domain assumption Stable training requires cumulative spectral norm of per-layer Jacobians O(1), and Var(F)=O(1/L) suffices.
    Theorem 4.1 and Appendix A; standard signal propagation criterion but not sufficient for non-linear training dynamics.

pith-pipeline@v1.3.0-alltime-deepseek · 25717 in / 16467 out tokens · 168214 ms · 2026-08-03T06:31:07.085782+00:00 · methodology

0 comments
read the original abstract

The success of Large Language Models (LLMs) hinges on the stable training of deep Transformer architectures. A critical design choice is the placement of normalization layers, leading to a fundamental trade-off: the ``PreNorm'' architecture ensures training stability at the cost of potential performance degradation in deep models, while the ``PostNorm'' architecture offers strong performance but suffers from severe training instability. In this work, we propose SpanNorm, a novel technique designed to resolve this dilemma by integrating the strengths of both paradigms. Structurally, SpanNorm establishes a clean residual connection that spans the entire transformer block to stabilize signal propagation, while employing a PostNorm-style computation that normalizes the aggregated output to enhance model performance. We provide a theoretical analysis demonstrating that SpanNorm, combined with a principled scaling strategy, maintains bounded signal variance throughout the network, preventing the gradient issues that plague PostNorm models, and also alleviating the representation collapse of PreNorm. Empirically, SpanNorm consistently outperforms standard normalization schemes in both dense and Mixture-of-Experts (MoE) scenarios, paving the way for more powerful and stable Transformer architectures.

Figures

Figures reproduced from arXiv: 2601.22580 by Bei Li, Chao Wang, Jiaqi Zhang, Jingang Wang, Linkun Lyu, Peng Pei, Tong Xiao, Xin Chen, Xinyu Liu, Xunliang Cai, Yuchun Fan.

Figure 1
Figure 1. Figure 1: Comparisons of our proposed SpanNorm with PostNorm, PreNorm, and other advanced variants. Here, we take the dense model as an instance, and MHA denotes the multi-head attention, FFN denotes the feedforward network. Note that MHA can also be replaced by GQA, MLA and other attention variants. When switching to the MoE models, FFN could be replaced by MoE modules. proach, employing a QKV-Norm (where normaliza… view at source ↗
Figure 2
Figure 2. Figure 2: Early-stage training stability analysis. We train dense models with a fixed hidden dimension d = 1536 across increasing depths (12, 24, 48 layers) for 5000 steps to evaluate the impact of initialization on stability. by modifying the information flow in the second residual connection of a PostNorm block. Given the input to the block X′ l , the computation proceeds as follows: Yl = LN(MHA(X ′ l ) + X ′ l ) … view at source ↗
Figure 3
Figure 3. Figure 3: Gradient norm dynamics (Analysis of the 24-layer model in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance and scalability across scales. (Left) SpanNorm consistently outperforms baselines on a 64-laye 5B dense model (200B tokens). (Middle & Right) On an ultra-deep 128-layer 6.5B model (400B tokens), SpanNorm achieves superior performance with robust convergence, whereas PostNorm and HybridNorm suffer from catastrophic divergence in the early phase (Right). variants from [PITH_FULL_IMAGE:figures/fu… view at source ↗
Figure 5
Figure 5. Figure 5: Cosine similarity between the outputs of each pair of layers for MoE-A2.4B-16B model. 2 4 6 8 10 12 14 16 18 20 22 24 26 0.2 0.4 0.6 0.8 1.0 Layer Difference (|Li − Lj |) Average Similarity PreNorm SpanNorm [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of FFN weight gradient norms between our SpanNorm model and standard PostNorm using the 740M model configuration (detailed in Appendix B) across three learning rate conditions. (Left and Middle) At a high learning rate (> 8 × 10−5 ), PostNorm training collapses, showing near-zero gradients in all but the final layer, while SpanNorm remains stable. (Right) At a lower learning rate (8 × 10−5 ), Po… view at source ↗
Figure 8
Figure 8. Figure 8: SpanNorm achieves superior spectral utilization compared to competitive normalization baselines. We evaluate the layer-wise dynamics (left) and average performance (right) of PreNorm, Mix-LN, HybridNorm, and SpanNorm across three spectral metrics. (Top) Hard Spectral Rank: SpanNorm demonstrates the strongest ability to preserve dominant eigen-directions, achieving the highest average rank of 0.23, signific… view at source ↗
Figure 9
Figure 9. Figure 9: Eigenspectrum Analysis of Input Embeddings. We analyze the distribution of sorted eigenvalues normalized by their median value (log scale). While Mix-LN (teal) mitigates the extreme outliers seen in PreNorm, we observe that it suffers from a rapid decay in the tail spectrum, indicating potential dimensional collapse. In contrast, we demonstrate that SpanNorm (red) maintains a significantly flatter and more… view at source ↗
Figure 10
Figure 10. Figure 10: Layer-wise condition number analysis of FFN weights. We analyze the spectral properties of the MLP1 (up-projection) and MLP2 (down-projection) matrices across the network depth. SpanNorm demonstrates superior stability, maintaining significantly lower condition numbers, particularly in the deeper layers, which effectively prevents spectral degradation. In contrast, PreNorm exhibits elevated condition numb… view at source ↗
Figure 11
Figure 11. Figure 11: Stress-testing “Deeper is Better”. We train proxy models (d = 384) with depths scaling exponentially from 32 to 512 using a fixed learning rate. SpanNorm shows a strict monotonic decrease in training loss, confirming robust stability and the effective avoidance of depth degradation at extreme scales. D.2. Width Scaling and Predictability While we prioritize depth scaling for structural stability, handling… view at source ↗
Figure 12
Figure 12. Figure 12: Validation of width scaling stability transfer. We compare a 310M proxy (hidden dimension d = 640) and a 5B target (d = 2560) under a fixed depth (L = 64). As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Training throughput record of the 5B model experiments. The plot compares the real-time throughput (tokens/s) of SpanNorm and PreNorm. The vertical dashed line marks a Data Parallelism (DP) scale-up event at 21.5B tokens, resulting in a step increase in throughput. SpanNorm maintains identical speed to the baseline both before and after the scaling, demonstrating consistent efficiency. E. Computational Ef… view at source ↗
Figure 14
Figure 14. Figure 14: Comprehensive layer-wise gradient norm dynamics comparison (PreNorm vs. SpanNorm), which extends the original analysis of [PITH_FULL_IMAGE:figures/full_fig_p019_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth

    cs.LG 2026-07 conditional novelty 7.0

    Rank survival in Transformer blocks is governed by a branch-to-skip ratio law (βα^M√L), a mean-spike coherence c_ℓ=E[σ]²/E[σ²], and a Marchenko–Pastur width threshold m/d=1/p(σ).

  2. Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning

    cs.LG 2026-07 conditional novelty 5.0

    Applying mHC as a PEFT method shows that learned residual mixing is unnecessary — even harmful — in finetuning, and mHC+LoRA combinations give small task-dependent gains.

Reference graph

Works this paper leans on

24 extracted references · 9 linked inside Pith · cited by 2 Pith papers

  1. [1]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

    1901

  2. [3]

    Jha, N. K. and Reagen, B. Spectral scaling laws in lan- guage models: emphhow effectively do feed-forward net- works use their latent space? InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 35047–35058,

    2025

  3. [4]

    This isolates the impact of the architecture from the initialization scheme

    as our default strategy for all models, including the baselines (PostNorm, PreNorm, HybridNorm, etc.). This isolates the impact of the architecture from the initialization scheme. To ensure a fair and rigorous comparison in the 5B dense model experiments in Table 2, we carefully calibrated the configurations for each model: • Standard Learning Rate ( 2×10...

    2025

  4. [8]

    Li, P., Yin, L., and Liu, S

    Association for Computational Linguistics. Li, P., Yin, L., and Liu, S. Mix-ln: Unleashing the power of deeper layers by combining pre-ln and post-ln. InThe Thirteenth International Conference on Learning Rep- resentations, ICLR 2025, Singapore, April 24-28,

    2025

  5. [9]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Pith/arXiv arXiv

  6. [12]

    net/forum?id=Orpf8yDjdj

    URL https://openreview. net/forum?id=Orpf8yDjdj. Takase, S., Kiyono, S., Kobayashi, S., and Suzuki, J. Spike no more: Stabilizing the pre-training of large language models.arXiv preprint arXiv:2312.16903,

    Pith/arXiv arXiv

  7. [13]

    G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ram ´e, A., et al

    Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ram ´e, A., et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

    Pith/arXiv arXiv

  8. [14]

    L., Li, B., Lei, B., Wang, B., Rong, B., Wang, C., Zhang, C., Gao, C., Zhang, C., Sun, C., et al

    Team, M. L., Li, B., Lei, B., Wang, B., Rong, B., Wang, C., Zhang, C., Gao, C., Zhang, C., Sun, C., et al. Longcat- flash technical report.arXiv preprint arXiv:2509.01322,

    arXiv

  9. [15]

    Llama: Open and efficient foundation language models.CoRR, abs/2302.13971,

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozi `ere, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models.CoRR, abs/2302.13971,

    Pith/arXiv arXiv

  10. [16]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  11. [17]

    Improving deep transformer with depth-scaled initialization and merged attention

    Zhang, B., Titov, I., and Sennrich, R. Improving deep transformer with depth-scaled initialization and merged attention. In Inui, K., Jiang, J., Ng, V ., and Wan, X. (eds.),Proceedings of the 2019 Conference on Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 8...

    2019

  12. [18]

    10 SpanNorm: Reconciling Training Stability and Performance in Deep Transformers A

    URL https: //openreview.net/forum?id=NligLHO7yG. 10 SpanNorm: Reconciling Training Stability and Performance in Deep Transformers A. Proof of Theorem 1 Proof.The forward pass for thel-th layer of our SpanNorm architecture is given by: Yl =LN(MHA(X ′ l) +X ′ l)(13) X′ l+1 =LN(FFN(Y l) +X ′ l)(14) The stability of backpropagation depends on the spectral nor...

    2023

  13. [19]

    The input to the FFN, Yl, is similarly normalized. Under the standard initialization assumption that the residual branch F(Y l) and the identity pathX ′ l are uncorrelated, the variance of their sum is the sum of their variances: Var(Zl) =Var(X ′ l) +Var(F(Y l)) = 1 +Var(F(Y l))(19) For the condition σZl = 1 +O(1/L) to hold, the variance must be Var(Zl) =...

    2025

  14. [22]

    In contrast, SpanNorm demonstrates superior representational capability

    We can observe that PreNorm suffers from severe spectral collapse, exhibiting the lowest utilization across all metrics (e.g., an average normalized HardRank of only 0.09). In contrast, SpanNorm demonstrates superior representational capability. For the dominant modes, SpanNorm achieves the highest average Hard Rank of 0.23, significantly outperforming th...

    2025

  15. [24]

    All models are trained on the same subset of the SlimPajama dataset (from 30B to 200B) with the Mistral tokenizer Jiang et al

    on various configurations. All models are trained on the same subset of the SlimPajama dataset (from 30B to 200B) with the Mistral tokenizer Jiang et al. (2023). The last column shows the average over all benchmarks that use (normalized) accuracy as the metric. Model Param Tokens Wiki. LMB. LMB. PIQA Hella. SciQ ARC-c Wino. Avg. ppl↓ppl↓ acc↑acc↑acc norm↑...

    2023

  16. [25]

    • Stability of Scale Init: When paired with our proposed Scale Init (Blue lines in bottom rows), SpanNorm effectively 17 SpanNorm: Reconciling Training Stability and Performance in Deep Transformers maintains healthy gradient magnitudes ( ≈10 −1) across all depths, matching the stability profile of the PreNorm baseline. 18 SpanNorm: Reconciling Training S...

    2000

  17. [64]

    It employs an MLA architecture, activating 6 out of 64 experts

    For the MoE model, we use the Deepseek-V3 small-scale model (Liu et al., 2024), activated by 2.4 billion parameters within approximately 16 billion parameters in total. It employs an MLA architecture, activating 6 out of 64 experts. The model consists of a total of 27 layers, with a hidden dimension of

    2024

  18. [384]

    SpanNorm shows a strict monotonic decrease in training loss, confirming robust stability and the effective avoidance of depth degradation at extreme scales

    with depths scaling exponentially from 32 to 512 using a fixed learning rate. SpanNorm shows a strict monotonic decrease in training loss, confirming robust stability and the effective avoidance of depth degradation at extreme scales. D.2. Width Scaling and Predictability While we prioritize depth scaling for structural stability, handling width scaling i...

    2024

  19. [2016]

    Shallow-to-deep training for neural ma- chine translation

    Li, B., Wang, Z., Liu, H., Jiang, Y ., Du, Q., Xiao, T., Wang, H., and Zhu, J. Shallow-to-deep training for neural ma- chine translation. In Webber, B., Cohn, T., He, Y ., and Liu, Y . (eds.),Proceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing (EMNLP), pp. 995–1005, Online, November

    2020

  20. [2020]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7,

    2021

  21. [2021]

    Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B

    URL https://arxiv.org/ abs/2110.09456. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi- billion parameter language models using model paral- lelism.arXiv preprint arXiv:1909.08053,

    Pith/arXiv arXiv 1909

  22. [2023]

    A., Khyalia, S., Jung, J., Goka, H., and Lee, H

    Kedia, A., Zaidi, M. A., Khyalia, S., Jung, J., Goka, H., and Lee, H. Transformers get stable: An end-to-end signal propagation theory for language models.arXiv preprint arXiv:2403.09635,

    Pith/arXiv arXiv

  23. [2024]

    Kim, J., Lee, B., Park, C., Oh, Y ., Kim, B., Yoo, T., Shin, S., Han, D., Shin, J., and Yoo, K. M. Peri-ln: Revisiting normalization layer in the transformer architecture.arXiv preprint arXiv:2502.02732,

    Pith/arXiv arXiv

  24. [2025]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de Las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b.CoRR, abs/2310.06825,

    Pith/arXiv arXiv