Pith. sign in

REVIEW 4 major objections 4 minor 4 cited by

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture

T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By normalizing both the input and the output of each sublayer, the Peri-LN design curbs the exponential hidden-state growth that destabilizes Pre-LN transformers, and the paper's experiments up to 3.2B parameters show it beats both Pre-LN…

desk verdict Honest, compute-heavy empirical study of output normalization that mostly delivers, with a theoretical bound that is real but loosely framed. read the letter →

arxiv 2502.02732 v3 pith:PHQ7GOGA submitted 2025-02-04 cs.LG cs.AIcs.CL

Jeonghoon Kim , Byeongchan Lee , Cheonbok Park , Yeontaek Oh , Beomjun Kim , Taehwan Yoo , Seongjin Shin , Dongyoon Han
show 2 more authors
Jinwoo Shin Kang Min Yoo
This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords layernormalizationplacementTransformerarchitecturemassiveactivationstrainingstabilityvarianceanalysislargelanguagemodelpretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard choice of where to put layer normalization in a Transformer is leaving training stability and final performance on the table. Its central claim is that placing normalization on both sides of each sublayer—a scheme the authors call Peri-LN, written $y_l = x_l + \mathrm{Norm}(\mathrm{Module}(\mathrm{Norm}(x_l)))$—controls the growth of hidden-state variance without sacrificing gradient flow, and that this placement outperforms Pre-LN and Post-LN consistently across 400M, 1.5B, and 3.2B parameter models trained on 30 billion tokens. A sympathetic reader would care because the paper identifies a concrete mechanism: Pre-LN, the default in many open models, lets 'massive activations' grow exponentially once training starts, causing gradient spikes and divergences, whereas Peri-LN keeps the residual path well-conditioned. The paper backs the mechanism with a variance recurrence, a gradient-norm bound, and seed-averaged benchmarks.

What carries the argument

The carrying object is the Peri-LN layer formula $y_l = x_l + \mathrm{Norm}(\mathrm{Module}(\mathrm{Norm}(x_l)))$ (Eq. 3), i.e., layer normalization both before and after each attention or MLP sublayer, with the residual added outside the output normalization. Two formal pieces do the work: the variance recurrence $\mathrm{Var}(x_{l+1}) \approx \mathrm{Var}(x_l) + \beta_0$ (Eq. 4), which converts exponential hidden-state growth into linear growth, and the gradient bound of Proposition 3.1, $\|\partial L/\partial W^{(2)}_{i,j}\| \le 4\gamma\sqrt{D}\,\|h\|/\|a\|$, which shows the output normalizer introduces a damping factor $\|a\|$ that keeps final-layer gradients bounded even when the hidden state $h$ is massive. The same bound shows why Pre-LN explodes: with no output normalizer, the gradient is proportional to $\|h\|$ with no denominator to absorb it.

What would settle it

Remove the output normalizer from a trained Peri-LN model and continue training under the same schedule; if the run keeps its stable loss and does not develop Pre-LN-style gradient spikes and variance growth, the output normalizer is not the active stabilizer.

Watch

Extended reading notes

Core claim

The central discovery this paper sets out to establish is that the third, largely unnamed normalization placement used by recent open-source transformers deserves to be a first-class design choice. In Peri-LN, each sublayer is wrapped by normalization on both ends: input normalization before the module and output normalization after it, followed by the residual addition, with embedding and final outputs normalized as well. The paper's analysis shows that Pre-LN's hidden-state variance, which grows only linearly at initialization, later escalates exponentially under training, because nothing damps large module outputs carried by the residual 'highway.' Peri-LN adds an output normalizer whose near-constant variance $\beta_0$ makes the per-layer variance recurrence $\mathrm{Var}(x_{l+1}) \approx \mathrm{Var}(x_l) + \beta_0$, yielding linear rather than exponential growth. Its gradient bound (Proposition 3.1) shows the output normalizer damps the final-layer gradient by a factor $\|a\|$, the norm of the module output, which Pre-LN lacks. Empirically, the paper reports that Peri-LN consistently achieves superior loss curves over the whole 400M to 3.2B range, fewer divergences across seeds, lower seed-to-seed benchmark variance, and hidden states that stay below the FP16 overflow threshold throughout training.

Load-bearing premise

The whole variance argument assumes that the output-normalized module $y_l = x_l + \mathrm{Norm}(\mathrm{Module}(\mathrm{Norm}(x_l)))$ keeps a near-constant output variance $\beta_0$ throughout training, so that hidden-state variance rises only linearly with depth; if parameter updates break that near-constant variance, the claimed difference between linear Peri-LN growth and exponential Pre-LN growth no longer holds.

Editorial extensions

If this is right

  • Pretraining runs that use Peri-LN in place of Pre-LN, with all other settings fixed, are less likely to diverge and need no extra learning-rate tuning to reach the same loss.
  • Peri-LN keeps the top hidden-state magnitudes below the FP16 maximum throughout a 3.2B-parameter run, so the design extends the useful lifetime of FP16 hardware and mixed-precision training.
  • Because benchmark variance across training seeds drops by more than half with Peri-LN, results from a single run are more reproducible.
  • The gradient bound implies that the output normalizer acts as a built-in stabilizer that protects the final layer even when earlier layers produce huge activations.
  • Recent open models that already place a normalizer at the sublayer output become instances of the same mechanism, giving a unified explanation of their stability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The variance-tracking diagnostic the paper uses could be repurposed as an early warning: monitoring final-layer hidden-state variance during the first training steps would flag a Pre-LN-style run that is about to diverge, before the gradient spike appears.
  • The gradient bound suggests output normalization should also reduce quantization outliers, since massive activations are the same values that strain low-precision formats; this connects the normalization-placement question to compression, a link the paper only gestures at.
  • A testable prediction follows: at larger scales than 3.2B, the gap between Pre-LN and Peri-LN should widen, because the exponential variance growth of Pre-LN compounds with depth while Peri-LN's linear growth does not.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies where to place layer normalization inside Transformer sublayers and identifies a third strategy, termed Peri-LN, which applies normalization both before and after each Attention/MLP module (Eq. 3). The authors claim that Peri-LN controls hidden-state variance growth, keeps gradient norms stable, and prevents the early-training divergence that can occur with Pre-LN. They support this with a theoretical analysis in §3.3–§3.4 and Appendix C, and with large-scale experiments on 400M, 1.5B, and 3.2B models trained on 30B tokens, reporting lower pre-training loss and higher benchmark scores than Pre-LN and Post-LN (Table 1). The paper also documents that several existing open models (Gemma, OLMo) already use this design, and it positions Peri-LN as a unifying term for that practice.

Significance. If the empirical claims hold, this is a practically useful study of a normalization design already used in production-scale open models, offering a systematic comparison that was previously missing. The experimental effort is substantial: three model sizes, five seeds, learning-rate sweeps, 30B-token pre-training, and ablations over weight decay, initialization variance, sequence length, warmup, LayerNorm vs RMSNorm, and OLMo2-style QK-Norm. The paper also gives credit where due by noting that Gemma and OLMo adopted this design earlier. However, the advertised 'comprehensive analytical foundation' is not delivered as stated: the main gradient bound is a ratio that is not uniformly bounded, and the benchmark comparison excludes diverged runs, which biases the averages in favor of the architectures that diverge.

major comments (4)
  1. [§3.4, Prop. 3.1(2); Appendix C.2, Eq. (21)] The proposition states that Peri-LN 'ensures that the gradient norm remains bounded,' but the bound proved in Eq. (21) is ||∂L/∂W^(2)|| ≤ 4γ√D ||h||/||a||. This is not bounded independent of activations: if ||h|| is large while ||a|| is moderate, the bound can be arbitrarily large. The 'damping factor' ||a|| only converts an absolute scale in ||h|| into a ratio; for the stated conclusion one would need a separate argument that ||a|| grows at least proportionally to ||h||, and none is provided. Because this proposition is invoked in §4.2, §5.3, §6.1, and §7.1 as the analytical explanation for Peri-LN's stability, the theoretical claim as written is an overstatement. The proposition should be rephrased as a relative bound (Peri-LN divides the Pre-LN gradient by ||a||) or supplemented with the missing condition that controls ||h||/||a||.
  2. [Table 1 and Appendix J.2.1] The caption of Table 1 says 'Diverged checkpoints are excluded from the evaluation score computation,' and Table 11 shows that two of the five Pre-LN 400M seeds and three of the five Pre-LN 3.2B seeds diverged. Excluding those runs from the average removes exactly the failures that the paper cites as Peri-LN's advantage ('often prevents divergence,' §7.1). As a result, the reported Pre-LN benchmark averages are optimistic and the comparison is not apples-to-apples. Please report the averages with diverged runs counted as failures, report the divergence rate for each configuration, or restrict the comparison to the subset of seeds that completed training in all architectures. This is load-bearing for the benchmark-superiority claim.
  3. [§3.4 and Appendix C] The formal propositions analyze a single MLP block whose output is followed directly by a softmax loss, not a multi-layer Transformer. The main text then uses these bounds to explain 'gradient spikes, and occasional divergences that arise in Pre-LN during large-scale pre-training' (§4.3) across the full stacked architecture. The transfer from the simplified one-block setting to a deep Transformer is asserted rather than derived. Please state this limitation explicitly in the main text and either provide a per-layer accumulation argument or present the bound as a heuristic motivation for the empirical results.
  4. [§3.3, Eq. (4)] The forward-pass argument claims that if Norm(Module(Norm(x_l))) has near-constant variance β0, then hidden-state variance grows approximately linearly with depth (Eq. 4). This near-constant-variance assumption is not derived and could fail after parameter updates; Figures 6 and 10 provide indirect empirical support, but the paper does not directly measure the per-block output variance over training. If the assumption fails, the claimed distinction between linear (Peri-LN) and exponential (Pre-LN) variance growth loses its theoretical basis. Please state this as an empirical regularity and provide a direct measurement of Norm(Module(Norm(x_l))) variance across depth and training steps.
minor comments (4)
  1. [Figure 3(a) caption] The caption contains a typo: 'llustrates' should be 'illustrates.'
  2. [§4.2] There is a missing space in 'learning rate2 × 10−3'; it should read 'learning rate 2 × 10−3.'
  3. [§3.2] The sentence beginning 'However, Recent studies have also reported...' has a capitalization inconsistency; 'Recent' should not be capitalized mid-sentence.
  4. [Table 1 and Appendix J.1] Some numeric entries in Table 1 appear with irregular spacing (e.g., '7 .46' and '3 .43'); please normalize the formatting for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical bounds are derived from first principles and the empirical claims are validated on external benchmarks.

full rationale

The paper's derivation chain is self-contained. Peri-LN is defined by Eq. (3), and the variance recursion in Eq. (4) follows directly from that definition under the stated assumption that the normalized module output has near-constant variance; it is not a fitted quantity or a renamed prediction. Proposition 3.1 and its proof in Appendix C are explicit chain-rule bounds, with no parameter fitted to the data and no conclusion imported from a self-citation. The empirical comparisons use external datasets (DCLM, C4, ARC, HellaSwag, PIQA, SIQA, Winogrande) and multiple seeds, so the central claim that Peri-LN improves stability and benchmark performance is not forced by construction. The paper does cite prior work by its own authors (e.g., HyperCLOVA X, quantization papers), but these citations are background context and are not load-bearing for the variance or gradient analysis. The only notable concern is that the informal phrase 'ensures that the gradient norm remains bounded' in Proposition 3.1(2) overstates the formal inequality in Eq. (8), which scales with ||h||/||a||; however, this is a correctness or interpretation issue, not circularity, because the bound is an honest derivation rather than an input disguised as an output.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim relies on standard calculus and norm inequalities plus three domain assumptions: the simplified one-block MLP model stands in for full Transformers, module outputs after normalization have near-constant variance (Eq. 4), and the RMSNorm scale γ stays positive. None of these are fitted to data, and the paper does not introduce new entities. The main risk is the first assumption, which is acknowledged in Appendix C.

assumptions (4)
  • domain assumption A single MLP block with ReLU and RMSNorm (Appendix C) represents the gradient behavior of the full Attention+MLP Transformer sublayer.
    Section 3.4 and Appendix C derive the gradient bounds for a simplified sequence (RMSNorm, ReLU MLP, residual add), then use them to explain training stability of full Transformers, including attention blocks.
  • domain assumption During training, Norm(Module(Norm(x_l))) has near-constant variance β0, so Peri-LN's hidden-state variance grows additively with depth (Eq. 4).
    Section 3.3 builds the linear-vs-exponential variance argument on this assumption, but it is asserted rather than proved from network weights.
  • domain assumption RMSNorm's learnable scale γ remains strictly positive throughout training.
    Appendix C states this is empirically verified, and the bound's positivity depends on it.
  • standard math Standard chain rule, softmax derivative, and norm inequalities in Appendix C are accepted without proof.
    Routine calculus used in Prop C.1-C.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Peri-LN: Revisiting Normalization Layer in the Transformer Architecture." pith.science (2026). https://pith.science/paper/PHQ7GOGA

@misc{pith2026250202732,
  author       = {Pith},
  title        = {Pith review of: Peri-LN: Revisiting Normalization Layer in the Transformer Architecture},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PHQ7GOGA}},
  note         = {Machine review of arXiv:2502.02732}
}
abstract

Selecting a layer normalization (LN) strategy that stabilizes training and speeds convergence in Transformers remains difficult, even for today's large language models (LLM). We present a comprehensive analytical foundation for understanding how different LN strategies influence training dynamics in large-scale Transformers. Until recently, Pre-LN and Post-LN have long dominated practices despite their limitations in large-scale training. However, several open-source models have recently begun silently adopting a third strategy without much explanation. This strategy places normalization layer peripherally around sublayers, a design we term Peri-LN. While Peri-LN has demonstrated promising performance, its precise mechanisms and benefits remain almost unexplored. Our in-depth analysis delineates the distinct behaviors of LN strategies, showing how each placement shapes activation variance and gradient propagation. To validate our theoretical insight, we conduct extensive experiments on Transformers up to $3.2$B parameters, showing that Peri-LN consistently achieves more balanced variance growth, steadier gradient flow, and convergence stability. Our results suggest that Peri-LN warrants broader consideration for large-scale Transformer architectures, providing renewed insights into the optimal placement of LN.

Figures

Figures reproduced from arXiv: 2502.02732 by the authors.

Figure 1
Figure 1. Illustration of hidden-state variance across different model depths and training iterations. From initialization through training on 6.3 billion tokens, we observe the growth in hidden￾state variance for both Pre-LN and Post-LN architectures. The analysis is based on a 1.5B-parameter model. Detailed settings and additional results are provided in Section 5.1. not persist as training progresses as shown in [PITH_FUL… view at source ↗
Figure 2
Figure 2. Placement of normalization in Transformer sub-layer. studies have also reported large output magnitudes in both the pre-trained Attention and MLP modules (Dehghani et al., 2023; Wortsman et al., 2024; Fishman et al., 2024). To bridge the gap from initialization to the fully trained stage, we extend our empirical observations in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison of Post-, Pre-, and Peri-LN Transformers during pre-training [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (36 more)
Figure 4
Figure 4. Figure 4: Common case of early stage instability in pre-training. In most of our experiments across different random seeds, the Pre-LN architecture exhibited early-stage instability. Although we initially suspected that a high learning rate might be the root cause, lowering it d…
Figure 5
Figure 5. Figure 5: Final-layer gradient norms for seeds 4 and 5. 4.3. Early Stage Instability in Pre-Training Early in pre-training, Pre-LN models consistently show gra￾dient spikes, loss surges, and occasional divergence across seeds and scales ( [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Forward hidden state growth patterns for each LN strategy in a 1.5B-parameter Transformer. (a) Grad-norm at init. (b) Grad-norm at final (c) Grad-variance at init. (d) Grad-variance at final [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Backward gradient norm and variance of 1.5B Post-, Pre-, and Peri-LN Transformers at initialization (init.) and final training. Wikitext dataset (Merity et al., 2016) [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Freezing learnable parameter γ of output normalization layer in Peri-LN. we set γ to its initial value of 1 and keep it fixed. (a) After 30B tokens training (b) Learnable scale γ in Output-LN [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Angular distance between hidden states after training [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Effects of weight decay and initialization (init.) on massive activations. 10(a) Strong weight decay relieves the variance explosion in Pre-LN. 10(b) Strong weight decay (0.33, which is 10× the baseline.) suppresses Pre-LN divergence. 10(c) Smaller-scale initializatio…
Figure 11
Figure 11. Figure 11: Evolution of extreme hidden state absolute magnitudes during training. Colored bands trace the range of the global top￾100 absolute activations. Pre-LN (blue) quickly surpasses the FP16 representable maximum, while Peri-LN (red) stays below it. for Pre-LN surpasses th…
Figure 12
Figure 12. Figure 12: Learning rate explorations for Post-LN architectures. E.2. Best Performing Checkpoints Comparisons of Other Model Sizes As an extension of Section 4.2, we present below the results for additional model sizes that were omitted previously due to space constraints. In Fi…
Figure 13
Figure 13. Figure 13: Performance comparison of Post-LN, Pre-LN, and Peri-LN Transformers during pre-training for other two. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: The forward growth patterns of hidden state for different architectures highlight the structural impact of normalization placement. 3.2B size model. (a) Absolute magnitude growth (b) Variance growth [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: The forward growth patterns of hidden state for different architectures highlight the structural impact of normalization placement. 400M size model. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: , even as the number of training tokens increases, there is no observable shift in the optimal learning-rate range. Based on these findings, we conclude that our overall results remain consistent, even when the training token count is further increased. Furthermore, a…
Figure 17
Figure 17. Figure 17: Learning rate explorations of Pre-& Peri-LN architecture with sequence length 2048 configuration. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: , the overall trend remained unchanged, and Peri-LN continued to exhibit better performance than Pre-LN in terms of loss. Furthermore, we observed that increasing the warmup rate from 10% to 30% did not reduce the frequency of gradient norm spikes in Pre-LN [PITH_FUL…
Figure 19
Figure 19. Figure 19: LayerNorm vs. RMSNorm on Peri-LN architecture. 400M size model. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]
Figure 20
Figure 20. Figure 20: Loss and Gradient-norm curves comparing the presence and absence of Embedding LN in the Peri-LN architecture. 400M size model. (a) Pre-training loss curve (b) Gradient-norm curve [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]
Figure 21
Figure 21. Figure 21: Loss and Gradient-norm curves comparing the presence and absence of Embedding LN in the Peri-LN architecture. 1.5B size model. (a) Pre-training loss curve (b) Gradient-norm curve [PITH_FULL_IMAGE:figures/full_fig_p025_21.png]
Figure 22
Figure 22. Figure 22: Loss and Gradient-norm curves comparing the presence and absence of Embedding LN in the Peri-LN architecture. 3.2B size model. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_22.png]
Figure 23
Figure 23. Figure 23: Comparison of pre-training loss and gradient norms for Pre-LN and Peri-LN architectures with the weight decay coefficient fixed at 0, while all other hyperparameters are held constant. (a) Pre-training loss curve (b) Gradient-norm curve [PITH_FULL_IMAGE:figures/full_…
Figure 24
Figure 24. Figure 24: Comparison of pre-training loss and gradient norms for Pre-LN and Peri-LN architectures with the weight decay coefficient fixed at 0.0033, while all other hyperparameters are held constant. G.8.3. WEIGHT INITIALIZATION We run an additional ablation on weight-initializ…
Figure 25
Figure 25. Figure 25: Comparison of pre-training loss and gradient norms for Pre-LN and Peri-LN architectures with the weight decay coefficient fixed at 0.033, while all other hyperparameters are held constant. (a) Pre-training loss curve (b) Gradient-norm curve [PITH_FULL_IMAGE:figures/f…
Figure 26
Figure 26. Figure 26: Comparison of pre-training loss and gradient norms for Pre-LN and Peri-LN architectures with the weight decay coefficient fixed at 0.33, while all other hyperparameters are held constant. (1/d), and two scaled variants, 10/d and 1/(10d). As [PITH_FULL_IMAGE:figures/f…
Figure 27
Figure 27. Figure 27: Comparison of pre-training loss and gradient norms for Pre-LN and Peri-LN architectures with the weight initialization variance set to 10/d, while all other hyperparameters are held constant. (a) Pre-training loss curve (b) Gradient-norm curve [PITH_FULL_IMAGE:figure…
Figure 28
Figure 28. Figure 28: Comparison of pre-training loss and gradient norms for Pre-LN and Peri-LN architectures with the weight initialization variance set to 2/d (He init (He et al., 2015)), while all other hyperparameter are held constant. (a) Pre-training loss curve (b) Gradient-norm curv…
Figure 29
Figure 29. Figure 29: Comparison of pre-training loss and gradient norms for Pre-LN and Peri-LN architectures with the weight initialization variance set to 1/d (LeCun init), while all other hyperparameter are held constant. (a) Pre-training loss curve (b) Gradient-norm curve [PITH_FULL_I…
Figure 30
Figure 30. Figure 30: Comparison of pre-training loss and gradient norms for Pre-LN and Peri-LN architectures with the weight initialization variance set to 1/(10d), while all other hyperparameter are held constant. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_30.png]
Figure 31
Figure 31. Figure 31: QK-layer normalization in the Attention module. (a) Pre-training loss curve (b) Gradient-norm curve [PITH_FULL_IMAGE:figures/full_fig_p030_31.png]
Figure 32
Figure 32. Figure 32: Comparison of pre-training loss and gradient norm between OLMo2-Style Peri-LN and the Peri-LN architecture. To ensure an accurate comparison, we present the pre-training loss over the final 5B tokens. 400M size model. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_32.png]
Figure 33
Figure 33. Figure 33: Comparison of pre-training loss and gradient norm between OLMo2-Style Peri-LN and the Peri-LN architecture. To ensure an accurate comparison, we present the pre-training loss over the final 5B tokens. 1.5B size model. (a) Pre-training loss curve (b) Gradient-norm curv…
Figure 34
Figure 34. Figure 34: Comparison of pre-training loss and gradient norm between OLMo2-Style Peri-LN and the Peri-LN architecture. To ensure an accurate comparison, we present the pre-training loss over the final 5B tokens. 3.2B size model. I. Training the Transformer using Stochastic Gradi…
Figure 35
Figure 35. Figure 35: Learning Rate Exploration of Pre-& Peri-LN architecture trained with SGD optimizer. The individual training-loss and gradient-norm curves appear in Figures 36, 37, and 38. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_35.png]
Figure 36
Figure 36. Figure 36: Using SGD with learning rate 5e − 3. (a) Pre-training loss curve (b) Gradient-norm curve [PITH_FULL_IMAGE:figures/full_fig_p032_36.png]
Figure 37
Figure 37. Figure 37: Using SGD with learning rate 3e − 3. (a) Pre-training loss curve (b) Gradient-norm curve [PITH_FULL_IMAGE:figures/full_fig_p032_37.png]
Figure 38
Figure 38. Figure 38: Using SGD with learning rate 1e − 3. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_38.png]
Figure 39
Figure 39. Figure 39: Angular distance of hidden state is presented in [PITH_FULL_IMAGE:figures/full_fig_p037_39.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Splitting weight matrices into a fixed-norm direction and learnable per-row/column magnitudes improves LLM training over AdamW/Muon, removes weight decay and warmup, and transfers the optimal LR across width.

  2. LoRDO: Distributed Low-Rank Optimization with Infrequent Communication

    cs.LG 2026-02 conditional novelty 6.0 of 10

    LoRDO combines global low-rank projections with full-rank quasi-hyperbolic momentum to let infrequent-synchronization distributed training match low-rank DDP at roughly 10x less communication.

  3. GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling

    cs.LG 2025-06 conditional novelty 6.0 of 10

    GPAS scales down intermediate activations while preserving backward gradients, reducing activation variance growth in Pre-LN transformers and improving pretraining convergence and downstream performance.

  4. SpanNorm: Reconciling Training Stability and Performance in Deep Transformers

    cs.CL 2026-01 conditional novelty 5.0 of 10

    SpanNorm—a block-level residual with PostNorm-style normalization—trains deeper transformers more stably and outperforms PreNorm and hybrid normalization on LM benchmarks.

Reference graph

Works this paper leans on

45 extracted references · 15 canonical work pages · cited by 4 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    J., Kiros, J

    Ba, L. J., Kiros, J. R., and Hinton, G. E. Layer normalization. CoRR, abs/1607.06450, 2016. URL http://arxiv.org/abs/1607.06450

    arXiv 2016

  3. [3]

    Piqa: Reasoning about physical commonsense in natural language

    Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020

    2020

  4. [4]

    L., and Simonyan, K

    Brock, A., De, S., Smith, S. L., and Simonyan, K. High-performance large-scale image recognition without normalization. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research, pp.\ 1059--1071. PMLR , 2021 a ....

    work page 2021

  5. [5]

    L., and Simonyan, K

    Brock, A., De, S., Smith, S. L., and Simonyan, K. High-performance large-scale image recognition without normalization. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research, pp.\ 1059--1071. PMLR , 2021 b ....

    work page 2021

  6. [6]

    M., Thorne, J., and Yun, S

    Chung, W., Hong, J., An, N. M., Thorne, J., and Yun, S. Stable language model pre-training by reducing embedding variability. In Al - Onaizan, Y., Bansal, M., and Chen, Y. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024 , pp.\ 10852--10863. Association for Com...

    work page 2024

  7. [7]

    Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457

    arXiv 2018

  8. [8]

    Csord \' a s, R., Irie, K., Schmidhuber, J., Potts, C., and Manning, C. D. Moeut: Mixture-of-experts universal transformers. CoRR, abs/2405.16039, 2024. doi:10.48550/ARXIV.2405.16039. URL https://doi.org/10.48550/arXiv.2405.16039

    arXiv doi:10.48550/arxiv.2405.16039 2024

Show all 45 references
  1. [9]

    and Smith, S

    De, S. and Smith, S. L. Batch normalization biases residual blocks towards the identity function in deep networks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Inf...

    2020

  2. [10]

    P., Caron, M., Geirhos, R., Alabdulmohsin, I., Jenatton, R., Beyer, L., Tschannen, M., Arnab, A., Wang, X., Ruiz, C

    Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., Jenatton, R., Beyer, L., Tschannen, M., Arnab, A., Wang, X., Ruiz, C. R., Minderer, M., Puigcerver, J., Evci, U., Kumar, M., van Steenkiste,...

    2023

  3. [11]

    Gpt3.int8(): 8-bit matrix multiplication for transformers at scale

    Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3.int8(): 8-bit matrix multiplication for transformers at scale. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Confere...

    2022

  4. [12]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  5. [13]

    Scaling FP8 training to trillion-token llms

    Fishman, M., Chmiel, B., Banner, R., and Soudry, D. Scaling FP8 training to trillion-token llms. CoRR, abs/2409.12517, 2024. doi:10.48550/ARXIV.2409.12517. URL https://doi.org/10.48550/arXiv.2409.12517

    2024 doi arXiv

  6. [14]

    A framework for few-shot language model evaluation, 07 2024

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...

    2024

  7. [15]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  8. [16]

    Identity mappings in deep residual networks

    He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks. In Leibe, B., Matas, J., Sebe, N., and Welling, M. (eds.), Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV , volum...

    2016 doi

  9. [17]

    A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vin...

    2022 doi arXiv

  10. [18]

    A., Khyalia, S., Jung, J., Goka, H., and Lee, H

    Kedia, A., Zaidi, M. A., Khyalia, S., Jung, J., Goka, H., and Lee, H. Transformers get stable: An end-to-end signal propagation theory for language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.ne...

    2024

  11. [19]

    Y., Bansal, H., Guha, E

    Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S. Y., Bansal, H., Guha, E. K., Keh, S., Arora, K., Garg, S., Xin, R., Muennighoff, N., Heckel, R., Mercat, J., Chen, M., Gururangan, S., Wortsman, M., Albalak, A., Bitton, Y., Nezhurina, M., Abbas, A., Hsieh, C., Gho...

    2024 doi arXiv

  12. [20]

    Mix-ln: Unleashing the power of deeper layers by combining pre-ln and post-ln

    Li, P., Yin, L., and Liu, S. Mix-ln: Unleashing the power of deeper layers by combining pre-ln and post-ln. arXiv preprint arXiv:2412.13795, 2024 b

    2024 arXiv

  13. [21]

    ngpt: Normalized transformer with representation learning on the hypersphere

    Loshchilov, I., Hsieh, C., Sun, S., and Ginsburg, B. ngpt: Normalized transformer with representation learning on the hypersphere. CoRR, abs/2410.01131, 2024. doi:10.48550/ARXIV.2410.01131. URL https://doi.org/10.48550/arXiv.2410.01131

    2024 doi arXiv

  14. [22]

    Pointer sentinel mixture models, 2016

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016

    2016

  15. [23]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018

    2018

  16. [24]

    2 olmo 2 furious

    OLMo, T., Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024

    2024 arXiv

  17. [25]

    L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models t...

    2022

  18. [26]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21 0 (1): 0 5485--5551, 2020

    2020

  19. [27]

    G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \' e , A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C

    Rivi \` e re, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \' e , A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., ...

    2024 doi arXiv

  20. [28]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64 0 (9): 0 99–106, August 2021. ISSN 0001-0782. doi:10.1145/3474381. URL https://doi.org/10.1145/3474381

    2021 doi

  21. [29]

    L., and Choi, Y

    Sap, M., Rashkin, H., Chen, D., Bras, R. L., and Choi, Y. Social iqa: Commonsense reasoning about social interactions. In Inui, K., Jiang, J., Ng, V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna...

    2019 doi

  22. [30]

    Z., and Liu, Z

    Sun, M., Chen, X., Kolter, J. Z., and Liu, Z. Massive activations in large language models. CoRR, abs/2402.17762, 2024. doi:10.48550/ARXIV.2402.17762. URL https://doi.org/10.48550/arXiv.2402.17762

    2024 doi arXiv

  23. [31]

    Spike no more: Stabilizing the pre-training of large language models

    Takase, S., Kiyono, S., Kobayashi, S., and Suzuki, J. Spike no more: Stabilizing the pre-training of large language models. CoRR, abs/2312.16903, 2023. doi:10.48550/ARXIV.2312.16903. URL https://doi.org/10.48550/arXiv.2312.16903

    2023 doi arXiv

  24. [32]

    C ommonsense QA : A question answering challenge targeting commonsense knowledge

    Talmor, A., Herzig, J., Lourie, N., and Berant, J. C ommonsense QA : A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologi...

    2019 doi

  25. [33]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Inf...

    2017

  26. [34]

    GLUE : A multi-task benchmark and analysis platform for natural language understanding

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. GLUE : A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : Analyzing and Interpreting Neural Networks for NLP , pp.\ 353-...

    2018 doi

  27. [35]

    L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: St...

    2020

  28. [36]

    J., Xiao, L., Everett, K

    Wortsman, M., Liu, P. J., Xiao, L., Everett, K. E., Alemi, A. A., Adlam, B., Co - Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl - Dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, S. Small-scale proxies for large-scale transformer training instabil...

    2024

  29. [37]

    On layer normalization in the transformer architecture

    Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T. On layer normalization in the transformer architecture. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , v...

    2020

  30. [38]

    and Hu, E

    Yang, G. and Hu, E. J. Tensor programs IV: feature learning in infinite-width neural networks. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Mach...

    2021

  31. [39]

    J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J

    Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs V: tuning large neural networks via zero-shot hyperparameter transfer. CoRR, abs/2203.03466, 2022. doi:10.48550/ARXIV.2203.03466. URL https://doi....

    2022 doi arXiv

  32. [40]

    Tensor programs VI: feature learning in infinite depth neural networks

    Yang, G., Yu, D., Zhu, C., and Hayou, S. Tensor programs VI: feature learning in infinite depth neural networks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/f...

    2024

  33. [41]

    The super weight in large language models

    Yu, M., Wang, D., Shan, Q., and Wan, A. The super weight in large language models. arXiv preprint arXiv:2411.07191, 2024

    2024 arXiv

  34. [42]

    H ella S wag: Can a machine really finish your sentence? In Korhonen, A., Traum, D., and M \`a rquez, L

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. H ella S wag: Can a machine really finish your sentence? In Korhonen, A., Traum, D., and M \`a rquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4...

    2019 doi

  35. [43]

    Zhai, S., Likhomanenko, T., Littwin, E., Busbridge, D., Ramapuram, J., Zhang, Y., Gu, J., and Susskind, J. M. Stabilizing transformer training by preventing attention entropy collapse. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), ...

    2023

  36. [44]

    and Sennrich, R

    Zhang, B. and Sennrich, R. Root mean square layer normalization. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d'Alch \' e - Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Process...

    2019

  37. [45]

    LIMA: less is more for alignment

    Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., Zhang, S., Ghosh, G., Lewis, M., Zettlemoyer, L., and Levy, O. LIMA: less is more for alignment. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advan...

    2023

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.