Pith. sign in

REVIEW 3 major objections 5 minor 12 references

Training nGPT

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A normalized Transformer recipe reaches the same loss as a 14B hybrid MoE baseline with roughly half the tokens.

desk verdict A practical nGPT recipe for hybrid MoE models with a consistent but single-run 2x token-efficiency claim; worth serious review, not yet a stable number. read the letter →

arxiv 2608.01284 v1 pith:2AOGQUPD submitted 2026-08-02 cs.LG cs.AI

Ilya Loshchilov , Boris Ginsburg This is my paper · ORCID
classification cs.LGcs.AI
keywords normalizedTransformernGPThypersphericalrepresentationlearninghybridMamba-2mixture-of-expertstrainingrecipedataefficiencyGatedAdamWLogitGradientPreconditioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that normalized Transformer training, extended with a practical recipe, makes a modern hybrid Mamba-2–Transformer mixture-of-experts model reach the same validation loss using roughly half as many training tokens as the same architecture trained unnormalized with AdamW. The headline evidence is at 14B total parameters, with consistent ~2.5% lower validation loss across 1B, 2B, 4B, 7B, and 14B models. A sympathetic reader would care because training tokens are the dominant cost of pretraining: if the claim holds, the recipe cuts the data and compute budget needed to hit a given loss in this model class roughly in half. The paper is explicit that this is a recipe-level claim: normalization plus several optimizer changes, not any single component in isolation.

What carries the argument

The object that carries the argument is the nGPT parameterization: every parameter vector that forms a matrix, and every activation recombined into the hidden stream, is constrained to the unit hypersphere, so each update is a rotation of a vector and the forward pass reads as a sequence of learned steps between spherical points. On top of that parameterization, the recipe layers optimizer controls: Logit Gradient Preconditioning removes the time-varying global multiplier that the learned logit scale otherwise injects into the backward pass; Logarithmic Learning Rate Decay front-loads the learning-rate budget and then decays slowly; GatedAdamW separates the numerical epsilon from a tunable coordinate-wise gate on the adaptive update; and the Angular Step Cap limits how far any normalized vector rotates in one optimizer step. Optional mechanisms, including tangent projection, second-moment clipping, and exploration noise, contribute modest measured improvements. After each GatedAdam update, the selected parameter vectors are retracted back onto the sphere by normalization, making the spherical constraint the invariant that the optimizer must respect.

What would settle it

Retrain the 14B nGPT model over a sweep of its peak learning rate, expert and router learning-rate multipliers, gate sharpness, and angular cap, keeping the reported AdamW baseline fixed; if the best-found nGPT run needs more than about half the baseline's tokens to match the baseline's final validation loss, the central data-efficiency claim is falsified. A cheaper partial check is to rerun the reported 14B configuration with a modestly different global learning rate and confirm the half-token crossover is not a knife-edge artifact.

Watch

Extended reading notes

Core claim

The central claim is that the normalized Transformer can be trained at modern hybrid-MoE scale with a complete recipe, and that the result materially beats the same architecture trained with AdamW: the 14B-total-parameter nGPT run matches the baseline's training and validation losses after approximately half as many tokens (Figure 7). Across the five model sizes tested, nGPT validation loss is consistently about 2.5% lower at a fixed token budget, with the 14B points deviating slightly from the trend. The authors attribute most of the gain to the normalized parameterization and the full recipe rather than to GatedAdamW alone, since the gating choice by itself accounts for only a 15–20% token-count saving within nGPT.

Load-bearing premise

The load-bearing premise is that the recipe's hyperparameters, chosen on smaller models and then transferred unchanged to the 14B run, are close enough to optimal for 14B; if that premise is false, the observed curves are real but the half-token comparison overstates the method's benefit.

Editorial extensions

If this is right

  • Reaching a target validation loss on this hybrid MoE class would cost roughly half as many training tokens, which translates into about half the data and a large reduction in training FLOPs.
  • The recipe transfers across sizes without joint retuning of its central hyperparameters, showing consistent ~2.5% lower validation loss at 1B, 2B, 4B, 7B, and 14B.
  • Because the same normalization and optimizer rules apply to dense feed-forward blocks when router and expert modifications are dropped, the recipe is not inherently MoE-specific.
  • The comparison is recipe-plus-parameterization versus a tuned AdamW baseline with WSD decay, so within this paper the benefit is relative to that specific baseline, not to all possible unnormalized training methods.
  • The GatedAdamW ablation positions normalized training as the dominant cause; gating alone would recover only a small share of the token savings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not pursue: even with the ~6% overhead it reports for the 14B run, the practical wall-clock saving will be close to but not exactly 2x, and the net depends on whether that overhead is FLOPs or communication.
  • The logarithmic growth of the learned logit scale points to a hidden, time-varying gradient multiplier that should also affect standard Transformers whose output-embedding row norms play the same role; applying Logit Gradient Preconditioning there is a testable transfer.
  • The fixed 1-degree angular cap suggests a geometric, width-independent step-size limit for normalized parameters; one could try to derive or schedule the cap from the projected gradient rather than treating it as a tuned constant.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents a training recipe for the normalized Transformer (nGPT) applied to hybrid Mamba-2--Transformer Mixture-of-Experts language models. The recipe includes Logit Gradient Preconditioning, logarithmic learning rate decay, GatedAdamW, angular update control, and several optional mechanisms. The main empirical claim is that a 14B-total-parameter nGPT model reaches the same validation loss as an unnormalized GPT baseline trained with AdamW using approximately half as many training tokens, with consistent validation-loss improvements of about 2.5-3.4% across model sizes from 1B to 14B.

Significance. If the central claim holds, this is a practically significant result: roughly halving the data cost to reach a given loss in a modern hybrid MoE architecture would be a substantial contribution to large-scale training efficiency. The paper has notable strengths: it gives a detailed, component-by-component description of the recipe; it reports a consistent loss gap across five model sizes; it includes convergence curves for the 14B model; and it honestly scopes the lack of exhaustive ablations. The recipe is specific enough that the experiments could in principle be reproduced or extended. The significance is tempered by the fact that the headline token-efficiency factor rests on a single 14B run with hyperparameters transferred from smaller models and a baseline that is only described through personal communication.

major comments (3)
  1. [§4.1, Figure 7] The 2x token-efficiency claim is supported by a single 14B nGPT run whose main recipe hyperparameters were fixed in smaller-scale experiments and not jointly retuned at 14B. The paper states that the expert LR multiplier 1.5 and router multiplier 2.0 "were selected based on experiments with smaller models" and that computational constraints did not allow testing alternatives at scale; the same applies to rho=0.05, a=0.5, the 1.0-degree angular cap, and the 0.24/sqrt(d_model) peak LR. Since the 14B run is the only direct evidence for the 2x claim, the comparison is conditional on these transferred settings being near-optimal at 14B. The authors should either provide sensitivity experiments at 14B for at least the most influential hyperparameters, or explicitly reframe the claim as for one specific configuration, which would substantially reduce the strength of the conclusion.
  2. [§4.2, Table 1, Figure 7] The paper reports validation-loss differences at fixed token counts, but the central "approximately half as many training tokens" factor is derived only from the 14B curves in Figure 7 without describing how the horizontal interpolation was performed or what uncertainty it carries. Table 1 gives no token-equivalent savings for the 1B, 2B, 4B, or 7B models, even though the abstract claims the recipe scales across the models considered. Figure 7 shows one GPT curve and one nGPT curve, both single-run, with no error bars or repeated evaluations. The authors should report the token-efficiency ratio for each model size, specify the exact loss-match criterion, and provide some quantification of uncertainty, even if only from multiple checkpoints or small-scale seed variation.
  3. [§4.1-§4.2] The reported comparison changes many variables simultaneously: normalization, optimizer (AdamW vs. GatedAdam), learning-rate schedule (WSD vs. logarithmic), weight decay, betas, warmup conventions, and additional recipe components such as LGP, ASC, PMEN, and SMGC. The paper explicitly notes this for the a=0.5 vs. a=1 ablation, but the main GPT-nGPT comparison has no control for the non-normalization components. As a result, the 2x factor is an attribute of the full recipe relative to one baseline configuration, not of normalized training per se. If the central claim is intended for the full recipe, this should be stated unambiguously in the abstract and conclusion; if the claim is that normalization is the driver, a matched baseline (e.g., GPT trained with the same optimizer and schedule, or nGPT trained with AdamW) is needed.
minor comments (5)
  1. [Abstract] There is a typo in "hyperspherica l representation learning" in the abstract; the space in the middle of the word should be removed.
  2. [§2.1, Eq. (4)] In Equation (4), the notation (s_z/mean(s_z))^q should specify that the division is elementwise and mean(s_z) is a scalar; the current text is understandable but could be clearer for readers implementing LGP.
  3. [§2.3] In Equation (12), the gate is written as sigma(a log(d_t/epsilon_gate)); the relationship to "Adam's epsilon" would be clearer if the text explicitly stated that epsilon_gate is not necessarily equal to the numerical epsilon, since the paper later sets epsilon_gate=1e-8 and epsilon_num=1e-14.
  4. [References] The baseline results in Table 1 and Figure 7 are attributed to "Khona et al. (2026)" which is listed only as "Personal communication". For a claim of this importance, the authors should either provide a public technical report or include a detailed appendix with the baseline hyperparameters, data mix, evaluation split, and loss curves.
  5. [Figure 7] The caption says nGPT reaches the same losses using approximately half the tokens, but it does not state the exact loss value used for the match or the token counts at which the match occurs. Adding these numbers, or drawing a horizontal reference line at the matched loss, would make the claim easier to verify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central token-efficiency claim is an empirical comparison of measured loss curves, not a derivation from fitted quantities.

full rationale

The paper's central claim is that a 14B-total-parameter nGPT model reaches the same validation loss as an unnormalized GPT baseline using approximately half as many training tokens. This claim is supported directly by the measured training and validation loss curves in Figure 7 and the loss values in Table 1. No equation in the paper defines the loss reduction or token-efficiency factor in terms of a fitted parameter; the result is an empirical observation, not a derivation. The nGPT recipe components (LGP, logarithmic decay, GatedAdamW, angular cap, etc.) are presented as design choices, and the paper explicitly notes that the recipe's hyperparameters were selected in smaller-scale experiments and were not jointly retuned at scale (Section 4.1). That is a limitation regarding transferability and whether the comparison is optimally tuned, but it is not circularity: the 14B curves are not forced by construction, and the 'approximately half as many tokens' statement is read off the measured convergence curves, not computed from the tuning choices. The paper's self-citations to Loshchilov et al. (2024) and Loshchilov (2023) introduce the nGPT architecture and prior normalization ideas, but the present evaluation is tested against an external baseline supplied by Khona et al. (2026), and the central result does not rest on an unverified self-citation chain. The paper also candidly marks missing ablations and scaling-law studies as future work (Scope paragraph). No circular step meeting the required evidence standard is present.

Assumptions & free parameters 13 free parameters · 6 assumptions · 0 invented entities

The central claim is an empirical training comparison, so the free parameters are recipe hyperparameters rather than physical constants. The main axioms are the expressiveness of the normalized model class, the fairness of the baseline, and the transferability of smaller-model hyperparameters to 14B. No new physical or model entities are postulated.

free parameters (13)
  • nGPT peak learning rate = 0.24 / sqrt(d_model)
    Global peak LR for nGPT, chosen by hand and not jointly retuned; central to the token-efficiency comparison.
  • Expert LR multiplier = 1.5 for routed/shared expert matrices
    Selected based on smaller-model experiments; affects the 14B result.
  • Router LR multiplier = 2.0
    Selected on smaller models; may be unnecessary; not tested at scale.
  • Logarithmic decay shape rho = 0.05
    Shape parameter of LR schedule; chosen based on smaller experiments.
  • GatedAdam gate sharpness a = 0.5
    Controls the soft gate; a=1 recovers Adam; 0.5 chosen for nGPT runs.
  • Gate threshold eps_gate = 1e-8
    Chosen equal to AdamW epsilon; sets the transition scale of the gate.
  • Numerical epsilon eps_num = 1e-14
    Chosen for numerical stability; separated from the gate threshold.
  • Angular Step Cap theta_max = 1.0 degrees after warmup
    Safeguard cap on angular displacement; warmed up over first 10% of training.
  • Mamba/MoE input scale init = 0.5 * sqrt(d_model)
    Initial values for smamba and smoe to place SiLU/router inputs at appropriate scale.
  • Parameter-specific peak LRs = s_z: 0.5, alpha_A: 0.2, alpha_M: 0.2, s_qk: 0.2
    Effective peak LRs after reparameterization; chosen values were not scaled with model size.
  • PMEN noise strength = lambda_noise = 10
    Exploration noise strength; effect on final loss is modest.
  • SMGC growth cap R = 100
    Permissive cap on second-moment growth; authors note values as low as 2 are comparable.
  • GatedAdam beta coefficients = beta1 = beta2 = 0.975
    Chosen for GatedAdam; differs from AdamW baseline beta2 = 0.95.
assumptions (6)
  • domain assumption Constraining all parameter vectors and activation vectors to the unit hypersphere preserves enough expressive capacity for next-token prediction.
    The nGPT approach (Sections 1, 3.1) assumes the normalized model class is not meaningfully weaker than the unnormalized model class.
  • domain assumption The AdamW baseline from Khona et al. is a fair, well-tuned reference point.
    The comparison uses recommended hyperparameters from a personal communication; no evidence of equal tuning effort for the baseline is provided.
  • domain assumption Mamba-2 input/output projections can be normalized and RMSNorm removed without degrading the model.
    Section 3.2 states this was found to work in experiments but no proof or full comparison is shown.
  • domain assumption Scalar scaling factors for Mamba-2 and MoE inputs are sufficient; vector scales would not improve results meaningfully.
    Section 3.2 reports no significant observed difference between scalar and vector choices, based on limited experiments.
  • ad hoc to paper The 14B model can be treated with the same recipe hyperparameters as smaller models, without joint retuning, without loss of optimality.
    Section 4.1 explicitly states LR multipliers were selected on smaller models and not tested at scale; this is the main transfer assumption.
  • domain assumption The data blend used for both GPT and nGPT runs is held fixed and is representative.
    Training data details are not public; comparisons are only valid under an identical data distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training nGPT." pith.science (2026). https://pith.science/paper/2AOGQUPD

@misc{pith2026260801284,
  author       = {Pith},
  title        = {Pith review of: Training nGPT},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2AOGQUPD}},
  note         = {Machine review of arXiv:2608.01284}
}
read the original abstract

The normalized Transformer (nGPT) realizes hyperspherical representation learning by constraining model parameter vectors and activation vectors to the unit hypersphere. In this paper, we describe a practical training recipe for nGPT and evaluate it on modern hybrid Mamba-2--Transformer Mixture-of-Experts (MoE) models. The recipe introduces Logit Gradient Preconditioning, Logarithmic Learning Rate Decay, GatedAdamW, angular update control, and optional exploration mechanisms. Compared with an unnormalized model of the same hybrid MoE architecture trained with AdamW, the 14B-total-parameter nGPT model reaches the same validation loss using approximately half as many training tokens. The recipe scales across the models considered, which contain up to 14B total parameters.

Figures

Figures reproduced from arXiv: 2608.01284 by the authors.

Figure 1
Figure 1. nGPT’s forward pass as a multi-step optimization o [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Mean value of the logit scale vector sz for an 8B dense nGPT model trained on 4T tokens. The fit is computed from iteration t ≥ 1000. This couples the learned logit scales to the effective learning rate of the output layer and all preceding layers. In particular, if the mean of sz grows during training, then the average gradient scale entering the network also grows, even though this change is not part of the explic… view at source ↗
Figure 3
Figure 3. Logarithmic annealing schedules compared with co [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Effect of the GatedAdamW sharpness parameter [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Scaling results for hybrid Mamba-2–Transformer M [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Histogram of second-moment scales and the corresp [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Training (left) and validation (right) losses as fu [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Validation loss comparison between nGPT trained b [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Validation loss change after removing Pre-Moment [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Evolution of the number of parameter coordinates [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 4 canonical work pages

  1. [1]

    Nemotron 3 nano: Open, efficient mixture-of-experts hybrid mamba-tra nsformer model for agentic reasoning

    Aaron Blakeman, Aaron Grattafiori, Aarti Basant, Abhibha Gu pta, Abhinav Khattar, Adi Renduch- intala, Aditya V avre, Akanksha Shukla, Akhiad Bercovich, A leksander Ficek, et al. Nemotron 3 nano: Open, efficient mixture-of-experts hybrid mamba-tra nsformer model for agentic reasoning. arXiv preprint arXiv:2512.20848 ,

    arXiv

  2. [5]

    SP AM: Spike- aware adam with momentum reset for stable LLM training

    Tianjin Huang, Ziquan Zhu, Gaojie Jin, Lu Liu, Zhangyang Wan g, and Shiwei Liu. SP AM: Spike- aware adam with momentum reset for stable LLM training. arXiv preprint arXiv:2501.06842 ,

    arXiv

  3. [8]

    SGDR: Stochastic gradien t descent with warm restarts

    Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradien t descent with warm restarts. arXiv preprint arXiv:1608.03983,

    arXiv

  4. [10]

    A walk with sgd

    Chen Xing, Devansh Arpit, Christos Tsirigotis, and Y oshua Bengio. A walk with sgd. arXiv preprint arXiv:1802.08770,

    arXiv

  5. [12]

    F or each coordinate with dt,i > 0, γt,i = 1 1 + exp ( −a log dt,i ǫgate ) (25) = 1 1 + (ǫgate/dt,i)a (26) = d a t,i d a t,i +ǫ a gate

    14 A G ATEDADAM W’ S GATE WITHOUT AN EXPLICIT SIGMOID The sigmoid-of-log gate has an equivalent power-law form. F or each coordinate with dt,i > 0, γt,i = 1 1 + exp ( −a log dt,i ǫgate ) (25) = 1 1 + (ǫgate/dt,i)a (26) = d a t,i d a t,i +ǫ a gate . (27) Thus, in vector notation, γt = d a t d a t +ǫ a gate , (28) where powers, divisions, and additions are ...

    work page 2017

  6. [2016]

    An empirical study of mamba- based language models

    Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et a l. An empirical study of mamba- based language models. arXiv preprint arXiv:2406.07887 ,

    arXiv

  7. [2018]

    Spherical latent spaces for st able variational autoencoders

    Jiacheng Xu and Greg Durrett. Spherical latent spaces for st able variational autoencoders. arXiv:1808.10805,

    arXiv

  8. [2020]

    Shengding Hu, Y uge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Lo ng, Zhi Zheng, Y ewei Fang, Y uxiang Huang, Weilin Zhao, et al

    doi: 10.18653/ v1/2020.findings-emnlp.379. Shengding Hu, Y uge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Lo ng, Zhi Zheng, Y ewei Fang, Y uxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the po tential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395 ,

    arXiv 2020

Show all 12 references
  1. [2021]

    Weight norm control

    Ilya Loshchilov. Weight norm control. arXiv preprint arXiv:2311.11446 ,

    arXiv

  2. [2022]

    Rotational equilibrium: How weight decay bal- ances learning across neural networks

    Atli Kosson, Bettina Messmer, and Martin Jaggi. Rotational equilibrium: How weight decay bal- ances learning across neural networks. arXiv:2305.17212,

    arXiv

  3. [2023]

    Query-key normalization for transformers

    Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawa r, and Y uxuan Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics : EMNLP 2020, pp. 4246–4253. Association for Computational Linguistic s,

    2020

  4. [2024]

    J¨ org K. H. Franke, Michael Hefenbrock, Gregor Koehler, and Frank Hutter. Constrained parameter regularization. arXiv:2311.09058,

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.