{"id":"6bffe048-b98a-4f00-b922-7073b2561711","arxiv_id":"2608.01284","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":13,"one_line_summary":"An nGPT training recipe with new optimizer and normalization components cuts pretraining tokens roughly in half for a 14B hybrid mixture-of-experts language model compared with an AdamW baseline.","lead":"This paper describes a training recipe for the normalized Transformer (nGPT) and applies it to hybrid Mamba-2 and mixture-of-experts language models. The authors report that a 14B-parameter nGPT model reaches the same validation loss as an unnormalized baseline using roughly half as many training tokens.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 2x token-efficiency claim rests on a single 14B nGPT run whose key hyperparameters were transferred from smaller models without retuning; with both sides differing in many recipe dimensions, the factor is not yet a stable estimate.","rationale":"I agree with the reader's weakest assumption in substance: the least secure premise is not whether the curves exist but whether the single 14B nGPT run is a fair representative of the method. I would frame it symmetrically: since neither side is independent and the nGPT side was not retuned, the observed 2x factor could move if either hyperparameter set were changed. The paper has real supporting evidence: consistent lower validation loss across 1B-14B, an ablation isolating a=0.5 gating, and honest caveats that the recipe components were not exhaustively ablated. These make the result plausible and directionally believable, but they do not pin down the factor 2.0. The concern is testable, albeit expensive, and until tested the verdict should remain CONDITIONAL. I therefore leave the reader's verdict unchanged; my only adjustment is to avoid claiming that an undertuned nGPT run would overstate the benefit, since off-peak nGPT settings would more naturally understate it.","tokens_in":14088,"tokens_out":13418,"duration_ms":129035,"concrete_test":"Re-run the 14B nGPT comparison with the expert/router LR multipliers set to 1.0 and with the global peak LR swept by +/-20% around 0.24/sqrt(2048), all else fixed, and measure the token count at which the best variant reaches the baseline's final validation loss 1.2827. If that count is more than about 330B (60% of the 560B baseline), the 2x factor depends on small-scale-selected multipliers; if it stays near 280B, the transfer concern is answered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing point is the transfer of the nGPT recipe's main hyperparameters from smaller models to the single 14B run that produces the headline factor. Section 4.1 says the expert LR multiplier 1.5 and router multiplier 2.0 'were selected based on experiments with smaller models' and could not be tested at scale; the global peak LR 0.24/sqrt(d_model), the logarithmic decay shape rho=0.05, the gate sharpness a=0.5, and the angular cap of 1.0 degrees were likewise fixed in smaller-scale experiments without joint retuning. The GPT baseline, in contrast, comes from the Khona et al. scaling ladder with per-size tuned peak LRs, different betas, weight decay, and a WSD schedule. The Figure 7 comparison therefore measures one particular nGPT configuration against one particular baseline configuration. The central claim requires that this pair is representative; if either run is off-peak, the horizontal distance to the matched loss is not a stable estimate of the recipe's token efficiency. The paper explicitly flags this limitation in the Scope paragraph and in Section 4.1, so the reader's conditional stance is warranted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a training recipe for the normalized Transformer (nGPT) applied to hybrid Mamba-2--Transformer Mixture-of-Experts language models. The recipe includes Logit Gradient Preconditioning, logarithmic learning rate decay, GatedAdamW, angular update control, and several optional mechanisms. The main empirical claim is that a 14B-total-parameter nGPT model reaches the same validation loss as an unnormalized GPT baseline trained with AdamW using approximately half as many training tokens, with consistent validation-loss improvements of about 2.5-3.4% across model sizes from 1B to 14B.","tokens_in":14454,"tokens_out":3901,"duration_ms":38981,"significance":"If the central claim holds, this is a practically significant result: roughly halving the data cost to reach a given loss in a modern hybrid MoE architecture would be a substantial contribution to large-scale training efficiency. The paper has notable strengths: it gives a detailed, component-by-component description of the recipe; it reports a consistent loss gap across five model sizes; it includes convergence curves for the 14B model; and it honestly scopes the lack of exhaustive ablations. The recipe is specific enough that the experiments could in principle be reproduced or extended. The significance is tempered by the fact that the headline token-efficiency factor rests on a single 14B run with hyperparameters transferred from smaller models and a baseline that is only described through personal communication.","major_comments":[{"comment":"The 2x token-efficiency claim is supported by a single 14B nGPT run whose main recipe hyperparameters were fixed in smaller-scale experiments and not jointly retuned at 14B. The paper states that the expert LR multiplier 1.5 and router multiplier 2.0 \"were selected based on experiments with smaller models\" and that computational constraints did not allow testing alternatives at scale; the same applies to rho=0.05, a=0.5, the 1.0-degree angular cap, and the 0.24/sqrt(d_model) peak LR. Since the 14B run is the only direct evidence for the 2x claim, the comparison is conditional on these transferred settings being near-optimal at 14B. The authors should either provide sensitivity experiments at 14B for at least the most influential hyperparameters, or explicitly reframe the claim as for one specific configuration, which would substantially reduce the strength of the conclusion.","section":"§4.1, Figure 7"},{"comment":"The paper reports validation-loss differences at fixed token counts, but the central \"approximately half as many training tokens\" factor is derived only from the 14B curves in Figure 7 without describing how the horizontal interpolation was performed or what uncertainty it carries. Table 1 gives no token-equivalent savings for the 1B, 2B, 4B, or 7B models, even though the abstract claims the recipe scales across the models considered. Figure 7 shows one GPT curve and one nGPT curve, both single-run, with no error bars or repeated evaluations. The authors should report the token-efficiency ratio for each model size, specify the exact loss-match criterion, and provide some quantification of uncertainty, even if only from multiple checkpoints or small-scale seed variation.","section":"§4.2, Table 1, Figure 7"},{"comment":"The reported comparison changes many variables simultaneously: normalization, optimizer (AdamW vs. GatedAdam), learning-rate schedule (WSD vs. logarithmic), weight decay, betas, warmup conventions, and additional recipe components such as LGP, ASC, PMEN, and SMGC. The paper explicitly notes this for the a=0.5 vs. a=1 ablation, but the main GPT-nGPT comparison has no control for the non-normalization components. As a result, the 2x factor is an attribute of the full recipe relative to one baseline configuration, not of normalized training per se. If the central claim is intended for the full recipe, this should be stated unambiguously in the abstract and conclusion; if the claim is that normalization is the driver, a matched baseline (e.g., GPT trained with the same optimizer and schedule, or nGPT trained with AdamW) is needed.","section":"§4.1-§4.2"}],"minor_comments":[{"comment":"There is a typo in \"hyperspherica l representation learning\" in the abstract; the space in the middle of the word should be removed.","section":"Abstract"},{"comment":"In Equation (4), the notation (s_z/mean(s_z))^q should specify that the division is elementwise and mean(s_z) is a scalar; the current text is understandable but could be clearer for readers implementing LGP.","section":"§2.1, Eq. (4)"},{"comment":"In Equation (12), the gate is written as sigma(a log(d_t/epsilon_gate)); the relationship to \"Adam's epsilon\" would be clearer if the text explicitly stated that epsilon_gate is not necessarily equal to the numerical epsilon, since the paper later sets epsilon_gate=1e-8 and epsilon_num=1e-14.","section":"§2.3"},{"comment":"The baseline results in Table 1 and Figure 7 are attributed to \"Khona et al. (2026)\" which is listed only as \"Personal communication\". For a claim of this importance, the authors should either provide a public technical report or include a detailed appendix with the baseline hyperparameters, data mix, evaluation split, and loss curves.","section":"References"},{"comment":"The caption says nGPT reaches the same losses using approximately half the tokens, but it does not state the exact loss value used for the match or the token counts at which the match occurs. Adding these numbers, or drawing a horizontal reference line at the matched loss, would make the claim easier to verify.","section":"Figure 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its limitations, including the absence of at-scale hyperparameter retuning and the use of a personal-communication baseline. My recommendation is conditional on whether the authors can strengthen the evidence for the central token-efficiency claim, for example by adding at least small-scale sensitivity analyses or by reframing the claim as configuration-specific rather than general. The result is promising, but the current evidence is too thin for the strength of the abstract's 2x statement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper reports the first application of nGPT to a hybrid Mamba-2–Transformer MoE, with a training recipe (LGP, logarithmic decay, GatedAdamW, angular cap) that consistently lowers validation loss across five model sizes—about 2.5% relative at matched tokens. Second, the headline 'half the tokens' claim comes from a single 14B run per configuration, and the comparison measures one nGPT package against one baseline package, with the baseline supplied via personal communication.\n\nWhat it does well: the recipe is concrete, the writing is clear, and the authors are honest about what is not ablated. The internal comparisons (a=0.5 vs Adam gating, PMEN on/off, tangent projection) show small, consistent effects, which supports the idea that the gain is not from one magical knob. The loss gap is monotonic and relatively flat across sizes, which is what you'd expect if the normalized parameterization genuinely helps.\n\nThe soft spots are in proportion: the central result rests on a single run per point. No error bars, no seeds. The nGPT hyperparameters—peak LR 0.24/sqrt(d), expert/router multipliers, log-decay shape rho, angle cap—were fixed on smaller models and not jointly retuned at 14B, as Section 4.1 admits. The GPT baseline uses a WSD schedule and AdamW; nGPT uses log decay and GatedAdam. So the '2x' is the horizontal distance between two particular curves, not a stable estimate of the recipe's efficiency. If either run is off-peak, the factor shifts. The paper's own Scope paragraph acknowledges the absence of exhaustive ablations.\n\nI don't think the central argument collapses; it's consistent and plausible. But the quantitative headline is a single observation with many simultaneous changes. The authors also note some components (SMGC, PMEN) have modest or negligible impact, which is fine but means the recipe has more parts than demonstrated contributions.\n\nWho this is for: LLM pretraining practitioners who want a concrete alternative to RMSNorm+AdamW, and optimizer researchers who care about normalized optimization. It deserves a serious referee—the idea is important and the evidence is honestly presented, even if the headline needs more runs. My recommendation: accept for peer review, but ask for at least one repeat of the 14B comparison or an explicit uncertainty statement, and a clearer decomposition of which recipe components carry the gain.","headline":"A practical nGPT recipe for hybrid MoE models with a consistent but single-run 2x token-efficiency claim; worth serious review, not yet a stable number.","tokens_in":14934,"tokens_out":2505,"would_cite":true,"duration_ms":21450,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A normalized Transformer recipe reaches the same loss as a 14B hybrid MoE baseline with roughly half the tokens.","keywords":["normalized Transformer","nGPT","hyperspherical representation learning","hybrid Mamba-2 Transformer mixture-of-experts","training recipe","data efficiency","GatedAdamW","Logit Gradient Preconditioning"],"falsifier":"Retrain the 14B nGPT model over a sweep of its peak learning rate, expert and router learning-rate multipliers, gate sharpness, and angular cap, keeping the reported AdamW baseline fixed; if the best-found nGPT run needs more than about half the baseline's tokens to match the baseline's final validation loss, the central data-efficiency claim is falsified. A cheaper partial check is to rerun the reported 14B configuration with a modestly different global learning rate and confirm the half-token crossover is not a knife-edge artifact.","tokens_in":13917,"feed_emoji":"📉","tokens_out":12454,"duration_ms":102658,"temperature":0.7,"pith_summary":"The paper argues that normalized Transformer training, extended with a practical recipe, makes a modern hybrid Mamba-2–Transformer mixture-of-experts model reach the same validation loss using roughly half as many training tokens as the same architecture trained unnormalized with AdamW. The headline evidence is at 14B total parameters, with consistent ~2.5% lower validation loss across 1B, 2B, 4B, 7B, and 14B models. A sympathetic reader would care because training tokens are the dominant cost of pretraining: if the claim holds, the recipe cuts the data and compute budget needed to hit a given loss in this model class roughly in half. The paper is explicit that this is a recipe-level claim: normalization plus several optimizer changes, not any single component in isolation.","feed_headline":"Normalized model halves the tokens needed to hit a target loss","feed_subtitle":"A 14B mixture-of-experts model reaches the same validation loss after about half as many training tokens.","key_machinery":"The object that carries the argument is the nGPT parameterization: every parameter vector that forms a matrix, and every activation recombined into the hidden stream, is constrained to the unit hypersphere, so each update is a rotation of a vector and the forward pass reads as a sequence of learned steps between spherical points. On top of that parameterization, the recipe layers optimizer controls: Logit Gradient Preconditioning removes the time-varying global multiplier that the learned logit scale otherwise injects into the backward pass; Logarithmic Learning Rate Decay front-loads the learning-rate budget and then decays slowly; GatedAdamW separates the numerical epsilon from a tunable coordinate-wise gate on the adaptive update; and the Angular Step Cap limits how far any normalized vector rotates in one optimizer step. Optional mechanisms, including tangent projection, second-moment clipping, and exploration noise, contribute modest measured improvements. After each GatedAdam update, the selected parameter vectors are retracted back onto the sphere by normalization, making the spherical constraint the invariant that the optimizer must respect.","core_discovery":"The central claim is that the normalized Transformer can be trained at modern hybrid-MoE scale with a complete recipe, and that the result materially beats the same architecture trained with AdamW: the 14B-total-parameter nGPT run matches the baseline's training and validation losses after approximately half as many tokens (Figure 7). Across the five model sizes tested, nGPT validation loss is consistently about 2.5% lower at a fixed token budget, with the 14B points deviating slightly from the trend. The authors attribute most of the gain to the normalized parameterization and the full recipe rather than to GatedAdamW alone, since the gating choice by itself accounts for only a 15–20% token-count saving within nGPT.","pith_inferences":["A natural extension the paper does not pursue: even with the ~6% overhead it reports for the 14B run, the practical wall-clock saving will be close to but not exactly 2x, and the net depends on whether that overhead is FLOPs or communication.","The logarithmic growth of the learned logit scale points to a hidden, time-varying gradient multiplier that should also affect standard Transformers whose output-embedding row norms play the same role; applying Logit Gradient Preconditioning there is a testable transfer.","The fixed 1-degree angular cap suggests a geometric, width-independent step-size limit for normalized parameters; one could try to derive or schedule the cap from the projected gradient rather than treating it as a tuned constant."],"forward_implications":["Reaching a target validation loss on this hybrid MoE class would cost roughly half as many training tokens, which translates into about half the data and a large reduction in training FLOPs.","The recipe transfers across sizes without joint retuning of its central hyperparameters, showing consistent ~2.5% lower validation loss at 1B, 2B, 4B, 7B, and 14B.","Because the same normalization and optimizer rules apply to dense feed-forward blocks when router and expert modifications are dropped, the recipe is not inherently MoE-specific.","The comparison is recipe-plus-parameterization versus a tuned AdamW baseline with WSD decay, so within this paper the benefit is relative to that specific baseline, not to all possible unnormalized training methods.","The GatedAdamW ablation positions normalized training as the dominant cause; gating alone would recover only a small share of the token savings."],"supporting_citations":[{"why":"defines the original nGPT hyperspherical parameterization and eigen learning-rate mechanism that this recipe extends to MoE models.","marker":"Loshchilov et al., 2024"},{"why":"provides the Nemotron-3 architecture and industry-grade data blend on which all training runs are built.","marker":"Blakeman et al., 2025"},{"why":"supplies the scaling-ladder models, the GPT baseline results with AdamW, and the WSD-based hyperparameters used for the comparison.","marker":"Khona et al., 2026"},{"why":"defines AdamW, the baseline optimizer and the update that GatedAdamW generalizes.","marker":"Loshchilov & Hutter, 2019"},{"why":"provides the Warmup-Stable-Decay schedule that sets the baseline learning-rate curve.","marker":"Hu et al., 2024"},{"why":"defines Mamba-2, the state-space block whose projections are normalized in the hybrid model.","marker":"Dao & Gu, 2024"}],"fun_headline_variants":["nGPT recipe cuts tokens to reach target loss in half","Half the tokens, same loss: nGPT scaling on hybrid MoE","Normalized transformer matches AdamW loss with 50% fewer tokens","nGPT: 14B model halves training tokens to match loss","nGPT recipe: half the training tokens for same loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the recipe's hyperparameters, chosen on smaller models and then transferred unchanged to the 14B run, are close enough to optimal for 14B; if that premise is false, the observed curves are real but the half-token comparison overstates the method's benefit.","fun_headline_variants_meta":{"raw":{"variants":["nGPT recipe cuts tokens to reach target loss in half","Half the tokens, same loss: nGPT scaling on hybrid MoE","Normalized transformer matches AdamW loss with 50% fewer tokens","nGPT: 14B model halves training tokens to match loss","nGPT recipe: half the training tokens for same loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000897,"raw_usage":{"total_tokens":3798,"prompt_tokens":812,"completion_tokens":2986,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":428,"completion_tokens_details":{"reasoning_tokens":2898}},"tokens_in":428,"tokens_out":2986,"duration_ms":18885,"temperature":1.0,"reasoning_tokens":2898,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:08:43.421680+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the 14B nGPT model over a sweep of its peak learning rate, expert and router learning-rate multipliers, gate sharpness, and angular cap, keeping the reported AdamW baseline fixed; if the best-found nGPT run needs more than about half the baseline's tokens to match the baseline's final validation loss, the central data-efficiency claim is falsified. A cheaper partial check is to rerun the reported 14B configuration with a modestly different global learning rate and confirm the half-token crossover is not a knife-edge artifact.","supporting_citations":[],"review_version":2}