{"id":"1c700d0c-3859-43b1-b9e2-cdc56d50b1c0","arxiv_id":"2502.05967","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"µnit Scaling combines unit-variance initialization, rearranged LayerNorm, a square-root softmax analysis, and µ-Parametrization-style learning-rate rules to train 1B-13B LLMs in FP8 with no dynamic scaling and zero-shot hyperparameter transfer.","lead":"µnit Scaling is a training recipe that runs every hidden linear layer of a large language model in 8-bit floating point without computing per-tensor dynamic scale factors. The authors report stable FP8 training from 1B to 13B parameters, with learning rates and weight decay transferred from a width-256 model, and throughput 25-33% higher than BF16 training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hyperparameter transfer is demonstrated only at depth 4, but applied to depths 24–40; the large-scale runs confound width transfer with depth effects.","rationale":"I read the paper as an empirical recipe with two coupled claims: static FP8 training works at scale, and zero-shot hyperparameter transfer across widths enables that scale. The FP8 stability evidence is substantial: four model sizes, stable loss curves, and consistent activation distributions. The transfer claim is the most load-bearing part of the contribution, and it has a specific evidential gap that the reader's weakest_assumption does not quite name. Section 3.1's transfer sweeps use four-layer models; every large model in Table 4 is 24 to 40 layers. The paper treats τ as the only depth-dependent hyperparameter, but nothing shows that η and λ are depth-invariant, and Appendix A.3 demonstrates that at least one optimization-related quantity does change with depth. The depth confound matters because a failed transfer would not make the FP8 runs invalid, but it would collapse the advertised 'no special hyperparameters / zero-shot transfer' benefit into a depth-matched tuning story. I am not claiming the method is wrong; the empirical results may well hold. The concern is that the evidence as reported does not isolate the width-transfer mechanism from depth effects. This reinforces, rather than overturns, the reader's conditional verdict: the paper should be required to add depth-matched transfer evidence or soften the transfer claim. No ad hominem is intended; the gap is in the experimental design, not in the authors' conduct.","tokens_in":16920,"tokens_out":17253,"duration_ms":172181,"concrete_test":"Run a depth ablation at fixed width d=1024 with µS: train depths 4, 24, and 40 under the Section 3.1 protocol and sweep η and λ on a coarse grid at each depth. Identify the optimal η* and λ* at each depth, then train depth 40 with the depth-4 optima transferred by the Table 2 width rule (η scaled by sqrt(1024/d) for hidden layers). If the transferred (η, λ) falls outside the depth-40 optimal plateau (e.g., final loss more than 0.25% above the depth-40 sweep optimum, the same criterion used in Appendix A.3), the reported 13B results are confounded by depth. A cheaper secondary check is to retrain the 3B model using η and λ obtained from a depth-32 small-model sweep instead of the depth-4 proxy and compare final losses from Fig. 7.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 establishes optimal η and λ transfer across widths only for four-layer models. Section 3.2 then transfers those optima to 1B–13B models with depths 24, 32, and 40 (Table 4). The only explicitly depth-dependent component in the recipe is τ, fitted separately in Appendix A.3 on depths 20–100; there is no depth sweep of η or λ and no check that the depth-4 proxy's optima remain optimal when depth changes. The large-model loss curves in Fig. 7 therefore jointly depend on the width-scaling rule and on unmeasured depth effects. Since Fig. 9 shows that the optimization geometry (τ*) shifts with depth, the same could plausibly hold for η* and λ*. If so, the advertised zero-shot width transfer to the large models is not established by the reported experiments, and the central claim of simple hyperparameter transfer is weakened.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes µnit Scaling (µS), a recipe for FP8 LLM training with static per-layer scaling factors, unit-variance initialization/output scaling, Res-Post-LayerNorm, a fixed residual coefficient τ, and µP-style learning-rate transfer. The authors derive a variance analysis of causal attention, propose square-root softmax attention but abandon it in favor of Res-Post-LN, and validate the final recipe by training 1B, 3B, 7B, and 13B models in FP8 with no dynamic scaling, reporting loss and downstream-task quality comparable to BF16 baselines plus 25–33% throughput gains. The paper also reports a controlled width sweep at four layers showing that optimal learning rate and weight decay are stable under µS from width 256 to 8192.","tokens_in":17017,"tokens_out":9028,"duration_ms":93202,"significance":"If the transfer and scalability claims hold, this is a practically important contribution: it would show that statically scaled FP8 training can be applied to all hidden linear layers at multi-billion-parameter scale while retaining BF16-quality loss trajectories and enabling zero-shot hyperparameter transfer across widths. The paper is transparent about its empirical scope, includes large-scale (1B–13B) runs with standard evaluation, and openly reports failures such as SP-FP8 divergence at 13B and Unit Scaling divergence at 7B. The central weakness is that the reported width-transfer experiments use only depth-4 proxies, while the large-scale runs use depths 24–40; because the paper itself shows that the residual coefficient must be tuned with depth, the large-scale experiments do not cleanly separate width transfer from depth effects. The derivation for attention variance is internally consistent under its stated iid assumptions, but the observed training behavior deviates from it, so the 'first-principles' narrative is only partially supported.","major_comments":[{"comment":"The hyperparameter-transfer claim is validated only at fixed depth four: Figure 6 sweeps widths 256–8192 for four-layer models, while the 1B–13B runs in Table 4 have depths 24, 32, and 40. Because Appendix A.3 shows that the optimal residual coefficient τ* is depth-dependent, depth is not a neutral variable in this setup, yet no sweep of η or λ versus depth is reported. The large-scale loss curves in Figure 7 therefore jointly depend on the width-scaling rule and on unmeasured depth effects, so the statement in §3.2 that these experiments 'validate zero-shot hyperparameter transfer using µS' is not fully supported. I recommend either adding a same-depth transfer experiment (e.g., a depth-24 base model with width 256 transferred to width 2048 at the same depth) or reporting η* and λ* versus depth at fixed width.","section":"§3.1, §3.2, Table 4"},{"comment":"The value τ=0.4 used for the four-layer models in §3.1 is an extrapolation: the sweep in Appendix A.3 covers depths 20, 40, 60, 80, and 100, not depth 4. Moreover, in the §3.1 sweeps, η and λ are varied while τ is fixed, so the reported optimal η* and λ* are conditioned on a residual coefficient that has not been directly validated for the proxy architecture. Since τ controls the balance between the residual stream and residual branches (Eq. 11), a misspecified τ could shift the apparent optima of η and λ. At minimum, the paper should report a sensitivity analysis of η* and λ* to τ at the base width, or a joint τ–η–λ sweep for depth 4.","section":"§3.1, Appendix A.3"},{"comment":"The theoretical analysis of attention variance is not quantitatively consistent with the observations that motivate the final design. Proposition 2.1 predicts a 1/k decay in attention-output variance under iid value tokens, but Figure 2 shows that observed variance is higher than predicted and that the square-root softmax correction (Eq. 9), which is the direct consequence of the variance-preserving argument in Eqs. 7–8, overcorrects during training. The method ultimately adopted, Res-Post-LayerNorm, is an architectural change justified by correlated-value-token statistics (Fig. 3) and ablations (Fig. 4), not derived from Proposition 2.1. This is not a fatal flaw because the empirical case for Res-Post-LN is separate and convincing, but §1.1's claim that the paper identifies 'root causes' overstates the role of the derivation; the narrative should more clearly separate the iid analysis from the heuristic, empirically-motivated fix.","section":"§2.1, Fig. 2"},{"comment":"The learning-rate transfer rule in Table 2 rests on the exact scale invariance of Eq. 16, which is an infinite-width result for Adam-like optimizers. Appendix A.4 argues that Lion has the same coordinatewise normalization structure as Adam, but it does not address how finite width or fully decoupled weight decay modifies the invariance. Since the paper transfers λ as constant across widths (Table 1) and uses the √(d_base)/√(d_new) rule at widths up to 5120, the theoretical basis would be stronger with either a finite-width analysis or an explicit statement that weight-decay transfer is an empirical finding supported only by the depth-4 sweep in Fig. 6.","section":"§2.3, Appendix A.4"}],"minor_comments":[{"comment":"In the displayed derivation, the text writes σ²_a(k) = ||c||² = √(Σ c_i²) = √(Σ s_i) = 1, but ||c||² is Σ c_i², not its square root; the equality should read ||c||² = Σ c_i² = Σ s_i = 1. This is a typographical error but it makes the variance argument confusing at first reading.","section":"§2.1, Eq. 7–8"},{"comment":"The caption in the main text refers to 'further details in Appendix 3.3', but the efficiency discussion is in Section 3.3, not an appendix. Please correct the cross-reference.","section":"Fig. 8 caption"},{"comment":"The expression 'sign(c_t) = c_t / c_t²' is incorrect as written; the sign function is c_t / |c_t| = c_t / √(c_t²). The subsequent substitution of s_t^Lion = c_t² suggests the intended identity is c_t / √(c_t²), so the statement should be fixed to avoid a dimensional inconsistency.","section":"Appendix A.4, Eq. 24"},{"comment":"The abstract says the method requires 'no dynamic scaling factors or special hyperparameters', but the final recipe includes the residual coefficient τ, which is tuned as a function of depth (Appendix A.3). This is a standard and small hyperparameter set, but the wording could be read as claiming τ-free training; I suggest a more precise phrasing such as 'no special hyperparameters beyond a residual-stream coefficient'.","section":"Abstract, §2.2"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is substantial and the large-scale results are credible, but the central transfer claim needs a depth-controlled experiment or a clearly softened interpretation. I would ask the authors to address the depth confound directly; the other issues are presentation and rigor points that are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core recipe is genuinely useful: combine unit-variance init, 1/sqrt(fan_in) output scaling, Res-Post-LayerNorm, and a fixed residual coefficient to get stable FP8 training with static scaling, no dynamic scale factors, and hyperparameter transfer across widths up to 20x. The loss curves at 1B–13B are convincing, and the fact that SP FP8 diverges at 13B while µS trains stably is a strong practical data point. The efficiency gains (25–33% over BF16, 1–6% over TransformerEngine) are concrete and believable. Second, the paper overstates its quality and transfer claims, and you should not take the abstracts at face value.\n\nWhat is actually new: the specific integration of known pieces (Unit Scaling, µP, Res-Post-LN) into a simple three-hyperparameter scheme, plus a careful diagnosis of attention variance collapse. The square-root softmax analysis is honest—they derive it, show it fails on real data because value tokens are correlated, and pivot to normalization. That is a good example of reporting a negative result clearly. The activation distribution analysis (Appendix A.6) is also a nice diagnostic.\n\nWhere it is soft, in order of severity:\n\n1. \"Equal or better quality\" is not supported by Table 5. On several tasks, µS FP8 is worse than SP BF16 (e.g., 1B Jeopardy, 1B SQuAD, 3B GSM8K). The final train losses are close, but eval is mixed. This needs a softer wording or a serious look at variance.\n\n2. Single runs everywhere. No seeds, no error bars. For a paper that makes 1B–13B claims, that is a real limitation, not a nitpick. At least for the small proxy sweeps they could have repeated.\n\n3. The depth confound in hyperparameter transfer is legitimate. η and λ sweeps are done only at depth 4; large models have depths 24–40, and only τ is adjusted for depth. The stress-test note is correct: the large-scale runs confound width transfer with unmeasured depth effects on η* and λ*. This does not kill the paper—the LR scaling rule is derived from µP width invariance, and the recipe still works—but the phrase \"zero-shot hyperparameter transfer\" should be qualified as \"width-wise transfer at fixed shallow depth, with a depth-dependent τ.\"\n\n4. The theoretical grounding for LR transfer relies on the infinite-width Adam-invariance of Yang et al., applied at finite width and to Lion. That is standard practice in this literature and not a flaw per se, but it is an assumption.\n\nWho this is for: anyone training or evaluating FP8 LLMs, and researchers working on low-precision training schemes. It deserves a serious referee and likely acceptance after the claims are softened and the reproducibility items (code, configs, checkpoints) are addressed. I would not reject it; I would push for a revised version that drops the \"equal quality\" framing and explicitly acknowledges the depth-transfer gap.","headline":"A practical FP8 training recipe with real empirical support, but the 'equal quality' claim is over-strong and the width-transfer validation skips depth effects.","tokens_in":17618,"tokens_out":2412,"would_cite":true,"duration_ms":26309,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a small set of static scaling rules lets transformer language models train fully in FP8 at up to 13B parameters, matching BF16 quality while cutting training time by up to a third.","keywords":["FP8 training","low-precision training","hyperparameter transfer","unit scaling","large language models","static scaling","variance preservation","Res-Post-LayerNorm"],"falsifier":"Train a width-8192 µS model using the learning rate predicted from a width-256 base ($\\eta$ multiplied by $\\sqrt{256/8192}$) and independently sweep $\\eta$ on the same model; if the empirically optimal $\\eta$ differs from the predicted value by more than the sweep spacing, the scale-invariance identity at the heart of µS is not holding at finite width.","tokens_in":16660,"feed_emoji":"⚡","tokens_out":8250,"duration_ms":70264,"temperature":0.7,"pith_summary":"The paper argues that FP8 training of large language models fails mainly because standard transformer blocks are not variance-preserving, and that a small set of static interventions fixes this well enough to drop dynamic scaling factors entirely. The proposed method, µnit Scaling (µS), combines unit-variance weight initialization, a static $1/\\sqrt{\\mathrm{fan\\_in}}$ output multiplier on every hidden linear GEMM, Res-Post-LayerNorm placement, and a fixed depth-dependent residual coefficient $\\tau$. Under these rules the residual stream keeps stable scale, so all hidden linear layers can run in FP8 (e4m3 weights and activations, e5m2 gradients) with constant scaling. Training models from 1B to 13B parameters, the paper reports final loss and evaluation quality equal to or better than BF16 baselines, hyperparameters that transfer from a width-256 proxy to widths up to 5120 with no re-tuning, and throughput 25–33% higher than BF16.","feed_headline":"Train 13B LLMs in FP8 with zero dynamic scaling","feed_subtitle":"µnit Scaling keeps tensors in range with static constants, letting FP8 match BF16 quality and run up to 33% faster.","key_machinery":"The carrying object is a variance-preserving residual stream. Three pieces do the work: (i) Res-Post-LayerNorm, which places LayerNorm at the end of each residual branch so that whatever enters the stream is normalized before summation; (ii) the “fixed” residual modification $x_{l+1} = \\sqrt{1-\\tau}\\,x_l + \\sqrt{\\tau}\\,f(x_l)$, with $\\tau$ picked by depth from a fitted curve; and (iii) static GEMM scaling, where weights initialize with unit variance and every hidden linear layer multiplies its output by $1/\\sqrt{\\mathrm{fan\\_in}}$ in both forward and backward passes. A fourth piece, the Adam-like scale-invariance identity (the output of a hidden layer is unchanged when initialization variance, output multiplier, and learning rate are jointly re-scaled), turns these choices into the learning-rate transfer rule $1/\\sqrt{\\mathrm{fan\\_in}}$ and, across widths, $\\sqrt{d_{\\mathrm{base}}}/\\sqrt{d_{\\mathrm{new}}}$. Proposition 2.1—that attention output variance is $\\propto 1/k$ for iid inputs—motivates the architecture change but is not itself the final fix; the paper abandons a Square-Root Softmax variant because real data correlations break it.","core_discovery":"On its own terms, the paper claims that the apparent need for dynamic scaling factors in FP8 LLM training is an artifact of uncontrolled variance, not a property of the FP8 format. Through Proposition 2.1 it shows that causal self-attention output variance shrinks roughly as $1/k$ with sequence position $k$, and that real text correlations make the effect input-dependent, so no closed-form correction suffices. The discovery is that moving LayerNorm to the end of each residual branch (Res-Post-LayerNorm) and weighting residuals with a fixed $\\tau$ makes the residual stream variance-preserving regardless of token position or data statistics. Combined with unit-variance initialization and a static $1/\\sqrt{\\mathrm{fan\\_in}}$ GEMM multiplier in both forward and backward passes, every hidden linear layer can be cast to FP8 with no per-tensor scaling, and the Adam-like scale-invariance identity of prior hyperparameter-transfer theory then gives learning-rate transfer as $\\sqrt{d_{\\mathrm{base}}}/\\sqrt{d_{\\mathrm{new}}}$. The paper's central claim is that this package—not dynamic scaling—trains 1B–13B transformers in FP8 with quality equal to or better than BF16 and with zero-shot hyperparameter transfer across width.","pith_inferences":["If the variance-preservation story is right, the same static-scaling principle should extend to other low-precision formats (INT8, FP6) and to other architectures with residual streams, since the mechanism is about controlling tensor scale rather than about FP8 specifically.","The paper's Proposition 2.1 identifies a sequence-position-dependent scale collapse in attention; a testable consequence is that µS-trained models should show more uniform token-representation variance across long sequences, which could matter for long-context generalization even though the paper does not test it.","The depth–$\\tau$ curve was fitted on models up to depth 100 with widths 256–2048 and then transferred to the 1B–13B runs; whether $\\tau$ should also shift with width or token budget at even larger scales is left open.","The underflow analysis in Appendix A.7 suggests activation functions with a soft zero (GELU, SiLU) underflow in FP8 far more than ReLU; the paper keeps GELU for final-loss reasons, but a less underflow-prone activation or an asymmetric clip could push FP8 quality further."],"forward_implications":["FP8 training can be made static at billion-parameter scale: all hidden linear GEMMs run in FP8 with constant scaling factors, removing the forward and backward cost of computing dynamic scale factors.","Hyperparameters found on small proxies (width 256) transfer to models up to 20× wider with no re-tuning, which eliminates large-model hyperparameter sweeps.","Because training and inference both use FP8 weights and activations (W8A8), the precision mismatch that usually degrades quantized inference is removed.","At 1B–13B scale, FP8 with µS trains 25–33% faster than BF16 and 1–6% faster than TransformerEngine's dynamic FP8 under held-constant batch and distributed configurations.","µS needs only three hyperparameters (learning rate, weight decay, residual coefficient $\\tau$), versus six or seven in prior transfer schemes."],"supporting_citations":[{"why":"Supplies unit-variance initialization, static $1/\\sqrt{\\mathrm{fan\\_in}}$ output scaling, and the fixed/running-mean residual schemes that µS adopts.","marker":"Blake et al., 2023"},{"why":"Provides the Adam-like scale-invariance identity and the zero-shot hyperparameter-transfer framework that yields µS's learning-rate and width-transfer rules.","marker":"Yang et al., 2021"},{"why":"Establishes the u-µP combination of Unit Scaling and µ-transfer; µS simplifies it and extends FP8 to all hidden layers, since u-µP leaves critical matmuls in BF16.","marker":"Blake et al., 2024"},{"why":"Introduces Res-Post-LayerNorm, the architecture change µS uses to make the residual stream variance-preserving across sequence positions.","marker":"Liu et al., 2022"},{"why":"Gives the first-order Taylor moment approximations used in Proposition 2.1 to show attention output variance scales as $1/k$.","marker":"Casella & Berger, 2002"},{"why":"Defines FP8 e4m3/e5m2 formats and their ranges, which determine the cast and clip choices in µS's FP8 hidden layers.","marker":"Micikevicius et al., 2022"},{"why":"Baseline FP8 library using dynamic scaling; µS is benchmarked against it and claims to remove its per-tensor scaling overhead.","marker":"NVIDIA, 2023"},{"why":"Lion optimizer used in all training runs; the paper argues Lion is Adam-like, making the Adam transfer identities applicable.","marker":"Chen et al., 2023"}],"fun_headline_variants":["FP8 LLM training with zero dynamic scaling, up to 13B","Static scaling, not dynamic: FP8 LLM training matches BF16","µS: FP8 training for LLMs without per-tensor scaling, 33% faster","Simple FP8 training: static constants handle scaling, no dynamic needed","13B LLMs in FP8: zero dynamic scaling, full quality, 33% speedup"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The transfer and stability results assume that the infinite-width scale-invariance identity of Adam-like optimizers, which gives the learning-rate rule, continues to hold exactly at finite widths, with the Lion optimizer, and with weight-decay transfer.","fun_headline_variants_meta":{"raw":{"variants":["FP8 LLM training with zero dynamic scaling, up to 13B","Static scaling, not dynamic: FP8 LLM training matches BF16","µS: FP8 training for LLMs without per-tensor scaling, 33% faster","Simple FP8 training: static constants handle scaling, no dynamic needed","13B LLMs in FP8: zero dynamic scaling, full quality, 33% speedup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000904,"raw_usage":{"total_tokens":3907,"prompt_tokens":981,"completion_tokens":2926,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":2819}},"tokens_in":597,"tokens_out":2926,"duration_ms":21708,"temperature":1.0,"reasoning_tokens":2819,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T17:12:12.212149+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a width-8192 µS model using the learning rate predicted from a width-256 base ($\\eta$ multiplied by $\\sqrt{256/8192}$) and independently sweep $\\eta$ on the same model; if the empirically optimal $\\eta$ differs from the predicted value by more than the sweep spacing, the scale-invariance identity at the heart of µS is not holding at finite width.","supporting_citations":[{"cited_title":"Unit scaling: Out-of-the-box low-precision training","cited_arxiv_id":null,"evidence_quote":"Supplies unit-variance initialization, static $1/\\sqrt{\\mathrm{fan\\_in}}$ output scaling, and the fixed/running-mean residual schemes that µS adopts."},{"cited_title":"J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J","cited_arxiv_id":null,"evidence_quote":"Provides the Adam-like scale-invariance identity and the zero-shot hyperparameter-transfer framework that yields µS's learning-rate and width-transfer rules."},{"cited_title":"Y., Deiseroth, B., Cruz-Salinas, A","cited_arxiv_id":null,"evidence_quote":"Establishes the u-µP combination of Unit Scaling and µ-transfer; µS simplifies it and extends FP8 to all hidden layers, since u-µP leaves critical matmuls in BF16."},{"cited_title":"and Berger, R","cited_arxiv_id":null,"evidence_quote":"Gives the first-order Taylor moment approximations used in Proposition 2.1 to show attention output variance scales as $1/k$."},{"cited_title":"Transformer E ngine, 2023","cited_arxiv_id":null,"evidence_quote":"Baseline FP8 library using dynamic scaling; µS is benchmarked against it and claims to remove its per-tensor scaling overhead."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Lion optimizer used in all training runs; the paper argues Lion is Adam-like, making the Adam transfer identities applicable."}],"review_version":1}