{"id":"12def2ba-1281-473a-bb8d-4ef90c51a9f6","arxiv_id":"2606.04058","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Muon momentum matrices show layer-dependent power-law scaling of stabilized singular value quantiles with model size from 77M to 2.8B parameters.","lead":"The paper tracks singular value quantiles of momentum buffers in the Muon optimizer across models from 77M to 2.8B parameters and reports that after a short burn-in these quantiles stabilize at values following clean power laws in model size with layer-dependent exponents. Practitioners can use the observed exponents to select the minimum number of Newton-Schulz iterations per layer type when scaling to larger models.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"Power-law extrapolation of stabilization quantiles from 2.8B to frontier scales is untested","rationale":"The reader's weakest assumption correctly isolates the extrapolation risk as the load-bearing step for the practical recommendation about NS iteration counts. No more internal inconsistency (e.g., in the definition of stabilization or quantile choice) appears load-bearing once the scaling claim itself is granted within the measured range.","tokens_in":1899,"tokens_out":293,"duration_ms":13301,"concrete_test":"Train one additional model at ~10B parameters using the same Muon setup, measure the late-layer momentum singular-value quantiles after burn-in, and compare against the power-law prediction from the 77M-2.8B fit; deviation >30% falsifies reliable extrapolation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the measured layer-dependent exponents (M^{-0.25} to M^{-0.96}) continue to govern singular-value stabilization at scales >>2.8B. The study fits these exponents on models spanning only ~36x in size; no runs at intermediate or larger scales, no ablation on data mix or other hyperparameters, and no demonstration that the post-burn-in plateau is invariant to optimizer coefficients or initialization are provided. If the true scaling bends or the plateau depends on training details, the predicted NS failure regime for late layers does not follow.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper conducts the first systematic empirical study of singular-value quantiles in the momentum buffers of Muon-trained transformers. Tracking models from 77M to 2.8B parameters, it reports that after a short burn-in the quantiles stabilize to layer-dependent values that obey clean power-law scaling in model size M, with exponents ranging from approximately M^{-0.25} (early/mid layers) to M^{-0.96} (late layers). The authors conclude that standard 5-step Newton-Schulz (NS) will remain sufficient for most layers at frontier scale but that late layers will require additional iterations or retuned coefficients.","tokens_in":2018,"tokens_out":568,"duration_ms":17365,"significance":"If the reported layer-dependent power laws continue to hold, the work supplies a practical, data-driven recipe for choosing minimal NS iteration counts per layer, which could reduce unnecessary orthonormalization cost at scale while preserving update quality. The systematic measurement across depth and size, together with the remarkably clean observed fits, constitutes a useful empirical contribution to the study of orthonormalized optimizers. The study remains purely observational; no theoretical derivation or closed-form prediction is offered.","major_comments":[{"comment":"§4 (Scaling Laws) and Figure 4: The power-law exponents are obtained by fitting quantiles measured on models spanning only a factor of ~36 in size (77M–2.8B). The central claim that late layers will enter the NS failure regime at frontier scales rests on these exponents remaining invariant beyond the observed range; no runs at intermediate or larger scales, no ablation on data mix, learning-rate schedule, or NS coefficients, and no demonstration that the post-burn-in plateau is insensitive to initialization are provided.","section":"§4 and Figure 4"},{"comment":"§3.2 (Quantile measurement protocol): The definition of the “stabilized” quantile (post-burn-in average) and the precise exclusion rules for early training steps are not stated with sufficient precision to allow independent reproduction or assessment of statistical significance of the reported exponents.","section":"§3.2"}],"minor_comments":[{"comment":"Table 1: layer-depth binning boundaries are not explicitly listed; readers cannot map the reported exponents back to concrete layer indices without additional assumptions.","section":"Table 1"},{"comment":"Figure 2 caption: the y-axis label “quantile” should specify whether it is the 0.01, 0.05, or median singular value to avoid ambiguity when comparing panels.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the careful and constructive review. We respond point-by-point to the major comments below.","responses":[{"response":"We agree that the model-size range is limited and that the extrapolation to frontier scales assumes continued invariance of the observed exponents. The power-law fits are remarkably clean and consistent across layers within the studied range, which forms the core empirical contribution. We will add a limitations subsection explicitly discussing the restricted scale range, the assumptions required for extrapolation, and the desirability of future validation at larger scales. We cannot supply additional runs, ablations, or initialization-sensitivity tests at this time.","revision_made":"partial","referee_comment":"[§4 and Figure 4] §4 (Scaling Laws) and Figure 4: The power-law exponents are obtained by fitting quantiles measured on models spanning only a factor of ~36 in size (77M–2.8B). The central claim that late layers will enter the NS failure regime at frontier scales rests on these exponents remaining invariant beyond the observed range; no runs at intermediate or larger scales, no ablation on data mix, learning-rate schedule, or NS coefficients, and no demonstration that the post-burn-in plateau is insensitive to initialization are provided."},{"response":"We accept this criticism. The revised manuscript will state the protocol with full precision: the exact burn-in step threshold, the number of subsequent steps over which the quantile is averaged, the precise exclusion rule for early steps, and any associated statistical measures used to assess stability.","revision_made":"yes","referee_comment":"[§3.2] §3.2 (Quantile measurement protocol): The definition of the “stabilized” quantile (post-burn-in average) and the precise exclusion rules for early training steps are not stated with sufficient precision to allow independent reproduction or assessment of statistical significance of the reported exponents."}],"tokens_in":1530,"tokens_out":447,"duration_ms":18166,"standing_objections":["Additional experiments at larger model scales, ablations on data mix, learning-rate schedule, NS coefficients, and sensitivity of the post-burn-in plateau to initialization."]},"desk_editor":{"model":"grok-4.3","letter":"The paper tracks singular-value quantiles in Muon momentum buffers across models from 77M to 2.8B parameters. After burn-in these quantiles stabilize, and the stable values follow power laws in model size with exponents that vary by layer depth. Most layers scale mildly, around M to the -0.25, while some late layers reach nearly M to the -1.\n\nThey performed the first systematic collection of these spectra at multiple scales and depths. The clean power-law fits and the resulting practical rule for setting Newton-Schulz steps are the concrete advance. The measurements come from actual training runs, which gives the observations a direct empirical basis.\n\nThe range stops at 2.8B. The power laws are fitted inside that window, and the claim that late layers will enter the NS failure regime at much larger scales assumes the same exponents continue to hold. No runs at intermediate or bigger sizes, no sweeps over data mix or optimizer coefficients, and no test of whether the post-burn-in plateau is invariant to those choices. That is the main soft spot.\n\nThe work is aimed at researchers tuning Muon or similar orthonormalized optimizers on large models. Anyone who needs to balance NS cost against update quality will find usable numbers here. It deserves a serious referee because the measurements are new and the application is immediate, even though reviewers will press on the scale limitation.","headline":"The paper measures layer-dependent power laws in Muon momentum singular values up to 2.8B but the extrapolation to frontier scales rests on untested assumptions.","tokens_in":2529,"tokens_out":359,"would_cite":false,"duration_ms":18619,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Muon momentum matrices stabilize to layer-dependent power laws in model size after brief burn-in.","keywords":["Muon optimizer","Newton-Schulz iteration","singular value spectrum","scaling laws","momentum buffer","orthonormalization","large language models","optimizer stability"],"falsifier":"Train a model larger than 10B parameters, extract the singular-value quantiles of the momentum buffers in late layers, and check whether they lie on the extrapolated M^{-0.96} line from the 77M-2.8B data.","tokens_in":2781,"feed_emoji":"","tokens_out":723,"duration_ms":14335,"temperature":0.7,"pith_summary":"The paper tracks the singular-value quantiles of Muon momentum buffers across layers in models from 77M to 2.8B parameters. After an initial period the quantiles settle to stable values that obey clean power-law scaling with model size, where the exponents vary strongly by layer depth. Early and mid-late layers scale mildly, around M to the minus 0.25, so the usual five-step Newton-Schulz iteration keeps them orthonormalized at far larger scales. A subset of late layers scale as steeply as M to the minus 0.96 and will enter the regime where Newton-Schulz fails unless more iterations or adjusted coefficients are supplied. The measured exponents therefore supply a practical, layer-aware rule for choosing the smallest Newton-Schulz configuration that still orthonormalizes the directions that matter.","feed_headline":"Muon late layers scale as steeply as M to the minus 0.96","feed_subtitle":"Mild scaling for most layers keeps five Newton-Schulz steps sufficient; late layers will need more iterations at frontier scale.","key_machinery":"Stabilization of singular-value quantiles of the momentum buffer to layer-dependent power laws in model size.","core_discovery":"After a short burn-in, the quantiles of the singular value spectrum of the momentum buffer stabilize at values determined by layer type and model size; these stabilization values follow power laws in model size M with layer-dependent exponents, mild scaling around M^{-0.25} for layers up to mid-late depth and aggressive scaling up to M^{-0.96} for some late layers.","pith_inferences":["If the exponents persist, training runs at 100B+ parameters will need depth-dependent Newton-Schulz iteration counts that increase toward the output layers.","The clean power-law behavior suggests the training dynamics settle into a scale-invariant spectral regime after the burn-in phase.","The same measurement protocol could be applied to other orthonormalized optimizers to test whether comparable layer-dependent scaling appears."],"forward_implications":["The standard five-step Newton-Schulz iteration will continue to orthonormalize layers up to mid-late depth at frontier model sizes.","Late layers will require more Newton-Schulz iterations or retuned coefficients at large scales to avoid falling into the failure regime.","The measured exponents allow a layer-aware choice of the minimal Newton-Schulz configuration that still orthonormalizes important directions without extra computation.","The stabilization is a consistent property of the training dynamics across the tested range of model sizes."],"fun_headline_variants":["Muon late layers scale as M to the minus 0.96","Muon momentum follows power laws with model size","Layer dependent exponents in Muon scaling laws","Muon late layers will need more Newton-Schulz steps"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The power-law exponents measured on models up to 2.8B parameters will continue to describe the singular-value stabilization behavior at frontier scales.","fun_headline_variants_meta":{"raw":{"variants":["Muon late layers scale as M to the minus 0.96","Muon momentum follows power laws with model size","Layer dependent exponents in Muon scaling laws","Muon late layers will need more Newton-Schulz steps"]},"model":"grok-4.3","cost_usd":0.010709,"raw_usage":{"total_tokens":4769,"prompt_tokens":756,"num_sources_used":0,"completion_tokens":60,"cost_in_usd_ticks":107087000,"prompt_tokens_details":{"text_tokens":756,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3953,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":756,"tokens_out":60,"duration_ms":34843,"temperature":1.0,"reasoning_tokens":3953,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T11:16:54.860341+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Train a model larger than 10B parameters, extract the singular-value quantiles of the momentum buffers in late layers, and check whether they lie on the extrapolated M^{-0.96} line from the 77M-2.8B data.","supporting_citations":[],"review_version":1}