{"id":"aa7b9fd2-3756-41d2-8305-baab1b930cf1","arxiv_id":"2512.04632","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AOL column rescaling before Newton-Schulz improves the starting point enough to drop one iteration, making Muon's orthogonalization roughly 2.8x faster at similar final accuracy.","lead":"This paper speeds up Muon, an optimizer that orthogonalizes gradient updates, by first rescaling gradient columns with an almost-orthogonal preconditioner before the standard Newton-Schulz iterations. The change is drop-in, needs no tuning, and skips one expensive iteration, giving a few percent faster training in the tested language and vision benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing same-iteration baseline: the claimed speedup from AOL may instead be due to removing the fifth NS iteration or the fused Triton kernel.","rationale":"The reader identified the missing same-iteration baseline as the weakest assumption; my analysis converges on the same point. The paper's design systematically pairs Turbo-Muon at k iterations with baselines at k+1, so the observed runtime improvement cannot be uniquely attributed to AOL. The cleanest control is a within-Turbo-Muon ablation that swaps AOL for Frobenius normalization, holding everything else fixed; this directly answers whether AOL enables the iteration removal. Table 3's significant loss gap (0.0017, approx 2.8 SE) further complicates the 'matching reference performance' claim, though it may be acceptable in a speedrun context. No other concern — theoretical descent proof, polar-error measurements, heavy-tailed robustness — is as load-bearing. The theory shows only a descent direction, not that AOL is necessary; the heavy-tailed experiments are supportive. Therefore the verdict remains CONDITIONAL, pending the same-iteration comparison. I recommend no change to the reader's conditional verdict.","tokens_in":16615,"tokens_out":6509,"duration_ms":59988,"concrete_test":"Run the Modded-NanoGPT speedrun (4xH100, same recipe) with four configurations: (1) Turbo-Muon 4it as in the paper; (2) identical Turbo-Muon code but with AOL column scaling replaced by Frobenius normalization (one-line change), keeping 4 iterations, truncated coefficients, and the fused Triton kernel; (3) Muon+ 4it using the same truncated coefficient schedule; (4) Muon+ 5it as reference. Compare final validation loss and wall-clock runtime across 10 trials. If configuration (2) or (3) matches Turbo-Muon's loss within noise (e.g., within 0.001) while giving the same runtime, the central attribution to AOL fails. If (2) is measurably worse, AOL is load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that AOL preconditioning is what allows dropping one Newton-Schulz iteration without loss. But every head-to-head training comparison (Fig. 4b, Table 3) compares Turbo-Muon at k iterations against Muon/Muon+ at k+1 iterations, so the reported speedup conflates three changes: AOL scaling, one fewer iteration, and a fused Triton kernel. No experiment holds iteration count and kernel fixed and varies only the preconditioner. If Muon+ (or Turbo-Muon's kernel with Frobenius normalization) using four iterations already achieves the same NanoGPT validation loss as Turbo-Muon's four-iteration run, then removing one iteration is possible without AOL, and AOL is not the enabling factor. The paper's own Table 3 shows a small but statistically significant degradation for Turbo-Muon (4it) vs. Muon+ (5it): 3.2791 vs 3.2774, with SE ≈ 0.0006 (t ≈ 2.8), which also weakens the 'matching reference performance' wording. The polar-error evidence in Fig. 3a is suggestive but does not establish that the training loss is sensitive to that error difference at four iterations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Turbo-Muon, a modification of the Newton–Schulz orthogonalization step used in the Muon optimizer. The key idea is to replace Frobenius-norm pre-scaling with an Almost-Orthogonal Layer (AOL) column rescaling, which the authors argue reduces the initial polar error and allows one of the usual five Newton–Schulz iterations to be removed. The paper reports synthetic polar-error experiments, heavy-tailed robustness checks, CIFAR-10 and NanoGPT training runs, and a theoretical steepest-descent interpretation of the resulting update. It claims a parameter-free drop-in replacement that is faster than Muon/Muon+ while matching final model quality.","tokens_in":16869,"tokens_out":6050,"duration_ms":55876,"significance":"If the central attribution were established, the contribution would be practically valuable: a 20% reduction in orthogonalization cost, no hyperparameter tuning, public implementations, and consistent small end-to-end runtime gains. The steepest-descent-in-a-rescaled-norm result and the strict-descent-direction lemma are useful theoretical additions, and the paper correctly avoids fitted parameters or self-derived constants. However, as submitted, the central claim that AOL specifically enables removing one Newton–Schulz iteration is not isolated from other implementation changes, and several numerical statements in the abstract outrun the reported tables.","major_comments":[{"comment":"The central claim that AOL preconditioning enables removing one Newton–Schulz iteration is not isolated. Every head-to-head comparison gives Turbo-Muon k iterations while baselines get k+1. Since Turbo-Muon also differs by a fused Triton kernel and by the coefficient-truncation schedule (App. D), the speedup could be due to fewer matrix multiplications or kernel implementation rather than AOL. A necessary control is a 4-iteration Muon+ run with the same truncated coefficients and the same kernel as Turbo-Muon but with Frobenius normalization. If that baseline matches Turbo-Muon's final loss and polar error, the claimed enabling role of AOL is not supported. This is directly load-bearing for the abstract and conclusions.","section":"§4.2, Fig. 3a/4b, Table 3"},{"comment":"The abstract claims 5–10% end-to-end improvement and 'matching reference performance'. The reported end-to-end reductions are 273.75→266.65 s (2.6%) in Table 3 and 2.66→2.64 s (0.75%) in Table 1. The 8–10% figure in Fig. 4a is a step-time simulation, not end-to-end training time. Moreover, Table 3 shows 3.2791 (Turbo-Muon, 4it) vs 3.2774 (Muon+ PE, 5it) with stds ≈0.0013–0.0014 over 10 runs; this difference is statistically significant (t≈2.8), so the phrase 'matching reference performance' overstates the result. Claims should be aligned with the measured effect sizes and significance.","section":"Table 3 / Abstract"},{"comment":"The polar-error evidence is suggestive but does not establish that training metrics are insensitive to the polar-error gap at four iterations. Fig. 3a compares Turbo-Muon at 4 iterations with baselines at 5; a same-iteration polar-error plot is needed. Additionally, the bias analysis in §6 and Table 2 reports εbias≈0.1, which is not negligible relative to εapprox, yet final accuracy is unchanged; the paper does not explain why a large polar bias has no effect on optimization in that experiment. A brief analysis or reference would make the training-insensitivity claim credible.","section":"Fig. 3a, §6"}],"minor_comments":[{"comment":"A1 = s^T A0 s is dimensionally inconsistent for a matrix A1; the correct update is A1 = S A0 S with S = diag(s), or componentwise A1_ij = s_i A0_ij s_j.","section":"Algorithm 2, line 4"},{"comment":"Figures 6, 7, and 8 say they reproduce 'Fig. 1' and 'Fig. 3a' of the main paper, but the preconditioning comparison appears to be Fig. 2; please verify the cross-references.","section":"Appendix B"},{"comment":"The numerical claims are inconsistent across versions: the abstract mentions '~3%' training-time reduction and '5–10%' improvement, while Table 3 shows 2.6% and Table 1 shows 0.75%; the conclusion says 8–10% step-time speedups. Please reconcile these numbers so the reader can tell which quantity is being reported.","section":"Abstract and Conclusion"},{"comment":"There are typos such as 'high regimes' and 'hight regimes'; also clarify whether εbias and εapprox are normalized as in §3. The large εbias value deserves a one-sentence interpretation.","section":"Table 2 / §6"},{"comment":"The notation |X^T X| should be defined as the elementwise absolute value, and the diagonal inverse square root should be stated explicitly; this will avoid confusion with matrix absolute value or determinant.","section":"Eq. (6), §4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a legitimate systems/optimization contribution with public code. The missing same-iteration baseline is a standard ablation and should be feasible for the authors to run; I therefore recommend major revision rather than rejection. Please require the authors to add a 4-iteration Muon+ control (ideally with the same Triton kernel and coefficient schedule as Turbo-Muon) and to recalibrate the abstract's runtime and 'matching performance' claims against the measured 2.6% end-to-end gain and the statistically significant 0.0017 loss degradation in Table 3."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate optimization trick with a genuine new combination, but the paper's central causal claim—that AOL preconditioning is what allows dropping one Newton-Schulz iteration—is never actually isolated, and the abstract overstates the end-to-end gains. Worth engaging, but it needs a same-iteration baseline and some claim-shrinking before I'd trust the attribution.\n\nWhat's new: applying AOL column rescaling as a preconditioner inside Muon's Newton-Schulz, caching the Gram matrix so the preconditioner is nearly free, and using the improved starting point to drop one NS iteration. The components are known, but this specific combination is new. The numerical result that AOL cuts polar error, especially for large matrices, is real and clearly shown (Fig. 2, 3a). The steepest-descent-in-a-reweighted-norm argument in Sec. 6 is correct as far as it goes, and Lemma 1 (the update never points against the gradient) is fine. The heavy-tailed stress tests in App. B are a nice touch.\n\nSoft spots. The load-bearing claim—'AOL enables removing one iteration'—is not actually tested. Every head-to-head training comparison gives Turbo-Muon k iterations and the baselines k+1, so the speedup conflates three changes: AOL, the missing iteration, and the fused Triton kernel. There is no experiment that holds iteration count and kernel fixed and varies only the preconditioner. If Muon+ with four iterations already matches Turbo-Muon's four-iteration loss, then the speedup is just fewer iterations plus a better kernel, not AOL. Fig. 3a shows AOL improves polar error at equal iterations, but it doesn't show that training loss is sensitive to that difference at four iterations.\n\nThe abstract is also in tension with the numbers. The full-text abstract promises 5–10% end-to-end improvement; the tables show 2.6% on NanoGPT (273.75s to 266.65s) and 0.75% on CIFAR (2.66s to 2.64s). The shorter abstract at the top of the record says ~3%, which is closer but still not exactly Table 3. And Appendix E's direct comparison shows a small but statistically significant loss regression (3.2791 vs 3.2774, t≈2.8) for Turbo-Muon 4it vs Muon+ 5it, which undercuts the 'matching reference performance' wording. The code references are also vague—no URL or commit hash.\n\nOverall: the engineering is credible and the theory is sound, but the paper would read more honestly as 'we shave one NS iteration and fuse the kernel, with AOL giving a head start on polar error' rather than 'AOL is what enables the iteration removal.' With a same-iteration control and trimmed claims, this is a reasonable optimizer paper.\n\nRecommendation: send to peer review, but with a clear request for the missing baseline—Muon+ (and ideally Turbo-Muon with Frobenius normalization) at four iterations. If the authors won't add that, I'd be inclined to reject or downgrade.","headline":"Plausible trick, genuine new combination, but the central attribution is untested and the abstract oversells the gains.","tokens_in":17398,"tokens_out":5356,"would_cite":false,"duration_ms":46036,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AOL preconditioning lets the Muon optimizer drop one Newton-Schulz iteration without loss of accuracy.","keywords":["Muon optimizer","Newton-Schulz iteration","polar factor","almost-orthogonal preconditioning","orthogonalization","training speedup","steepest descent","gradient preconditioning"],"falsifier":"Run Muon+ with exactly four Newton-Schulz iterations using the same truncated coefficient schedule as Turbo-Muon, and compare polar error on random and heavy-tailed matrices plus end-to-end validation loss on the paper's GPT and CIFAR-10 speed-runs; if the four-iteration baseline matches Turbo-Muon's loss and polar error, the central claim that AOL enables dropping the iteration is false.","tokens_in":16529,"feed_emoji":"⚡","tokens_out":5101,"duration_ms":47622,"temperature":0.7,"pith_summary":"The paper tries to establish that replacing the usual Frobenius-norm normalization with an almost-orthogonal column rescaling before Newton-Schulz iterations makes the iterative orthogonalization converge fast enough that Muon can run four iterations instead of the usual five while keeping equal polar accuracy and final model quality. If true, this is a drop-in change: no hyperparameter retuning, roughly 20 percent lower orthogonalization cost, and consistent end-to-end runtime gains—around 3 percent in tightly tuned speed-run benchmarks and more at medium scales where the orthogonalization overhead is visible. The paper also argues that the bias introduced by preconditioning is benign: the resulting update remains a strict descent direction, so training cannot be pushed uphill by the approximation. Why this matters: the main practical barrier to orthogonality-based optimizers is the cost of the orthogonalization step, and any reduction widens the range of models where they are affordable.","feed_headline":"Preconditioner lets Muon drop one Newton-Schulz iteration","feed_subtitle":"Four iterations now match five: same loss on language and vision, up to 3% faster end-to-end.","key_machinery":"The central object is the almost-orthogonal (AOL) rescaling: a diagonal matrix S whose entries are the inverse square roots of the row sums of the Gram matrix A0 = X^T X, applied to the gradient columns before Newton-Schulz iterations. Its work is to bound the column-space coherence—via Gershgorin-style row-sum bounds on the Gram matrix—so that the initial matrix is closer to orthogonal and better conditioned than Frobenius-normalized input. The computational trick is that A0, once computed for the preconditioner, is reused as the first matrix product inside the first Newton-Schulz step, reducing the preconditioner's overhead to a cheap elementwise rescaling and making the entire scheme near","core_discovery":"The paper's central claim is that the initial normalization step shared by Muon and Muon+—dividing the gradient by its Frobenius norm—is a poor starting point for Newton-Schulz, because it preserves the condition number and typically sends the spectral norm far below 1. Turbo-Muon replaces it with AOL preconditioning: scale the columns of the gradient by the inverse square roots of the row sums of the Gram matrix X^T X. This yields an initial matrix much closer to orthogonal, especially for large matrices, and the Gram matrix computed for the preconditioner can be cached and reused as the first product of the first Newton-Schulz iteration, so the extra cost is nearly negligible. The improved","pith_inferences":["Our inference: because the benefit comes from a better-conditioned starting point, the same AOL rescaling likely transfers to other iterative orthogonalization schemes—higher-order Newton-Schulz variants, Chebyshev-type schedules, and non-square matrices—where a cheap, reuse-friendly preconditioner could similarly accelerate convergence.","Our inference: the appendix shows that polynomial coefficients tuned for Frobenius-normalized Newton-Schulz are suboptimal when combined with AOL. Retuning the coefficients for the preconditioned start could plausibly remove a second iteration or push polar error further down, a natural and testable follow-up that the paper leaves open.","Our inference: the Gershgorin-coherence interpretation suggests AOL rescaling actively shrinks updates for feature directions with high redundancy; this hints at a robustness property against feature collapse that goes beyond runtime, though the paper only gestures at it and does not claim it as a demonstrated effect.","Our inference: the descent-direction theorem is proven for the exact polar factor of the rescaled gradient; with a truncated four-iteration Newton-Schulz approximation, the guarantee transfers only approximately, so the practical safety of the method ultimately rests on how close the four-iteration approximation is to the exact polar factor in realistic training regimes."],"forward_implications":["Muon-family optimizers can adopt Turbo-Muon as a drop-in replacement, replacing the initial Frobenius normalization with AOL rescaling and reusing the Gram matrix, to gain about a 20 percent reduction in orthogonalization cost with no loss of final loss or accuracy.","End-to-end training time should improve by a few percent in regimes where orthogonalization is a visible fraction of each step (medium batch sizes, non-sharded training), with the largest gains on large matrices where Newton-Schulz iterations dominate.","The polar-error improvement grows with matrix size, so the method becomes more beneficial as layers widen, directly targeting the regime where exact SVD is impractical and Newton-Schulz is most expensive.","The descent-direction guarantee means the preconditioned update cannot diverge from the true gradient direction; thus the usual stability arguments for Muon carry over to Turbo-Muon even when many Newton-Schulz iterations are used.","Practitioners who need higher orthogonalization accuracy can keep five iterations and obtain lower polar error than the usual five-iteration baselines, or trade the saved iteration for other computational savings during training."],"fun_headline_variants":["Turbo-Muon: better start, fewer steps, same accuracy","Preconditioning lets Muon skip a step, no tuning, same accuracy","Cached Gram matrix cuts Muon overhead, no tuning needed","Muon's Newton-Schulz loses a step with Turbo-Muon preconditioner","Save one Newton-Schulz pass: Turbo-Muon drop-in speedup"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a same-budget baseline—Muon or Muon+ run with four Newton-Schulz iterations under the same truncated coefficient schedule—would be meaningfully worse in polar error and final loss; the paper's comparisons always give baselines one extra iteration, so the attribution of the speedup to AOL preconditioning depends on this unshown comparison.","fun_headline_variants_meta":{"raw":{"variants":["Turbo-Muon: better start, fewer steps, same accuracy","Preconditioning lets Muon skip a step, no tuning, same accuracy","Cached Gram matrix cuts Muon overhead, no tuning needed","Muon's Newton-Schulz loses a step with Turbo-Muon preconditioner","Save one Newton-Schulz pass: Turbo-Muon drop-in speedup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":2859,"prompt_tokens":751,"completion_tokens":2108,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":2008}},"tokens_in":495,"tokens_out":2108,"duration_ms":13350,"temperature":1.0,"reasoning_tokens":2008,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T18:33:17.602415+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Muon+ with exactly four Newton-Schulz iterations using the same truncated coefficient schedule as Turbo-Muon, and compare polar error on random and heavy-tailed matrices plus end-to-end validation loss on the paper's GPT and CIFAR-10 speed-runs; if the four-iteration baseline matches Turbo-Muon's loss and polar error, the central claim that AOL enables dropping the iteration is false.","supporting_citations":[],"review_version":1}