{"id":"386e93a6-9567-40e2-bc22-52aecfd06966","arxiv_id":"2608.11612","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Dion3 combines Gram-based Newton-Schulz, symmetric GEMM kernels, megabatched communication, and fractional row selection so Muon-class optimizers run up to about 6x faster per step while matching or slightly improving loss.","lead":"Dion3 is a rebuilt version of the Muon optimizer that makes each optimization step about six times cheaper by computing the same orthogonalization on a smaller matrix, using custom GPU kernels, and updating only a fraction of the weight rows at each step. It also reports slightly lower validation losses than the standard optimizer in the tested settings, so it matters for anyone training large models who wants Muon's benefits without its cost.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central quality claim that fractional row selection is loss-neutral or better rests on single-seed, single-dataset runs with hyperparameters tuned on the same sweep that reports the best losses; the paper itself flags that this improvement is unexpected and ungeneralized.","rationale":"The reader's weakest_assumption is that fractional row selection is a benign approximation across architectures, data mixtures, and scales. That is exactly the load-bearing concern I identify, and the manuscript itself provides the evidence for it: Section 8.1 says the improvement is unexpected, Section 9 says generalization needs further work, and Appendix A tests Gram Newton-Schulz with f=1 only, not the fractional update. The paper has real independent support for the non-quality claims: Theorem 2 is a parameter-free mathematical equivalence; the instability analysis in Appendix B gives a concrete mechanism and a restart schedule; megabatching has a plausible communication-round argument plus measured NCCL microbenchmarks. Those parts are not threatened by my concern. The quality claim, however, is the one that makes the abstract's 'improves on the loss' statement, and it rests on single-seed runs on one dataset (ClimbMix) with f and eta selected from the same sweep that reports the best numbers. The learning-rate transfer rule eta*sqrt(f)=0.01 is a heuristic derived from that same 1B sweep and then applied to 3B-14B runs, so those larger-scale results are not independent confirmations. The abstract's statement that Dion3 improves on Dion in both speed and performance is never benchmarked anywhere in the paper; Galley-style, this is an unsupported comparative claim, though it is about a concurrent line of work rather than the central contribution. None of this is fatal because the central speed claim and equivalence proof stand. The right verdict is CONDITIONAL: require a multi-seed, held-out-hyperparameter evaluation on at least one additional data mixture, or a softened abstract. My agreement_with_reader is 'agree' because the reader's weakest_assumption and my concern are the same claim, though I would put slightly more weight on the absence of any benchmark against Dion as an unfulfilled part of the abstract's strongest claim.","tokens_in":31053,"tokens_out":2107,"duration_ms":18356,"concrete_test":"Run the 1B ClimbMix comparison with repeated seeds (at least 3) and a held-out hyperparameter protocol: tune f and eta on a small training subset or on 10B tokens, then evaluate the chosen configuration on the full 100B-token run. Report the mean and spread of the final validation-loss gap between Dion3 (f=1/4, eta=0.02) and fully-tuned NorMuon (eta=0.01). If the mean gap is not robustly negative after this protocol, the 'improves on the loss' part of the claim fails and the abstract should be revised to 'matches loss.' Also useful, but secondary: benchmark Dion3 against Dion on identical hardware and report the validation-loss and step-time delta promised in the abstract.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that Dion3 'matches or improves on the loss achieved by Muon' while cutting step time. The speed half is well supported by benchmarks and a clean FLOP analysis. The quality half is the load-bearing weak point. Algorithm 4's fractional row selection changes the optimization trajectory, and the paper's own Section 8.1 concedes the loss improvement is 'a small but pleasant surprise' and Section 9 states 'Further work is needed to determine how widely this improvement generalizes.' The evidence for quality is: (i) a 1B sweep on a single dataset (ClimbMix) where the fraction f and learning rate eta are tuned on the same grid that reports the best losses, with no repeated seeds; (ii) 3B-14B runs on 10B tokens, again single seed, single dataset, with eta chosen by the transfer rule eta*sqrt(f)=0.01 derived from that same 1B ClimbMix sweep; and (iii) no benchmarked comparison against the Dion baseline named in the abstract, despite the abstract claiming Dion3 'improves on Dion ... in both speed and performance.' The reader's verdict is CONDITIONAL for the same reason, and I agree that the issue is correctable rather than fatal. The rest of the paper (Theorem 2 equivalence, Gram Newton-Schulz stability analysis, restart schedule, megabatching, kernels) has independent support and does not depend on the fractional update improving loss; if the 1B/14B loss gap vanished under proper evaluation, the headline should soften to 'matches Muon loss with up to 6x faster steps' on the tested settings.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Dion3, a revision of the Muon optimizer that targets the cost of the Newton–Schulz orthogonalization step at four levels: a Gram-matrix reformulation of Newton–Schulz (Gram Newton–Schulz), custom symmetric GEMM kernels written in CuteDSL, a fractional row-selection update rule with error feedback, and megabatched communication for distributed training. The authors prove an algebraic equivalence between standard Newton–Schulz and the Gram iteration (Theorem 2), provide a stability analysis with a restart schedule, and report experiments on 1B–14B dense transformers trained on ClimbMix. The headline claims are that Dion3 matches or improves the loss of Muon/NorMuon while reducing optimizer step time by up to 6x, and that the update rule improves on Dion in both speed and performance.","tokens_in":31379,"tokens_out":11435,"duration_ms":112130,"significance":"If the speed results hold, the practical significance is high: reducing Muon's optimizer-step overhead from roughly 26x AdamW to about 4x at 7B scale would make orthogonal optimizers substantially more accessible in distributed training. The paper has clear strengths: the Gram Newton–Schulz equivalence is a clean algebraic result; the FLOP accounting is careful (up to a typo noted below); the stability analysis, restart schedule, and benchmark methodology with interquartile ranges are well specified; and the authors provide open-source packages for the optimizer and kernels. The weak link is the loss-quality component of the central claim, which currently rests on single-seed, single-dataset experiments, and the abstract's claim of improvement over Dion is not backed by any direct Dion baseline experiment.","major_comments":[{"comment":"The claim that Dion3 'matches or improves on the loss achieved by Muon' is supported only by single-seed runs on a single data mixture (ClimbMix) with dense transformers at 1B–14B, and the hyperparameters (f, η) are selected from the same sweep whose best values are then reported (Figure 3). No repeated seeds or confidence intervals are given for the final losses in Figure 3, Figure 4, or Table 2, and the larger-scale runs in Table 2 use the learning rate from the 1B transfer rule η√f = 0.01 without per-scale validation. Section 9 itself states that 'Further work is needed to determine how widely this improvement generalizes.' Because this loss-quality claim is load-bearing for the abstract, the paper should either add repeated-seed or multi-dataset evidence, or explicitly restrict the claim to the tested setting.","section":"§8.1, Fig. 3, Table 2"},{"comment":"The abstract states that the update rule 'improves on Dion ... in both speed and performance,' and Section 3.2 says the update rule is 'simpler and faster than Dion,' but no experiment in the paper compares Dion3 against the Dion baseline. The comparisons to NorMuon/Muon in Section 8 do not support a claim about Dion. Add a direct Dion baseline to the speed and loss experiments, or remove the performance claim from the abstract and intro.","section":"Abstract, §3.2"},{"comment":"The FLOP comparison in the text says 'For a typical Muon application (T = 5, α = 43), it saves 55% of the FLOPs used by standard Newton-Schulz with symmetric GEMMs, or 68% compared to a typical implementation without symmetric GEMMs.' Plugging α = 43 into the paper's own formulas gives T(3α+1) = 650 and (4T+3α−3) = 146, i.e., a 77.5% saving, not 55%; the stated 55%/68% correspond to α = 4. Please correct the aspect-ratio typo and recompute any downstream percentages.","section":"§4.1"}],"minor_comments":[{"comment":"The text reports 'overall speedups of 3.6× and 6.5×,' while Figure 6 and the abstract say '6×.' Please harmonize these numbers so the abstract, figure, and body agree.","section":"§8.2, Fig. 6"},{"comment":"Section 6 says the distributed implementation selects the top-f fraction of rows from each shard, with global selection as an option, but Section 8.1 does not state which selection mode was used in the reported runs. Please specify this for reproducibility.","section":"Algorithm 4, §8.1"},{"comment":"The heatmap reports one number per (f, η) cell with no indication of the number of runs or run-to-run variability. With single runs, a bolded 'best' cell is not evidence of a significant difference from neighboring cells; please state the number of seeds per cell or add error bars.","section":"Figure 3"},{"comment":"The restart schedule is derived under the empirical bound that spurious negative eigenvalues are at least −4 × 10⁻⁴. Please add a sentence noting that this is an empirical assumption that should be re-validated if the precision, architecture, or coefficient set changes.","section":"Appendix B.2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.LG and the systems/optimization contributions are real. The most conspicuous gap is the missing Dion baseline in a paper whose abstract explicitly claims improvement over Dion; I would treat that as a blocking issue for the current wording. The single-seed loss-quality evidence should also be strengthened or the claim narrowed before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about Dion3. First, the speed story is real: the Gram Newton-Schulz reformulation is a clean piece of math (Theorem 2), the restart-based stability fix is a genuine contribution, and the benchmark reporting (median with IQR, explicit FLOP counts) is careful. Across the tested range, the claimed speedups look credible and likely reproducible. Second, the 'matches or improves loss' headline is much weaker than the abstract suggests. The loss improvement over NorMuon comes from single-seed runs on one dataset (ClimbMix), and the selection fraction f and learning rate were tuned on the same grid that produced the best final losses. The authors themselves call the improvement a 'small but pleasant surprise' and say more work is needed to know how widely it generalizes, but the abstract drops that caveat and asserts the claim as fact.\n\nWhat is actually new: the packaged system. The row-subsampled update with error feedback, the restart schedule that makes Gram Newton-Schulz usable in half precision, the symmetric CuteDSL kernels, and megabatching are not all present in earlier work. The paper is also honest about lineage: Section 3.1 says Gram Newton-Schulz is closely akin to an appendix of their own Polar Express paper, and the symmetric GEMM idea is cited to prior work. That kind of explicit credit is good to see.\n\nPlaces where I would push back. The abstract says Dion3 'improves on Dion in both speed and performance,' but there is no benchmarked comparison against Dion anywhere in the paper. Maybe that is a length or timing constraint, but as it stands the claim is unsupported. The row-selection heuristic (L1 norm) is not rigorously compared to alternatives; the text says random selection was 'not much worse' in initial experiments, but no numbers are shown. And the quality half of the central claim is genuinely under-powered: no repeated seeds, one data mixture, and the larger-scale runs (3B-14B) use the learning-rate transfer rule derived from the same single 1B sweep. If the loss gap disappears under proper evaluation, the headline would need to soften to 'matches NorMuon loss with up to 6x faster steps,' but the speed and equivalence results would still stand.\n\nBottom line: this deserves a serious referee. The algorithmic core is strong, the systems engineering is impressive, and the empirical weakness is correctable by adding seeds, datasets, and a Dion baseline. I would send it to review, and in the review I would ask for exactly those additions. I would also cite the Gram Newton-Schulz implementation if I were working on orthogonal optimizers; the stability analysis alone is worth pointing to.","headline":"The speed side of Dion3 is solid and worth taking seriously; the 'matches or improves loss' claim is a single-seed, single-dataset result that the abstract overstates.","tokens_in":31948,"tokens_out":1611,"would_cite":true,"duration_ms":19280,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dion3 shows a full-stack redesign of Muon's orthogonalization step can match or beat its loss while cutting optimizer step time up to 6x.","keywords":["Muon optimizer","Newton-Schulz orthogonalization","Gram matrix","fractional row selection","error feedback","distributed training","symmetric GEMM kernels","LLM pretraining"],"falsifier":"Train a 7B dense transformer on a second data mixture, such as FineWeb-Edu, with Dion3 at f=1/4 versus a tuned NorMuon, running several seeds; if Dion3's final validation loss is not at most NorMuon's, the claim that Dion3 matches or improves the loss fails to transfer outside the paper's setting.","tokens_in":30807,"feed_emoji":"⚡","tokens_out":5231,"duration_ms":52360,"temperature":0.7,"pith_summary":"Dion3 is a redesigned version of the Muon optimizer, which trains large language models by orthogonalizing momentum matrices. The paper tries to show that the expensive cubic-time Newton-Schulz orthogonalization step can be made cheap at every level: reformulating it on the small symmetric Gram matrix, accelerating that with custom symmetric GEMM kernels, and orthogonalizing only a fraction of the momentum matrix's rows via a simple selection rule with error feedback. If the paper is right, practitioners can keep Muon's convergence behavior in dense and sparse transformers while paying a small fraction of its optimizer overhead, with step time down by up to 6x. The paper also reports a mild quality gain: at 14B parameters, selecting 1/4 of rows each step lowers validation loss by 0.027 and raises downstream accuracy by 0.7 points relative to a tuned NorMuon baseline.","feed_headline":"Dion3 matches Muon's loss, cuts optimizer time 6x","feed_subtitle":"Fractional row selection plus Gram-matrix Newton-Schulz drops Muon's 26x AdamW overhead to 4x.","key_machinery":"The load-bearing mechanism is the identity polar(X) = (XX^T)^(-1/2)X combined with the factorization of odd polynomials p_t(x)=x h_t($x^{2}$). That factorization lets Newton-Schulz run on the small symmetric Gram matrix R_t = X_t X_t^T while keeping a cumulative Q_t, producing the same output as standard Newton-Schulz in exact arithmetic. The other half is the row-selection update rule: pick k=ceil(fn) rows of M with largest L1 norm, polarize the submatrix with Gram Newton-Schulz, update only those rows, and apply error feedback by decaying only selected momentum rows so unselected rows can build up and be chosen later. A restart after the second iteration stabilizes the Gram iteration against spurious negative eigenvalues introduced by half-precision arithmetic.","core_discovery":"The central claim is that Muon's polarization step can be replaced by a mathematically equivalent Gram-matrix iteration plus a coarser, cheaper update rule without losing Muon's optimization quality. Writing polar(X)=(XX^T)^(-1/2)X, Gram Newton-Schulz iterates on the n by n Gram matrix and needs only two rectangular matrix multiplications instead of the usual ten, while the fractional update rule selects k=ceil(fn) rows of largest L1 norm, orthogonalizes just those rows, updates only those weights, and dampens only those momentum rows via error feedback. The paper's experiments show that for 1B-14B dense transformers trained on ClimbMix, tuned fractional updates track below NorMuon's loss throughout training, and the optimizer step time falls by up to 6x, with communication volume scaling as the fraction f. It further establishes a transfer rule eta' = eta/$\\sqrt$(f) that keeps the Frobenius norm of the update matched to full Muon.","pith_inferences":["The improved final loss at f<1 suggests row selection acts as an implicit regularizer; if so, the same trick could be ported to other cubic-time update rules that orthogonalize or whiten momentum, but the paper does not test that.","Because the paper reports single-seed runs on one data mixture, the quality edge at 14B (-0.027) may be within run-to-run noise; a multi-seed replication on a second data mixture would settle whether the improvement is systematic.","Communication savings grow linearly in 1/f, so in settings where weights are sharded across data-center networks rather than local NVLink interconnects, the end-to-end speedup could exceed the 6x reported; the paper's benchmarks use local interconnects."],"forward_implications":["Using Dion3 at f=1/4 as a drop-in replacement for Muon should give essentially the same or slightly lower training loss while cutting optimizer step time by about 3.7x on top of Gram Newton-Schulz and kernel gains, for large dense models.","The eta*sqrt(f) approximately 0.01 rule means users do not need a full learning-rate sweep for each new fraction; set eta' = eta/sqrt(f).","In communication-bound regimes, megabatching collapses communication rounds to a constant and fractional selection reduces communication volume by a factor of 1/f, so the benefits should be largest when many small matrices are sharded across many GPUs.","Gram Newton-Schulz applies to any Muon variant, including NorMuon and MoE training, so improvements to the orthogonalization step compose with other optimizers that use Newton-Schulz as a black box."],"supporting_citations":[{"why":"Defines Muon and the polar/Newton-Schulz update that Dion3 accelerates.","marker":"[19]"},{"why":"Introduces Dion's low-rank compressed orthogonalization and error feedback, which Dion3 simplifies into row selection.","marker":"[2]"},{"why":"NorMuon, the normalized baseline that Dion3 is compared against in the quality experiments.","marker":"[23]"},{"why":"Supplies the Polar Express polynomial coefficients used by Gram Newton-Schulz.","marker":"[3]"},{"why":"ClimbMix, the data mixture used in all quality comparisons and loss measurements.","marker":"[10]"},{"why":"Identifies all-to-all layer-sharding communication as a central scaling obstacle that megabatching addresses.","marker":"[11]"},{"why":"Trion's crude subspace selection shows that error feedback can compensate for poor low-rank approximations, motivating the simpler row-selection rule.","marker":"[29]"}],"fun_headline_variants":["Dion3: 6x faster optimizer, same loss as Muon","Gram trick cuts Muon cost 6x, keeps quality","Fractional rows: Dion3 beats Muon speed, matches loss","Dion3 shaves optimizer time 6x, no loss hit","Muon overhead slashed 6x with Dion3's partial updates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quality claim stands on the premise that updating only a fraction of momentum rows each step, with error feedback for the rest, is a harmless or slightly helpful approximation across architectures, data, and scales; the experiments cover one data mixture and dense transformers from 1B to 14B.","fun_headline_variants_meta":{"raw":{"variants":["Dion3: 6x faster optimizer, same loss as Muon","Gram trick cuts Muon cost 6x, keeps quality","Fractional rows: Dion3 beats Muon speed, matches loss","Dion3 shaves optimizer time 6x, no loss hit","Muon overhead slashed 6x with Dion3's partial updates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1506,"prompt_tokens":940,"completion_tokens":566,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":472}},"tokens_in":556,"tokens_out":566,"duration_ms":5542,"temperature":1.0,"reasoning_tokens":472,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:33:59.448966+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a 7B dense transformer on a second data mixture, such as FineWeb-Edu, with Dion3 at f=1/4 versus a tuned NorMuon, running several seeds; if Dion3's final validation loss is not at most NorMuon's, the claim that Dion3 matches or improves the loss fails to transfer outside the paper's setting.","supporting_citations":[{"cited_title":"Muon: An optimizer for hidden layers in neural networks","cited_arxiv_id":null,"evidence_quote":"Defines Muon and the polar/Newton-Schulz update that Dion3 accelerates."},{"cited_title":"NorMuon: Making Muon more efficient and scalable","cited_arxiv_id":null,"evidence_quote":"NorMuon, the normalized baseline that Dion3 is compared against in the quality experiments."},{"cited_title":"The Polar Express: Optimal Matrix Sign Methods and their Application to the Muon Algorithm","cited_arxiv_id":null,"evidence_quote":"Supplies the Polar Express polynomial coefficients used by Gram Newton-Schulz."},{"cited_title":"Nemotron-CLIMB: Clustering-based Iterative Data Mixture Bootstrapping for Language Model Pre- training","cited_arxiv_id":null,"evidence_quote":"ClimbMix, the data mixture used in all quality comparisons and loss measurements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Identifies all-to-all layer-sharding communication as a central scaling obstacle that megabatching addresses."},{"cited_title":"Trion: FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of LLMs","cited_arxiv_id":null,"evidence_quote":"Trion's crude subspace selection shows that error feedback can compensate for poor low-rank approximations, motivating the simpler row-selection rule."}],"review_version":1}