{"id":"f3925eeb-1e3a-4419-9146-d234b7b9d8a8","arxiv_id":"2608.09763","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"GO-MUON combines K-FAC-style second-moment factors with Muon's polar map in a matched spectral oracle, and a four-step refresh schedule trades factor freshness for compute without claiming denoising.","lead":"GO-MUON is a new optimizer that wraps Muon's polar update in a data-dependent spectral geometry, replacing the plain orthogonal projection with a curvature-aware one. It shows small training-loss gains on character language models and large grokking speedups on modular addition, though the marquee speedups come from single-seed screens.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Grokking speedups rest on a single seed per modulus with learning rates selected on the same thresholded endpoint; multi-seed, LR-matched replication is required before the headline claim can stand.","rationale":"The reader's CONDITIONAL verdict is appropriate. Theorem 1 and Propositions 5-7 are a genuine strength: they are correct, attributed to prior pullback-oracle theory, and not oversold in the body. The PTB and Tiny Shakespeare experiments are honest in reporting small training-loss improvements (0.38% and 3.71%) with three paired seeds. The place where the strongest practical claim is least secure is the modular-addition grokking screen. Two features make this the load-bearing concern rather than the factor-approximation caveat: the reported effect size is enormous (8x-20.5x speedup), so it carries the abstract and the 'done right' narrative; and the experimental design selects both learning rates on the same grokking endpoint on a single seed, while the endpoint is a discontinuous first-crossing statistic. The admitted late departure from 99% on mod 107 shows that time-to-grok is not a stable property in this setup. A multi-seed replication with learning rates fixed from a tuning seed, plus an identity-geometry control, would settle whether the speedup is due to the matched K-FAC-root geometry or to learning-rate and seed luck. This is a request for evidence, not an accusation; the paper itself flags the screen as a formal one-seed-per-modulus result and the PTB result as exploratory. The remedy is to add seeds and controls; after that the CONDITIONAL verdict could move to ACCEPT or REJECT accordingly.","tokens_in":13185,"tokens_out":12870,"duration_ms":108589,"concrete_test":"Run the Section 5.2 modular-addition comparison with at least 20 independent train/test splits per modulus (103 and 107), fixing each optimizer's learning rate on a separate tuning seed and then evaluating on fresh seeds; report median and worst-case grokking step. Include a control in which GO-MUON's OAS factors are replaced by identity with its same learning-rate and refresh schedule, equivalent to Muon at the same selected learning rate. If the median speedup over best-tuned Muon is below 2x, or the identity control matches GO-MUON, the strong grokking claim should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The theory in Section 2 is sound and honestly scoped, but the paper's load-bearing empirical claim is the modular-addition speedup in Table 3, and that claim is not yet supported. Section 5.2 uses one deterministic split and initialization seed per modulus and selects learning rates on the same grokking criterion used as the endpoint: the screen over learning rates {0.08,0.16} for GO-MUON and {0.16,0.32} for Muon selects 0.080 and 0.160, respectively, under the grokking criterion below. Time-to-grok is defined as the first of five consecutive evaluations with test accuracy at least 99%, sampled every ten updates; this is a discontinuous, thresholded statistic and is highly seed- and learning-rate-sensitive. The paper itself records that the mod-107 GO-MUON trajectory later leaves the 99% region and returns only at update 4480, so the mod-107 endpoint is not stable; the abstract's 'sustained 99%' overstates this. Because Muon's single-seed grokking delays are 2320 and 4520 updates, a different seed or learning rate could shrink or reverse the reported 8x/20.5x speedups. If the speedup is an artifact of endpoint definition and learning-rate selection, the abstract's central empirical conclusion collapses even though Theorem 1 remains true.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GO-MUON, an optimizer that marries Muon's polar update with a data-dependent spectral geometry. The central theoretical object is Theorem 1: for any SPD maps P_A, P_B, the direction D = P_B Q(P_B M P_A) P_A is an exact maximizer for the weighted spectral support problem, with value ||P_B M P_A||_*. The paper also proves a quantitative bound on the mismatch between observed-label backward factors and the softmax Fisher/GGN factor (Proposition 3), a worst-case conditioning and alignment certificate for the quarter-power graft (Proposition 5), and an exact variance–lag characterization of sparse EMA refresh (Proposition 6). Experiments cover PTB and Tiny Shakespeare character language modeling, a modular-addition grokking screen, and a lazy-refresh comparison. The abstract claims that GO-MUON reaches sustained 99% held-out accuracy at updates 290 and 220 on moduli 103 and 107, versus 2320 and 4520 for Muon.","tokens_in":13458,"tokens_out":4273,"duration_ms":34504,"significance":"The theory in Section 2 is clean and correctly scoped: Theorem 1 is a direct duality argument, the paper explicitly credits the general pullback oracle to Veprikov et al., and Proposition 6 gives an exact variance–lag characterization of sparse refresh. Proposition 3 quantifies the observed-label/Fisher mismatch, and Proposition 5 provides a worst-case conditioning certificate. These are useful and, as far as I can tell, sound. The empirical portion is considerably weaker: the headline grokking speedups rest on one seed per modulus with learning rates selected on the same thresholded endpoint, the PTB 0.38% result is explicitly exploratory, and the paper does not measure the geometry-alignment quantities that would connect the theory to the gains. With the theoretical core intact, the paper is a plausible contribution to optimizer design, but the current abstract overstates the empirical support.","major_comments":[{"comment":"The modular-addition speedup claim is not yet supported. The table reports one deterministic split and initialization seed per modulus, and the learning-rate screen over {0.08,0.16} for GO-MUON and {0.16,0.32} for Muon selects rates using the same grokking criterion that defines the endpoint (first of five consecutive evaluations at ≥99% test accuracy, sampled every ten updates). This discontinuous, thresholded statistic is highly seed- and learning-rate-sensitive, so the reported 8.0× and 20.5× speedups could be artifacts of learning-rate selection or endpoint definition. The paper should provide multi-seed replication, a pre-specified learning-rate selection rule, and a stable endpoint or full retention curves before the abstract's speedup claim can stand.","section":"§5.2, Table 3"},{"comment":"The PTB result is weakened by post-hoc learning-rate selection. The paper states that ranking the recorded GO-MUON sweep by the three-window geometric mean selects learning rate 0.080, that this ranking was introduced after the original late-window rule (which selected 0.16), and that the hash-matched comparison is exploratory. A 0.38% trajectory reduction selected in this way is not a confirmatory result; the paper should either pre-register the selection rule or report the full sweep and all paired-seed summaries for every learning rate.","section":"§5.1, Table 2"},{"comment":"The attribution of empirical gains to curvature matching requires a direct check of the weakest assumption: that OAS-shrunken observed-label backward factors approximate Fisher/GGN factors closely enough during training to be a useful preconditioner. Corollary 4 requires realizability or predictive consistency, and the paper itself notes in Section 3.1 that absolute factor convergence does not imply relative inverse-root accuracy near interpolation, while Section 2.1 states that the coordinate-drift quantity Θ_t is not measured in the experiments. Without measuring factor alignment or Θ_t, the grokking and loss differences could stem from learning-rate or other side effects rather than from the matched geometry.","section":"§3.1, Corollary 4 and §2.1, Proposition 7"},{"comment":"The abstract's 'sustained 99%' overstates the mod-107 result. The paper records that the mod-107 GO-MUON trajectory later leaves the 99% region, with below-threshold evaluations as late as update 4470 before returning at update 4480. 'First of five consecutive evaluations' is not a description of sustained accuracy; the text should be revised to state the onset of a 99% threshold condition and to report the subsequent retention profile.","section":"§5.2, Table 3 and Abstract"}],"minor_comments":[{"comment":"The phrase 'formal one-seed-per-modulus screen' is not formal in a statistical sense; a screening run with one seed is better described as a pilot or exploratory result.","section":"Abstract and §5.2"},{"comment":"The grokking-delay diagnostic subtracts 'the first update with 99% training accuracy'; please clarify whether training accuracy is evaluated on the same ten-update schedule and whether it is sensitive to the same thresholding as the test endpoint.","section":"§5.2"},{"comment":"The Tiny Shakespeare control is described as 'Muon+Q/K control' while the PTB control is 'Muon'; a sentence explaining the difference and why the same optimizer variant was not used for both corpora would improve comparability.","section":"Table 2"},{"comment":"The sentence 'Both arms record zero negative momentum-source and current-gradient alignments' should define the alignment statistic or point to where it is defined, since the statement is otherwise unverifiable.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is unusually candid about its own limitations in Section 5.1 and Section 5.2, but the abstract does not carry that caution. I would ask the authors to align the abstract with the exploratory status of the empirical claims, particularly the modular-addition speedups and the PTB trajectory reduction. The theoretical contribution is sound and the related-work section is useful; the main risk is that the empirical headline overreaches relative to the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful content here is the theory, and it is genuinely good. Theorem 1 cleanly extends Muon's polar update to a matched spectral oracle with arbitrary SPD maps, and the paper honestly attributes the general pullback to Veprikov et al. Proposition 6 is the real gem: the sparse-EMA variance–lag tradeoff is derived correctly, and the observation that lazy refresh is a compute–statistics tradeoff, not a denoising mechanism, is a genuinely nice clarification. Proposition 3's mismatch bound and the careful caveats around when observed-label factors approach the Fisher/GGN are also well done. The paper is honest about what it does not prove—Corollary 4's predictive-consistency caveat, the K-FAC approximation error, and the fact that absolute factor convergence does not imply relative inverse-root accuracy. That intellectual honesty deserves credit.\n\nThe empirical claims, however, are not at the same standard. The PTB 0.38% result is selected post hoc by ranking the same three-window summary used as the endpoint; the paper itself labels it exploratory, and the middle window is tied. The Tiny Shakespeare result is better but still small-scale. The real problem is the grokking table. One seed per modulus, learning rates selected on the same thresholded grokking criterion, and a discontinuous endpoint that the mod-107 trajectory itself violates—the abstract's 'sustained 99%' is contradicted by the admitted drop below threshold as late as update 4470. A different seed or a slightly different LR could shrink or reverse the 8x/20.5x speedups. No code is released, and there is no comparison against Mousse or FISMO, which are the closest structural variants. The stress-test note is right: this is not yet a load-bearing empirical result, even though Theorem 1 remains true.\n\nWho benefits: researchers working on structured optimizers will get real value from the oracle formulation and the sparse-EMA analysis. The paper deserves a serious referee because the theory is substantive and the empirical methodology is fixable—but the referee should be told to focus on the grokking screen and demand multi-seed, LR-matched, endpoint-stable replication. As-is, the empirical conclusions should be treated as preliminary, not as established.","headline":"Solid theory and honest scoping, but the headline grokking speedup is a single-seed exploratory screen; peer review should demand real replication before anyone cites the empirical claims.","tokens_in":14041,"tokens_out":1542,"would_cite":true,"duration_ms":13611,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","65K10"],"pacs":[],"model":"deepseek-v4-flash","headline":"A matched spectral oracle makes Muon's update exact for any supplied geometry.","keywords":["GO-MUON","spectral preconditioning","Muon optimizer","K-FAC","Fisher information","generalized Gauss-Newton","OAS shrinkage","grokking"],"falsifier":"Measure the factor mismatch $\\lVert \\hat B_B - B_F\\rVert$ on a real training trajectory of a softmax network where the model is far from the data conditional; if GO-MUON still speeds up while the mismatch is large, curvature matching is not the cause. Alternatively, replace the estimated factors with random fixed SPD maps: the oracle guarantee still holds, so any loss of the reported speedups would show that the geometry values, not the exact update, drive the results.","tokens_in":12934,"feed_emoji":"⚙️","tokens_out":6515,"duration_ms":49369,"temperature":0.7,"pith_summary":"GO-MUON's central claim is that Muon's polar update, exact for an unweighted operator-norm ball, becomes exact for any weighted spectral geometry when the polar map is wrapped with matched positive-definite maps. The paper proves that the direction $D_0 = P_B Q(P_B M P_A) P_A$ solves $\\max_{\\lVert P_B^{-1} D P_A^{-1}\\rVert_{\\mathrm{op}}\\le 1}\\langle M, D\\rangle$ with value $\\lVert P_B M P_A\\rVert_*$, independently of how $P_A$ and $P_B$ are estimated or refreshed. It then instantiates the maps from OAS-shrunk second moments, interprets the backward factor as an approximate softmax Fisher/GGN factor under predictive consistency, and shows that refreshing the geometry every four steps is a compute-statistics tradeoff rather than a denoising mechanism. If the claim holds, GO-MUON's empirical gains on character language modeling and modular arithmetic are attributable to the curvature geometry, not to approximation error in the update.","feed_headline":"Muon's update made exact for any supplied spectral geometry","feed_subtitle":"GO-MUON solves the declared curvature ball exactly, and four-step factor refresh trades statistics for compute.","key_machinery":"The load-bearing object is the matched spectral oracle: the sandwich $C=P_B M P_A$, the support polar $Q(C)$, and the map-back $D_0=P_B Q(C) P_A$. This is a matrix instance of a preconditioned-norm pullback; the change of variables $U=P_B^{-1} D P_A^{-1}$ converts the weighted ball $\\lVert P_B^{-1} D P_A^{-1}\\rVert_{\\mathrm{op}}\\le 1$ into the unit operator ball, the polar maximizes the trace pairing there, and the map-back returns to the original tangent coordinates. The implementation uses inverse fourth powers of OAS-shrunk empirical second moments as the SPD maps, which gives the operator-norm analogue of square-root K-FAC geometry, and an equal-Frobenius-norm graft $D_0\\sqrt{k}/\\lVert D_0\\rVert_F$ that preserves the oracle ray and restores Muon's update energy.","core_discovery":"The paper's central discovery is a factor-agnostic identity. For any symmetric positive-definite $P_A$, $P_B$ and momentum source $M$, define $C=P_B M P_A$ and let $Q(C)$ be a support polar, a maximizer of $\\langle C, U\\rangle$ over $\\lVert U\\rVert_{\\mathrm{op}}\\le 1$. Then $D_0=P_B Q(C) P_A$ is an exact maximizer of the matched spectral support problem, with $\\langle M, D_0\\rangle=\\lVert C\\rVert_*$. The oracle is exact for the supplied geometry by construction; choosing and estimating $P_A$ and $P_B$ decides whether the geometry is useful. GO-MUON sets $P_A=A^{-1/4}$ and $P_B=\\bar B^{-1/4}$ from OAS-regularized activation and backward-gradient second moments with a Frobenius graft, and reuses the cached factors for four updates. The paper further proves that under softmax cross-entropy the observed-label backward factor approaches the model Fisher/GGN factor as the predictive distribution approaches the data conditional, and that sparse refresh preserves slow-signal lag while increasing stationary factor variance.","pith_inferences":["Editorial inference: because Theorem 1 is factor-agnostic, the GO-MUON template could be applied to any SPD factor estimator, including gradient-Gram moments or other curvature surrogates, and still carry the same exact-oracle guarantee; the paper does not itself claim this generalization beyond the OAS K-FAC-root instantiation.","Editorial inference: the blockwise coherence created by holding geometry for four steps is an untested mechanism; one could isolate it by comparing $K=4$ refresh against a control that changes the preconditioner every step under matched per-step compute, separating the variance-lag tradeoff from trajectory effects.","Editorial inference: Proposition 3 suggests a direct diagnostic for curvature-aware optimizers: track $\\lVert \\hat B_B - B_F\\rVert$ during early training and check whether GO-MUON's gains appear only after this mismatch shrinks; if the gains appear earlier, curvature matching is not the whole story.","Editorial inference: the modular-addition grokking speedup may be reproducible with a cheaper diagonal curvature approximation; the paper's full K-FAC-root machinery would then be sufficient but not necessary for the reported effect."],"forward_implications":["GO-MUON's raw update is an exact oracle for the weighted spectral ball it declares, so its behavior on a given loss landscape is determined by the factor geometry rather than by approximation in the polarity step.","Four-step refresh changes the optimizer's statistical properties in a precise way: it nearly preserves average lag on a slowly varying geometry while raising stationary factor noise and cutting per-step compute by roughly twenty percent in the reported setting.","For softmax cross-entropy, the backward factor converges to the Fisher/GGN factor only under predictive consistency, and absolute factor convergence does not guarantee relative inverse-root accuracy near interpolation.","In the reported paired comparisons, GO-MUON lowers the geometric-mean training-loss trajectory against Muon on Tiny Shakespeare and a hash-matched Penn Treebank run, and reaches sustained ninety-nine percent held-out grokking accuracy far earlier on the corrected modular-addition screen.","Because the oracle identity is independent of factor freshness, the exactness claim survives even when the geometry is stale; only closeness to the current ball requires controlling the coordinate-map drift quantity $\\Theta_t$."],"supporting_citations":[{"why":"Supplies the Muon polar update and the unweighted spectral geometry that GO-MUON extends.","marker":"(Jordan et al., 2024)"},{"why":"Supplies the K-FAC Kronecker-factor approximation whose activation and backward factors GO-MUON estimates with observed-label moments.","marker":"(Martens & Grosse, 2015)"},{"why":"Supplies the limitation of empirical Fisher approximations that motivates the conditional consistency result and its caveats.","marker":"(Kunstner et al., 2019)"},{"why":"Supplies the general preconditioned-norm pullback oracle of which Theorem 1 is the matrix specialization.","marker":"(Veprikov et al., 2025)"},{"why":"Supplies the OAS shrinkage estimator used to turn finite-minibatch second moments into numerical SPD factors.","marker":"(Chen et al., 2010)"},{"why":"Supplies the generalized Gauss-Newton interpretation of the softmax Fisher factor when logits are natural parameters.","marker":"(Martens, 2020)"}],"fun_headline_variants":["Exact curvature oracle for any geometry: GO-MUON","GO-MUON: exact Muon update for any spectral geometry","Factor-agnostic identity makes Muon exact for any geometry","Four-step refresh trades compute for statistics in GO-MUON","GO-MUON: exact polar update for any supplied geometry"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The practical value rests on the assumption that, during training, the OAS-shrunk observed-label backward second moments approximate the model Fisher/GGN factors closely enough to be a useful preconditioner; Corollary 4 needs predictive consistency $p_\\theta \\to q$ and bounded Jacobians, which the paper notes is not true for arbitrary trajectories.","fun_headline_variants_meta":{"raw":{"variants":["Exact curvature oracle for any geometry: GO-MUON","GO-MUON: exact Muon update for any spectral geometry","Factor-agnostic identity makes Muon exact for any geometry","Four-step refresh trades compute for statistics in GO-MUON","GO-MUON: exact polar update for any supplied geometry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000536,"raw_usage":{"total_tokens":2560,"prompt_tokens":916,"completion_tokens":1644,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":1557}},"tokens_in":532,"tokens_out":1644,"duration_ms":10457,"temperature":1.0,"reasoning_tokens":1557,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:11:18.973940+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the factor mismatch $\\lVert \\hat B_B - B_F\\rVert$ on a real training trajectory of a softmax network where the model is far from the data conditional; if GO-MUON still speeds up while the mismatch is large, curvature matching is not the cause. Alternatively, replace the estimated factors with random fixed SPD maps: the oracle guarantee still holds, so any loss of the reported speedups would show that the geometry values, not the exact update, drive the results.","supporting_citations":[{"cited_title":"Advances in Neural Information Processing Systems , volume =","cited_arxiv_id":null,"evidence_quote":"Supplies the limitation of empirical Fisher approximations that motivates the conditional consistency result and its caveats."},{"cited_title":"Eldar and Alfred O","cited_arxiv_id":null,"evidence_quote":"Supplies the OAS shrinkage estimator used to turn finite-minibatch second moments into numerical SPD factors."},{"cited_title":"Journal of Machine Learning Research , volume =","cited_arxiv_id":null,"evidence_quote":"Supplies the generalized Gauss-Newton interpretation of the softmax Fisher factor when logits are natural parameters."}],"review_version":1}