{"work":{"id":"4a43859c-1056-4934-bc0b-1d3e33bbc5b0","openalex_id":null,"doi":null,"arxiv_id":"2509.15816","raw_key":null,"title":"On the Convergence of Muon and Beyond","authors":null,"authors_text":"Da Chang, Yongxiang Liu, and Ganzhao Yuan","year":2025,"venue":"cs.LG","abstract":"The Muon optimizer has demonstrated remarkable empirical success in handling matrix-structured parameters for training neural networks. However, a significant gap remains between its practical performance and theoretical understanding. Existing analyses show that the Muon variants achieve only a suboptimal ergodic convergence rate of $\\mathcal{O}(T^{-1/4})$ in stochastic non-convex settings, where $T$ denotes the number of iterations. To study the theoretical limits of Muon, we analyze two momentum-based variance-reduced variants: the one-batch Muon-MVR1 and the two-batch Muon-MVR2. We provide the first rigorous proof that, under \\textbf{horizon-free} learning-rate schedules, variance reduction enables Muon-MVR2 to attain the optimal anytime convergence rate $\\widetilde{\\mathcal{O}}(T^{-1/3})$, matching the lower bound for this problem class. Under the Polyak--\\L{}ojasiewicz (PL) condition, we establish anytime guarantees for Muon-MVR1 and Muon-MVR2: they attain best-iterate rates of $\\widetilde{\\mathcal{O}}(T^{-1/4})$ and $\\widetilde{\\mathcal{O}}(T^{-1/3})$ for the expected square-root suboptimality, and, given an additional uniform gradient bound along the iterates, achieve last-iterate rates of $\\mathcal{O}(T^{-1/4})$ and $\\mathcal{O}(T^{-1/3})$ for the objective gap, respectively. Experiments on CIFAR-10 and C4 support the practical effectiveness of the proposed variance-reduced Muon variants. Code is available at \\href{https://github.com/MaeChd/MUON-MVR}{Muon-MVR} Codebase.","external_url":"https://arxiv.org/abs/2509.15816","cited_by_count":null,"metadata_source":"pith","metadata_fetched_at":"2026-07-04T07:29:38.596450+00:00","pith_arxiv_id":"2509.15816","created_at":"2026-05-11T05:00:56.667856+00:00","updated_at":"2026-07-04T07:29:38.596450+00:00","title_quality_ok":false,"display_title":"On the Convergence of Muon and Beyond","render_title":"On the Convergence of Muon and Beyond"},"hub":{"state":{"work_id":"4a43859c-1056-4934-bc0b-1d3e33bbc5b0","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":16,"external_cited_by_count":null,"distinct_field_count":3,"first_pith_cited_at":"2026-03-10T02:12:24+00:00","last_pith_cited_at":"2026-06-19T16:36:53+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-22T12:20:09.264504+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":3},{"context_role":"method","n":1}],"polarity_counts":[{"context_polarity":"background","n":3},{"context_polarity":"use_method","n":1}],"runs":{},"summary":{},"graph":{},"authors":[]}}