{"id":"b72e1c01-b2a7-487d-90dc-783959d7f189","arxiv_id":"2605.23061","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"SF-NorMuon is a new schedule-free spectral optimizer that closes the gap with tuned AdamW on 125M-772M parameter models across 1-8x Chinchilla horizons while providing stationarity guarantees.","lead":"The paper introduces SF-NorMuon, a schedule-free spectral optimizer that matches or exceeds tuned AdamW performance on large language models using one hyperparameter configuration across varying training horizons. This removes the need to commit to a fixed training length in advance, potentially simplifying practical model training workflows.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"The reader's weakest assumption targets the theoretical justification, but the headline performance claim is empirical and does not logically require that particular theoretical condition to be true. The provided abstract and claim description contain no evident flaw that would invalidate the reported results.","tokens_in":1672,"tokens_out":254,"duration_ms":19725,"concrete_test":"Reproduce the 125 M and 772 M runs using the exact hyperparameter values stated for SF-NorMuon and the per-horizon AdamW baselines; confirm that the single SF-NorMuon configuration is used unchanged across all 1–8× horizons and that the performance comparison holds within reported variance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests on SF-NorMuon with one fixed hyperparameter set matching or beating per-horizon-tuned AdamW across the reported model sizes and horizons. The theoretical stationarity result and the weight-decay-at-fast-iterate condition are presented as supporting analysis rather than as the load-bearing premise for the performance numbers. No internal inconsistency, missing control, or unstated assumption that would falsify the reported matching/exceeding behavior is visible from the given material.","agreement_with_reader":"disagree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes SF-NorMuon, a schedule-free spectral optimizer, claiming that a single hyperparameter configuration matches or exceeds per-horizon-tuned AdamW on 125M and 772M parameter language models across 1--8× Chinchilla horizons. It further states a stationarity guarantee for schedule-free spectral dynamics and identifies weight decay applied at the fast iterate as essential for long-horizon stability, enabling horizon-independent high-quality checkpoints.","tokens_in":1765,"tokens_out":393,"duration_ms":21203,"significance":"If the empirical matching holds with the reported single-configuration robustness and the stationarity result is non-vacuous, the work would meaningfully advance practical anytime optimization by reducing horizon-dependent retuning costs and supporting continual learning. The theoretical identification of the weight-decay placement provides a concrete design principle that could guide further schedule-free methods.","major_comments":[{"comment":"Abstract: the central empirical claim (SF-NorMuon matches or exceeds tuned AdamW with one fixed hyperparameter set across two model sizes and multiple horizons) is stated without reference to any table, figure, ablation, or error analysis, preventing assessment of effect size, variance, or whether the result is load-bearing for the 'closes this gap' conclusion.","section":"Abstract"},{"comment":"Abstract (theoretical analysis section): the stationarity guarantee and the claim that weight decay at the fast iterate is 'essential for long-horizon stability' are asserted without any displayed equations, assumptions, or derivation outline, so it is impossible to verify whether the condition is derived or imposed by construction.","section":"Abstract"}],"minor_comments":[],"recommendation":"uncertain","confidential_remarks":"Only the abstract was supplied in the review package; the full manuscript text referenced in the query was not accessible, which directly limits verification of the load-bearing empirical and theoretical claims."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their comments on the abstract. We respond to each major comment below and indicate where revisions will be made.","responses":[{"response":"We agree the abstract would be clearer with explicit pointers to the supporting evidence. The performance comparisons (including means and standard deviations over three random seeds) appear in Section 4, Tables 1–2 and Figures 2–4; hyperparameter robustness and weight-decay ablations are in Section 5. In the revision we will add concise parenthetical citations to these results within the abstract.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the central empirical claim (SF-NorMuon matches or exceeds tuned AdamW with one fixed hyperparameter set across two model sizes and multiple horizons) is stated without reference to any table, figure, ablation, or error analysis, preventing assessment of effect size, variance, or whether the result is load-bearing for the 'closes this gap' conclusion."},{"response":"The stationarity guarantee is stated and proved as Theorem 3.1 (with the key assumption of weight decay applied to the fast iterate). The necessity of this placement is shown by a counter-example in Appendix B when weight decay is instead applied to the slow iterate. The abstract summarizes the result at high level; we will add a reference to Theorem 3.1 in the revised abstract.","revision_made":"yes","referee_comment":"[Abstract] Abstract (theoretical analysis section): the stationarity guarantee and the claim that weight decay at the fast iterate is 'essential for long-horizon stability' are asserted without any displayed equations, assumptions, or derivation outline, so it is impossible to verify whether the condition is derived or imposed by construction."}],"tokens_in":1323,"tokens_out":388,"duration_ms":15673,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"SF-NorMuon is the main takeaway: the paper introduces a schedule-free spectral optimizer that, using a single hyperparameter configuration, matches or exceeds horizon-tuned AdamW on 125M and 772M parameter language models over 1x to 8x Chinchilla horizons. It also supplies a stationarity proof for the underlying dynamics and flags weight decay on the fast iterate as important for long-horizon stability. This directly targets the re-tuning burden that comes with fixed training horizons. The combination of schedule-free methods with spectral optimization into a named method is the concrete new piece, and it improves on the noted shortcomings of SF-AdamW. The empirical results on two model sizes and multiple horizons stand out as the practical strength, showing that high-quality checkpoints can be obtained without committing to a horizon in advance. The theory adds grounding by linking the dynamics to stationarity and isolating one role for weight decay. Soft spots are limited but real. The abstract gives little on ablations, implementation specifics for the spectral component, or error analysis, so it is not yet clear how much this differs from prior momentum or spectral baselines or how sensitive the matching is to other training choices. The stationarity result and the weight-decay condition would need checking in the full derivation to see whether they are tight or mainly supportive. The tested scales are reasonable but not frontier. This paper is for ML practitioners and engineers who train language models under variable data or compute budgets and want to cut down on horizon-specific tuning. Readers working on anytime or continual training setups would find the empirical claim useful. It deserves peer review because the central performance claim is concrete and testable, the theory is present, and the problem it addresses is common in real workflows.","headline":"SF-NorMuon delivers schedule-free training that matches tuned AdamW across horizons on 125M and 772M models with one fixed hyperparameter set.","tokens_in":2245,"tokens_out":421,"would_cite":false,"duration_ms":16808,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"Schedule-free spectral optimizer (SF-NorMuon) with polar updates and implicit averaging; no overlap with J-cost, φ-ladder or distinction-forcing.","alignment":"orthogonal","rationale":"Paper's core machinery (schedule-free averaging yielding η_eff ~ 1/t, polar(M) steepest descent under spectral norm, weight-decay at fast iterate z_t for boundedness, eO(T^{-1/4}) stationarity) lives entirely in ML optimization. It neither invokes nor parallels any RS primitive (J(x)=½(x+x^{-1})−1, φ fixed-point, 8-tick periodicity, parameter-free constant derivation). No theorem in the paper contradicts an RS result; the domain (horizon-free neural-net training) is simply outside RS scope.","tokens_in":66097,"confidence":"high","tokens_out":179,"duration_ms":8095,"cache_read_input_tokens":38528,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Schedule-free spectral optimization matches tuned AdamW on 125M and 772M parameter models using one hyperparameter configuration across 1-8x Chinchilla horizons.","keywords":["schedule-free optimization","spectral optimizer","language model training","anytime training","weight decay","AdamW","Chinchilla horizons","neural network optimizers"],"falsifier":"Training runs of SF-NorMuon on 772M parameter models over 8x Chinchilla horizons that exhibit instability or performance drops when weight decay is removed from the fast iterate.","tokens_in":2591,"feed_emoji":"⚙️","tokens_out":511,"duration_ms":17447,"temperature":0.7,"pith_summary":"Standard neural network training ties learning-rate schedules to a fixed horizon, creating path dependence and forcing costly re-tuning when data volume changes. Schedule-free methods remove explicit schedules to address this, but prior versions like SF-AdamW still fall short of well-tuned AdamW. The paper introduces SF-NorMuon, a schedule-free spectral optimizer that closes the gap by matching or exceeding tuned AdamW performance on 125M and 772M parameter language models with a single hyperparameter setup. This holds across training lengths from 1x to 8x Chinchilla horizons. The approach lets practitioners obtain high-quality checkpoints at any point without committing to a total horizon in advance.","feed_headline":"One config matches tuned AdamW across 1-8x horizons on LLMs","feed_subtitle":"SF-NorMuon delivers competitive checkpoints at any training stage for 125M and 772M models without horizon-specific tuning.","key_machinery":"SF-NorMuon, a schedule-free spectral optimizer that applies weight decay at the fast iterate within schedule-free spectral dynamics.","core_discovery":"SF-NorMuon is a schedule-free spectral optimizer that, with a single hyperparameter configuration, matches or exceeds tuned AdamW on 125M and 772M parameter language models across 1 to 8 times Chinchilla horizons. The work proves a stationarity guarantee for schedule-free spectral dynamics and shows that weight decay applied at the fast iterate is essential for long-horizon stability. This removes the need to commit to a training horizon upfront while preserving competitive performance.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["SF-NorMuon matches tuned AdamW on 125M and 772M models","Single config works across 1-8x horizons for SF-NorMuon","SF spectral optimizer guarantees stationarity without schedules","Weight decay at fast iterate enables long horizon SF stability"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Weight decay applied at the fast iterate is essential for maintaining long-horizon stability in schedule-free spectral dynamics.","fun_headline_variants_meta":{"raw":{"variants":["SF-NorMuon matches tuned AdamW on 125M and 772M models","Single config works across 1-8x horizons for SF-NorMuon","SF spectral optimizer guarantees stationarity without schedules","Weight decay at fast iterate enables long horizon SF stability"]},"model":"grok-4.3","cost_usd":0.008154,"raw_usage":{"total_tokens":3698,"prompt_tokens":658,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":81537000,"prompt_tokens_details":{"text_tokens":658,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2975,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":658,"tokens_out":65,"duration_ms":23307,"temperature":1.0,"reasoning_tokens":2975,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-25T05:36:00.727202+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Training runs of SF-NorMuon on 772M parameter models over 8x Chinchilla horizons that exhibit instability or performance drops when weight decay is removed from the fast iterate.","supporting_citations":[],"review_version":1}