{"id":"75566c8c-7311-42cc-bc68-56c827516485","arxiv_id":"2607.10959","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"WSqD uses a horizon-independent shifted inverse-square-root base plus linear cooldown to get optimal last-iterate rates in convex stochastic optimization and match or beat tuned WSD on LLM pretraining with one reused peak rate.","lead":"WSqD is a learning-rate schedule that swaps WSD's flat stable phase for a shifted inverse-square-root base, then finishes with a short linear cooldown. It supports post-hoc training extension without retuning the peak rate, with a convex last-iterate guarantee and competitive SlimPajama pretraining results.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The load-bearing transfer from convex last-iterate theory to AdamW transformers is untested at scale; small-model reuse of one base rate may not generalize.","rationale":"The reader's weakest_assumption correctly isolates the theory–practice transfer and the modest scale as the load-bearing risk. Theorem 1 is a clean last-iterate result under the stated convex assumptions and does not overclaim; the empirical design (continuation protocol, single-rate reuse, comparison to two-stage WSD and power schedule) is careful for the scale studied. No internal inconsistency or proof error is apparent from the manuscript. The concern is therefore not that the paper is wrong on its own terms, but that the strongest practical claim (horizon-free reuse of one base rate for LLM pretraining) depends on an untested transfer that the authors themselves flag. That keeps the verdict at CONDITIONAL rather than ACCEPT or REJECT: the contribution is real and usable at the reported scale, but should not yet be treated as settled practice for large-model training without the scale/optimizer check above. Confidence remains high that this is the right soft spot; no stronger load-bearing flaw (e.g., a broken proof step or contradictory experiment) is visible.","tokens_in":25183,"tokens_out":693,"duration_ms":6389,"concrete_test":"Replicate the §3.3 base-rate sweep (Figure 4) on a ≥1B model (or at least a different architecture/optimizer, e.g., Muon) for two horizons that differ by ≥2× tokens, using the same pilot-selected ηmax for WSqD and a retuned peak for WSD. If WSqD's optimal base rate shifts by more than ~20% or loses its advantage over retuned WSD at the longer horizon, the transfer assumption fails and the reuse claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central practical claim is that a single base scale c0 (or ηmax) selected on a short pilot can be reused without retuning across post-hoc horizons, because the inverse-square-root base is horizon-independent (Theorem 1, Remark 1; §3.3). That claim rests on the transfer of the convex, nonsmooth, bounded-subgradient mirror-descent analysis (Assumptions 1–3, Appendix A) to non-convex AdamW training of transformers. The paper itself labels the theory a stylized first step and notes that momentum and adaptive dynamics are not covered (§2, §5, Appendix B). Empirically, all main results use a single 213M LLaMA-style model on SlimPajama (plus limited OpenWebText2 / 124M ablations in Appendix C), with horizons only up to 60k steps. If the optimal base rate drifts with scale, architecture, or token budget the way WSD's peak does (Figure 4), the \"reuse one rate\" advantage collapses and WSqD becomes another schedule that still needs horizon-aware retuning. The theory–practice gap is therefore the single most load-bearing soft spot for the strongest claim.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes WSqD, a learning-rate schedule that replaces WSD’s constant stable phase with a shifted inverse-square-root base while retaining a final linear cooldown. In the nonsmooth stochastic convex setting, Theorem 1 shows that stochastic mirror descent under WSqD attains an O(1/√T) last-iterate rate for any fixed decay fraction α∈(0,1/2), with rate-optimal scale c0 independent of the horizon T (needed only to start the cooldown). Empirically, on 213M LLaMA-style pretraining on SlimPajama (plus limited OpenWebText2 and 124M ablations), a single base learning rate chosen on a short pilot and reused without retuning matches or improves over WSD across continuation horizons up to 60k steps, and is competitive with a power-law schedule and two-stage retuned WSD. The authors carefully distinguish practical horizon-freeness of the base phase from fully anytime schedules, and acknowledge that the theory does not cover AdamW/Muon dynamics.","tokens_in":25520,"tokens_out":977,"duration_ms":23691,"significance":"Post-hoc horizon extension is a genuine practical problem in modern LLM pipelines, and the paper gives a clean, first-principles schedule that is both theoretically justified and simple to implement. The main theoretical contribution is solid: combining a horizon-independent inverse-square-root base with a linear cooldown recovers minimax-optimal last-iterate rates under standard mirror-descent assumptions, with a self-contained two-part proof (suffix checkpoint + geometric block propagation). Empirically, the most useful finding is that WSqD’s preferred base scale is stable across horizons in their setting (Figure 4), whereas WSD’s optimum drifts—directly supporting the reuse protocol. Strengths include an honest continuation protocol, seed/corpus/size ablations in Appendix C, and an explicit comparison to the closest empirical competitor (power schedule). The work is a credible contribution to schedule design even if the theory–practice transfer remains stylized.","major_comments":[{"comment":"The load-bearing practical claim is that a single base scale selected on a short pilot can be reused without retuning when the horizon is extended (§1.2, §3.2–3.3, Figure 4). All main evidence is a 213M model, ≤60k steps, AdamW, and two corpora. Section 5 correctly flags this limit, but the title/abstract frame the result as a schedule “for large model training.” For the journal version, either add at least one larger-scale continuation check, or reframe abstract/intro claims so that the proven convex result and the small-scale transfer evidence are not oversold as established large-model practice.","section":null},{"comment":"Theorem 1 (Assumptions 1–3, Appendix A) is for nonsmooth convex stochastic mirror descent without momentum or adaptive state; all validation-loss experiments use AdamW (§3.1). Appendix B is only an informal geometric diagnostic of omitted dual-norm scales. This gap is acknowledged, but it is still the main correctness-risk for transferring the horizon-independence of c0 to practice. Please state more sharply in §2 and the abstract what is proven versus what is hypothesized, and avoid language that presents the AdamW results as confirmation of the last-iterate theorem.","section":null},{"comment":"§3.5 / Figure 6b: under the same short-horizon tuning protocol, the power schedule of Shen et al. slightly outperforms WSqD at the reported horizons (gap shrinks by 60k). The abstract’s “matches or outperforms … other baselines” is therefore only marginally accurate for this closest competitor. Please qualify the claim (e.g., matches power schedule while using a fixed −1/2 exponent and a simpler shift regularizer) and discuss when a practitioner should prefer WSqD over the power schedule rather than treating them as interchangeable.","section":null},{"comment":"Figure 4 / §3.3: the claim that WSqD’s optimum is fixed at η⋆_max=0.0015 across T∈{15k,…,60k} rests on a coarse grid {0.0005,0.001,0.0015,0.002,0.0025,0.003}. Mild horizon drift could be hidden between grid points. A denser sweep around 0.001–0.002 (or reporting a fitted optimum) would make the “remarkably stable” claim more convincing and better support the no-retuning protocol.","section":null}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful bit is simple: replace WSD’s flat stable phase with a shifted inverse-square-root base, keep the linear cooldown, and you get a schedule whose base scale does not need to know the final horizon. Theorem 1 shows that under standard nonsmooth convex mirror-descent assumptions this recovers the minimax O(1/√T) last-iterate rate; the horizon only appears when you decide to start the cooldown. That is the right formal property for post-hoc continuation, and the proof (suffix checkpoint + geometric block propagation) is standard and carefully written.\n\nEmpirically they do the honest test: pick one base rate on a short pilot, freeze it, and continue to 15k–60k steps on a 213M LLaMA-style model on SlimPajama. The optimal base for WSqD stays put while WSD’s drifts (Figure 4); they also beat or match carefully tuned WSD, two-stage retuned WSD, and the power schedule of Shen et al. Seed, OpenWebText2, and 124M ablations in the appendix are consistent. They are explicit that the power schedule is close and that their theory is stylized; Appendix B’s Adam/Muon geometry diagnostics are a nice extra rather than overclaim.\n\nSoft spots are real but proportional. The load-bearing transfer from convex mirror descent to AdamW transformers is untested at scale—exactly what the authors flag in §5. Horizons stop at 60k and the model is small; if the optimal base starts drifting the way WSD’s peak does once you move to larger models or longer budgets, the “reuse one rate” selling point shrinks. No code release. None of that breaks the convex result or the small-scale evidence; it just means the practical claim is still provisional.\n\nThis is for people who care about mid-training / continued pretraining schedules and for optimizers who want a clean last-iterate analysis that actually supports continuation. I would send it to referees. Worth reading and citing for the schedule + the controlled transfer protocol; not yet settled practice at scale.","headline":"Clean anytime-ish schedule with a correct last-iterate convex proof and honest small-scale continuation experiments; the theory-to-AdamW transfer is the real soft spot, not a hidden flaw in the math.","tokens_in":26122,"tokens_out":526,"would_cite":true,"duration_ms":5738,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"WSqD replaces WSD's flat phase with a shifted inverse-square-root base so one peak learning rate can be reused when training is extended, with a final linear cooldown that recovers optimal last-iterate rates in convex stochastic analysis.","keywords":["learning rate schedule","horizon-free training","continued pretraining","WSqD","warmup-stable-decay","stochastic convex optimization","last-iterate convergence","large language models"],"falsifier":"At a larger model scale or different corpus, re-tune the peak rate separately at each continuation horizon for both WSD and WSqD; if WSqD's transferred single rate falls clearly behind the re-tuned WSD (or the power schedule) at long horizons, the claimed horizon-free transfer fails.","tokens_in":26051,"feed_emoji":"📈","tokens_out":688,"duration_ms":6027,"temperature":0.7,"pith_summary":"Standard learning-rate schedules either lock to a fixed training length or keep a constant peak rate that drifts out of tune when that length is extended. This paper proposes WSqD: a short warmup, a shifted inverse-square-root base that never needs the final horizon, and a short linear cooldown that is started only when a deployable endpoint is chosen. In the classical convex stochastic setting the schedule recovers the minimax-optimal last-iterate rate of order 1 over square-root of T; the scale of the base is independent of T. On SlimPajama pretraining of a 213M LLaMA-style model, a single base rate chosen on a short pilot run and reused without retuning matches or beats carefully tuned WSD and related baselines out to 60k iterations. The practical payoff is that mid-training and post-hoc horizon extension become schedule-reuse problems rather than retuning problems.","feed_headline":"One learning rate works when training is extended","feed_subtitle":"WSqD keeps a horizon-free inverse-square-root base and only starts linear cooldown at the end.","key_machinery":"WSqD schedule: η_t = c0 / √(t + T0) until a fixed fraction of the committed horizon, then linear decay to zero. The inverse-square-root base is horizon-free; the linear cooldown is what removes the classical log-T last-iterate overhead and recovers the minimax rate.","core_discovery":"A shifted inverse-square-root base plus a final linear cooldown (WSqD) is horizon-independent in its base parameters, attains the optimal O(1/√T) last-iterate rate for stochastic mirror descent under standard convex assumptions, and, on language-model pretraining, lets one peak learning rate selected on a short pilot be reused across substantially longer continuations while matching or outperforming WSD and related schedules.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["WSqD reuses one peak LR across any training horizon","Horizon-free square-root base needs cooldown timing only","One peak rate works for short pilots and long runs","WSqD keeps inverse-square-root base independent of T","Shifted square-root schedule matches WSD without retuning"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That the convex, nonsmooth mirror-descent theory is a reliable guide for the non-convex AdamW training of transformers, so that a base rate tuned on a short pilot remains near-optimal when the run is extended.","fun_headline_variants_meta":{"raw":{"variants":["WSqD reuses one peak LR across any training horizon","Horizon-free square-root base needs cooldown timing only","One peak rate works for short pilots and long runs","WSqD keeps inverse-square-root base independent of T","Shifted square-root schedule matches WSD without retuning"]},"model":"grok-4.5","effort":"low","cost_usd":0.00583,"raw_usage":{"total_tokens":1493,"prompt_tokens":782,"num_sources_used":0,"completion_tokens":86,"cost_in_usd_ticks":58300000,"prompt_tokens_details":{"text_tokens":782,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":625,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":782,"tokens_out":86,"duration_ms":7780,"temperature":1.0,"reasoning_tokens":625,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T08:03:01.961244+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"At a larger model scale or different corpus, re-tune the peak rate separately at each continuation horizon for both WSD and WSqD; if WSqD's transferred single rate falls clearly behind the re-tuned WSD (or the power schedule) at long horizons, the claimed horizon-free transfer fails.","supporting_citations":[],"review_version":1}