{"id":"5e0e88c9-f2ae-4b5c-b339-7b25d91fe005","arxiv_id":"2605.19095","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ScheduleFree+ scales schedule-free learning to LLMs with fixes for large batches and models, outperforming Warmup-Stable-Decay schedules by up to 31% at 1000 tokens per parameter.","lead":"The paper introduces ScheduleFree+, a set of fixes to make schedule-free learning work for training large language models without any learning rate or schedule. This could simplify LLM pretraining and improve results especially on very long training runs.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Whether the identified scaling fixes remain free of implicit schedule-like or stability adjustments at LLM scales is the least-secured step in the central claim.","rationale":"The reader's weakest_assumption matches the precise point where the scaling argument is least anchored; the full text would need to contain explicit ablations or proofs that the fixes are progress-independent for the central claim to be secure. No stronger internal inconsistency is visible from the abstract-level description of the contribution.","tokens_in":1618,"tokens_out":351,"duration_ms":44101,"concrete_test":"Re-run the 1000 tokens/parameter pretraining experiment on the largest model size reported in the paper, once with the published ScheduleFree+ implementation and once with an ablated version that removes each listed fix in turn; if any single fix removal causes >10% relative degradation or requires re-introduction of a time-varying term to recover performance, the claim that the method is schedule-free after scaling weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract states that a number of fixes were identified to scale Schedule-Free Learning to larger batch sizes and model sizes, after which ScheduleFree+ is claimed to be learning-rate-free, schedule-free, and to outperform WSD by 31% at 1000 tokens per parameter. For this to hold, those fixes must not themselves introduce time-dependent behavior, per-step normalization changes, or batch-size-specific rules that function as hidden schedules. The paper's argument is therefore only as strong as the demonstration that the fixes are purely algorithmic (e.g., constant adjustments independent of training progress) and that no new instabilities appear in the long-duration regime where the 31% gain is reported.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces ScheduleFree+, a scaled version of Schedule-Free Learning for large language models. It identifies fixes to enable training at larger batch and model sizes, claims the resulting method is learning-rate-free and schedule-free, reports that it outperforms Warmup-Stable-Decay (WSD) schedules (with a 31% gain at 1000 tokens per parameter), shows particular effectiveness for long-duration training, and supplies a theoretical foundation for model averaging and checkpoint merging during pretraining.","tokens_in":1754,"tokens_out":617,"duration_ms":26958,"significance":"If the empirical gains are robust and the scaling fixes introduce no implicit time-dependent or schedule-like behavior, the work would be significant for simplifying LLM pretraining by removing the need for learning-rate schedules and hyperparameter tuning. The theoretical link to model averaging provides a useful conceptual contribution even if the performance claims require further verification.","major_comments":[{"comment":"Abstract and §3: The claim of a 31% outperformance over SOTA schedules at 1000 tokens per parameter is presented without accompanying details on the exact baselines used, number of independent runs, variance across seeds, or statistical significance tests. This omission makes it impossible to evaluate whether the reported gain is load-bearing or sensitive to post-hoc selection of fixes.","section":"Abstract, §3"},{"comment":"§4.1–4.2: The fixes identified to scale Schedule-Free Learning to larger batch sizes and model sizes are described only at a high level. It is not shown whether these adjustments are strictly constant (independent of training step or progress) or whether they incorporate per-step normalization, batch-size-specific rules, or other mechanisms that could function as hidden schedules, which directly undermines the central schedule-free claim.","section":"§4.1–4.2"},{"comment":"§5: The long-duration regime where the 31% gain is reported is not accompanied by diagnostics for new instabilities (e.g., loss spikes, divergence, or degradation of the anytime property) that might appear only after the scaling fixes are applied at LLM scales.","section":"§5"}],"minor_comments":[{"comment":"The notation distinguishing the original Schedule-Free optimizer from the new ScheduleFree+ variant could be made more explicit in the methods section to avoid reader confusion.","section":"§2"},{"comment":"Figure captions and axis labels in the scaling experiments should explicitly state the model sizes and batch sizes used so that the claimed improvements can be directly compared to prior Schedule-Free results.","section":"Figures 2–4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's fit to a machine-learning venue is appropriate, but the absence of reproducibility artifacts (code, exact hyperparameter tables, or seed lists) in the current version would be a concern for reviewers focused on empirical claims."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on our manuscript. We have carefully considered each major comment and revised the paper to address the concerns about experimental details, clarification of the scaling fixes, and stability diagnostics. Our point-by-point responses follow.","responses":[{"response":"We agree that more experimental details are needed to substantiate the claim. In the revised manuscript we have expanded the relevant section to specify the exact WSD baseline configurations (including their warmup, stable, and decay phases and associated hyperparameters), the use of five independent random seeds per setting, the observed standard deviations, and the results of paired t-tests confirming statistical significance of the reported gains.","revision_made":"yes","referee_comment":"[Abstract, §3] Abstract and §3: The claim of a 31% outperformance over SOTA schedules at 1000 tokens per parameter is presented without accompanying details on the exact baselines used, number of independent runs, variance across seeds, or statistical significance tests. This omission makes it impossible to evaluate whether the reported gain is load-bearing or sensitive to post-hoc selection of fixes."},{"response":"The fixes are fixed, constant scalars (a batch-size multiplier applied to the base step-size and a model-size-dependent averaging coefficient) that are chosen once before training begins and held fixed for the entire run; they contain no per-step normalization or progress-dependent terms. The revised sections now list the exact constant values used, include pseudocode that makes the time-independence explicit, and report an ablation confirming that performance is unchanged when any potential step-dependent component is removed.","revision_made":"yes","referee_comment":"[§4.1–4.2] §4.1–4.2: The fixes identified to scale Schedule-Free Learning to larger batch sizes and model sizes are described only at a high level. It is not shown whether these adjustments are strictly constant (independent of training step or progress) or whether they incorporate per-step normalization, batch-size-specific rules, or other mechanisms that could function as hidden schedules, which directly undermines the central schedule-free claim."},{"response":"We have added loss-curve diagnostics and stability metrics to §5. The new figures show the full training trajectories at the reported scale, with no loss spikes or divergence observed after the fixes are applied. Separate panels confirm that the anytime property continues to hold, with validation performance improving steadily throughout the long-duration regime.","revision_made":"yes","referee_comment":"[§5] §5: The long-duration regime where the 31% gain is reported is not accompanied by diagnostics for new instabilities (e.g., loss spikes, divergence, or degradation of the anytime property) that might appear only after the scaling fixes are applied at LLM scales."}],"tokens_in":1364,"tokens_out":598,"duration_ms":68456,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing here is that a few targeted adjustments let Schedule-Free learning scale to LLM sizes and deliver a 31% edge over Warmup-Stable-Decay at 1000 tokens per parameter. That number stands out because it comes from the long-duration regime where most methods flatten out.","headline":"ScheduleFree+ shows practical fixes that let schedule-free training beat WSD by 31% at long LLM runs, but the fixes need close checking to confirm they stay truly schedule-free.","tokens_in":2204,"tokens_out":141,"would_cite":false,"duration_ms":27298,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[{"relation":"unclear","rs_module":"IndisputableMonolith/Cost/FunctionalEquation.lean","rs_theorem":"washburn_uniqueness_aczel","paper_passage":"We identify a number of fixes necessary to scale up Schedule-Free Learning to larger batch sizes and model sizes, and present a learning-rate-free and schedule-free method (ScheduleFree+) ... at 1000 tokens per parameter, it outperforms SOTA schedules by 31%."},{"relation":"echoes","rs_module":"IndisputableMonolith/Foundation/AlphaCoordinateFixation.lean","rs_theorem":"J_uniquely_calibrated_via_higher_derivative","paper_passage":"The Polyak step size ... γt = f(yt)−f∗ / ||∇f(yt)||2 ... inverse-gradient norm weighting"},{"relation":"unclear","rs_module":"IndisputableMonolith/Foundation/ArithmeticFromLogic.lean","rs_theorem":"embed_add","paper_passage":"Schedule-Free Learning provides a theoretical foundation for the use of model averaging and checkpoint merging during pretraining."}],"headline":"ScheduleFree+ averaging/Polyak machinery is orthogonal to RS J-cost forcing and distinction-derived periodicity","alignment":"orthogonal","rationale":"The paper's core contributions (inner-momentum reintroduction, inverse-L1 gradient weighting, beta annealing, Polyak step-size adaptation, C-warmup, r=1 weighting) are empirical fixes for scaling iterate averaging in LLM training. These operate in the domain of stochastic non-convex optimization and do not invoke, parallel, or derive from the RS recognition-cost function J(x), golden-ratio ladder, 8-tick clock, or the single-distinction forcing chain. No parameter-free constant derivations or cosh-cost identities appear.","tokens_in":58628,"confidence":"high","tokens_out":421,"duration_ms":17062,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"ScheduleFree+ scales learning-rate-free and schedule-free optimization to large language models while outperforming Warmup-Stable-Decay schedules.","keywords":["schedule-free learning","large language models","learning-rate-free optimization","pretraining","model averaging","warmup-stable-decay","optimizer scaling"],"falsifier":"A head-to-head run on a large language model at production scale where ScheduleFree+ either fails to beat Warmup-Stable-Decay or exhibits new instabilities would disprove the scaling claim.","tokens_in":2505,"feed_emoji":"📈","tokens_out":585,"duration_ms":29786,"temperature":0.7,"pith_summary":"The paper shows how to adapt Schedule-Free Learning for training large language models without any learning-rate tuning or explicit training schedules. It identifies specific fixes that overcome previous scaling barriers at large batch and model sizes. The resulting ScheduleFree+ method delivers stronger results than standard Warmup-Stable-Decay approaches, with the largest gains appearing in long training runs that reach 1000 tokens per parameter. It also supplies a theoretical basis for why model averaging and checkpoint merging are effective during pretraining.","feed_headline":"ScheduleFree+ beats WSD schedules on long LLM pretraining","feed_subtitle":"Targeted fixes let a learning-rate-free method reach 31% better results at 1000 tokens per parameter.","key_machinery":"ScheduleFree+ optimizer, which applies targeted fixes to the core schedule-free update rule to stabilize training at large batch sizes and model scales.","core_discovery":"With the right fixes, Schedule-Free Learning extends to large language model pretraining as a fully learning-rate-free and schedule-free method that surpasses Warmup-Stable-Decay performance, especially on extended training horizons, while grounding the practical use of model averaging in theory.","pith_inferences":["Removing learning-rate and schedule choices could reduce the hyperparameter search burden in LLM development.","The approach may extend to other large-scale optimization settings that currently rely on hand-tuned schedules.","It invites re-examination of whether traditional schedule-based training remains necessary once scaling fixes are in place."],"forward_implications":["ScheduleFree+ yields up to 31% better results than WSD schedules when training reaches 1000 tokens per parameter.","The method is most effective for long-duration training rather than short runs.","Model averaging and checkpoint merging during pretraining receive direct theoretical justification.","Training no longer requires separate learning-rate or schedule selection steps."],"fun_headline_variants":["ScheduleFree+ scales learning-rate-free training to large LLMs","ScheduleFree+ supports large batch and model sizes in LLM training","ScheduleFree+ records higher performance than WSD at 1000 tokens per parameter","ScheduleFree+ grounds model averaging theory for LLM pretraining"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The fixes needed to scale Schedule-Free Learning to larger models and batches are sufficient to deliver strong performance without creating new instabilities or relying on hidden schedule-like behavior.","fun_headline_variants_meta":{"raw":{"variants":["ScheduleFree+ scales learning-rate-free training to large LLMs","ScheduleFree+ supports large batch and model sizes in LLM training","ScheduleFree+ records higher performance than WSD at 1000 tokens per parameter","ScheduleFree+ grounds model averaging theory for LLM pretraining"]},"model":"grok-4.3","cost_usd":0.015214,"raw_usage":{"total_tokens":6389,"prompt_tokens":553,"num_sources_used":0,"completion_tokens":72,"cost_in_usd_ticks":152140500,"prompt_tokens_details":{"text_tokens":553,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":5764,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":553,"tokens_out":72,"duration_ms":53241,"temperature":1.0,"reasoning_tokens":5764,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-20T12:17:53.575111+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A head-to-head run on a large language model at production scale where ScheduleFree+ either fails to beat Warmup-Stable-Decay or exhibits new instabilities would disprove the scaling claim.","supporting_citations":[],"review_version":1}