{"id":"0e4f4c6a-f4fc-4904-a646-32576b19ef4e","arxiv_id":"2501.12243","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"FOCUS, a Signum-based optimizer with an attraction force toward the EMA of parameters, is claimed to yield faster and more stable GPT-2 pretraining than Adam.","lead":"This paper introduces FOCUS, an optimizer that adds a pull toward moving averaged parameters to the Signum update, and reports it trains GPT-2 faster and more stably than Adam in their setup. The authors also propose that gradient noise, not just landscape sharpness, is a key limiting factor in large language model pretraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'faster than Adam' speedup relies on a cross-paper bfloat16 baseline; without a same-environment float16 Adam run, the central claim is unestablished.","rationale":"The paper has real strengths: the toy model and MNIST experiments give a coherent, partially independent motivation for the FOCUS update, and Algorithm 1 is clearly specified. The convergence proof (Theorem 4.2) is a useful sanity check, though it analyzes a variant with decaying beta1 and gamma that Algorithm 1 does not use. My stress-test focuses on the empirical headline, because the claim 'faster than Adam' is what would make FOCUS practically significant and what would support the inference that gradient noise limits LLM training. That claim depends entirely on the comparability of the Adam baseline. The paper itself states that Adam is unstable in float16 at the copied learning rate and only becomes stable at a much smaller learning rate; rather than run Adam at its best stable float16 configuration, the authors compare against a bfloat16 curve from another paper. This is a precision/hardware confound. A single controlled Adam run in the same environment—with a small lr sweep to identify a stable, reasonably fast configuration—would settle whether the 2x speedup is real or an artifact of the baseline. Because the paper does not provide that run, the honest verdict is conditional: the method is well motivated and the stability claims are plausible, but the headline speedup over Adam is unverified. The reader's weakest_assumption identifies the same issue; I agree.","tokens_in":17978,"tokens_out":5646,"duration_ms":49143,"concrete_test":"Run AdamW in the FOCUS environment (float16, 8 V100s, OpenWebText, batch size 480, gradient clipping 1.0, same codebase) with the Liu et al. (2023a) settings (lr=6e-4, beta1=0.9, beta2=0.95, wd=0.1) and with a small lr sweep (e.g., 1e-4, 2e-4, 3e-4, 4e-4, 5e-4) at fixed beta1/beta2/wd, running to 1e5 steps with the same cosine schedule. Compare loss-vs-step and loss-vs-wall-time against the FOCUS curves in Fig. 3d (including the 2e4 and 5e4 step runs). If any stable Adam configuration reaches the same loss as FOCUS at comparable step counts, the 2x speedup claim fails; if FOCUS still reaches the target loss in roughly half the steps across configurations, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—FOCUS is 'faster than Adam' in GPT-2 pre-training (Abstract, Fig. 3d)—is established only against an Adam curve copied from Liu et al. (2023a). That baseline was trained in bfloat16 on different hardware with its own hyperparameter tuning, while all FOCUS runs are float16 on 8 V100s with only a handful of hyperparameter choices tested (Sec. 3.1, 3.3). In the paper's own float16 environment, Adam at the copied settings (lr=6e-4) is unstable (Fig. 3b); the only stable float16 Adam shown (lr=1e-4) is much slower than FOCUS. Consequently, the reported 2x speedup may simply reflect that the Adam baseline required bfloat16 to be stable, not that FOCUS is intrinsically faster. No Adam run is performed in the FOCUS environment at a tuned, stable learning rate, and no seeds or error bars are reported for the GPT-2 runs, so single-run variance is uncontrolled. Until a same-environment Adam baseline is produced, 'faster than Adam' is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper hypothesizes that LLM pre-training loss landscapes behave like narrowing valleys with large gradient noise, argues from a toy landscape that Adam becomes too slow when noise is large relative to valley sharpness while Signum keeps a fixed step size, and proposes FOCUS, a Signum-like optimizer augmented by an attraction term toward the exponential moving average of parameters. The authors test FOCUS on GPT-2 small (125M) in float16, reporting that it is more stable than Signum and Adam and approximately twice as fast as a published Adam baseline, and they prove an O(sqrt(T)) regret bound in an online convex setting.","tokens_in":18295,"tokens_out":6565,"duration_ms":62545,"significance":"The proposed mechanism is interesting and the toy-to-real methodology is commendable: the MNIST batch-size transition in Figure 2c is a clean, falsifiable prediction that is independent of the toy model's construction, and the explicit regret analysis in Section 4 is a useful sanity check. If the GPT-2 speedup were established with a controlled baseline, FOCUS would be a practical contribution to LLM pre-training and would provide evidence that gradient noise is an important limiting factor. However, the central empirical claim currently rests on an uncontrolled cross-paper comparison, so the significance cannot be fully assessed yet.","major_comments":[{"comment":"The central claim that FOCUS is 'faster than Adam' is not established by the reported comparison. The Adam curve used as baseline is copied from Liu et al. (2023a), where training was done in bfloat16 on different hardware with its own tuned hyperparameters; the FOCUS runs are float16 on 8 V100 GPUs with a few manually chosen hyperparameters and no reported seeds or error bars. In the authors' own float16 environment, Adam at lr=6e-4 is unstable (Section 3.2), and the only stable Adam run shown (lr=1e-4) is much slower than FOCUS, so the speedup may reflect precision or hardware differences rather than the optimizer's intrinsic performance. Furthermore, Figure 3d compares FOCUS at 5e4 and 2e4 steps with shortened cosine schedules against Adam at 1e5 and 5e4 steps, so the '2x speedup' conflates optimizer choice with learning-rate schedule. A same-environment float16 Adam run with a tuned stable learning rate and multiple seeds is required before the speedup claim can be evaluated.","section":"Section 3.3, Figures 3c-d"},{"comment":"The toy-model phase diagram partially encodes its own conclusion. The multiplicative gradient noise model and the valley sharpness a enter Adam's normalized update through the signal-to-noise ratio m_t/sqrt(v_t), so by construction Adam's effective step shrinks as sigma or a grows; the toy experiment is therefore an illustration of the proposed mechanism rather than independent evidence for it. The MNIST batch-size transition (Figure 2c) is a genuinely independent and falsifiable test and does support the qualitative claim. The authors should explicitly present the toy result as a consequence of the model and place the evidentiary weight on the MNIST transition and the GPT-2 experiments.","section":"Section 2.2, Eq. (1), Figure 2a"},{"comment":"The theorem is stated for FOCUS with time-decaying beta_1,t and gamma_t, whereas Algorithm 1 uses constant beta_1 and gamma and includes a weight-decay term. The remark after the theorem that the extra summation terms 'may be small themselves' is not a proof that the bound carries over to Algorithm 1. If the regret bound is intended to cover the proposed algorithm, either the statement must be amended or the theorem should be explicitly labeled as an idealized variant.","section":"Section 4 and Appendix C, Theorem 4.2"}],"minor_comments":[{"comment":"The update for the parameter average uses the same symbol theta_t on both sides of the equation ('theta_t = beta_2 theta_{t-1} + (1 - beta_2) theta_t'), which is mathematically self-referential and confusing; introducing a separate symbol such as psi_t or theta_ema would clarify the algorithm.","section":"Algorithm 1"},{"comment":"There is a typo in the line 'Compute minibach loss', which should read 'minibatch'.","section":"Algorithm 1"},{"comment":"The phrase 'approximately a twice speedup' is vague; 'approximately a 2x speedup' or 'roughly half the training steps' would be clearer, especially because the comparison involves different learning-rate schedules.","section":"Abstract and Section 3.3"},{"comment":"In the Morales Brotons et al. reference, the author name is printed as 'V ogels' and should be 'Vogels'; the reference to Pagliardini et al. should be consistently rendered as 'AdEMAMix'.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The cross-paper baseline is the main concern: the central speedup claim is conditional on a controlled same-precision and same-environment Adam comparison. The MNIST batch-size experiment is a strong, independent piece of evidence and should be highlighted. If the authors can provide a proper Adam baseline and seed variation, the paper would be worthy of reconsideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a paper worth reading, but the headline speedup claim is not yet supported. What's actually new: the FOCUS update rule—Signum's sign-momentum step plus a sign-projected attraction toward an EMA of the parameters—doesn't appear in the cited literature. The toy-model phase diagram in the noise-vs-sharpness plane is a useful way to organize when Adam or Signum wins, and the MNIST batch-size transition is a genuinely independent, falsifiable check that backs the qualitative claim that large gradient noise favors sign-based steps.\n\nThe GPT-2 experiments hold the soft spots, and they are load-bearing. The 'faster than Adam' claim is established only against an Adam curve copied from Liu et al. (2023a), trained in bfloat16, on different hardware, with its own hyperparameter tuning. All FOCUS runs are float16 on 8 V100s, and the paper's own float16 Adam at the copied settings is unstable. The one stable float16 Adam shown (lr=1e-4) is slower than FOCUS, but it is not a tuned baseline for that environment. Without a same-environment, tuned Adam run, the 2x speedup is unestablished. There are also no seeds or error bars for the GPT-2 runs, and the fastest FOCUS configuration appears to have been selected post hoc.\n\nThe regret bound is a smaller problem, but real: Theorem 4.2 assumes time-decaying beta1_t and gamma_t, while Algorithm 1 uses constant beta1 and gamma. The weight decay term in Algorithm 1 is also missing from the proof. The bound is a sanity check, not a guarantee for the actual algorithm.\n\nWhat holds up: the toy model and MNIST studies are coherent and honestly presented, and the stability comparison against Signum in float16 is informative. The paper also flags the baseline dependence in its own discussion, which is more than some papers do.\n\nWho it's for: people designing or evaluating optimizers for LLM pre-training, and anyone interested in how gradient noise shapes optimizer choice. A serious referee should engage with it; before publication the authors should run Adam in their own environment with tuned hyperparameters and multiple seeds, and align the proof with Algorithm 1. I'd recommend revise, not reject.","headline":"FOCUS is a genuinely new optimizer idea with a coherent toy-model story, but the central '2x faster than Adam' claim rests on a cross-paper, cross-precision baseline and needs a same-environment rerun before it convinces.","tokens_in":18834,"tokens_out":4196,"would_cite":true,"duration_ms":37424,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68W40","90C25"],"pacs":[],"model":"deepseek-v4-flash","headline":"FOCUS, an optimizer that adds a self-attraction term to sign-based updates, trains a 125M-parameter GPT-2 model roughly twice as fast as an Adam baseline while remaining stable in float16.","keywords":["FOCUS optimizer","gradient noise","narrowing valley landscape","LLM pre-training","sign-based updates","Adam vs Signum","parameter moving average","optimizer stability"],"falsifier":"Run AdamW on the same eight V100 GPUs in float16, with the same batch size and codebase, scanning learning rates around $6\\times10^{-4}$ including the doubled-weight-decay variant, and compare loss curves against FOCUS at equal step counts. If Adam trained in FOCUS's environment matches or beats FOCUS, the twofold speedup collapses. A second check is to measure the gradient signal-to-noise ratio during GPT-2 pre-training: a consistently small ratio would support the large-noise premise, while a large ratio would undercut the explanation.","tokens_in":1907,"feed_emoji":"⚡","tokens_out":3256,"duration_ms":100371,"temperature":0.7,"pith_summary":"The paper argues that the loss landscape of LLM pre-training is a narrowing valley and, more importantly, that gradient noise, not just valley sharpness, is a key limiting factor. Using a synthetic toy landscape, the authors show that when noise is large relative to sharpness, Adam shrinks its effective step size too much and falls behind Signum, which keeps a fixed step size. They then build FOCUS, which adds an attraction force toward the moving average of the parameters to Signum's fixed-size steps. In GPT-2 (125M) pre-training, FOCUS is reported to be more stable than Signum and about twice as fast as a published Adam baseline. If correct, the work reframes optimizer design for LLMs: robustness to gradient noise, rather than per-coordinate normalization, may deserve the focus.","feed_headline":"FOCUS optimizer trains GPT-2 twice as fast as Adam","feed_subtitle":"By keeping step sizes large when gradients are noisy, FOCUS suggests noise—not sharpness—limits LLM training.","key_machinery":"The carrying mechanism is the FOCUS update rule $\\theta_{t+1} = \\theta_t - \\eta_t(\\text{sign}(m_t) + \\gamma\\,\\text{sign}(\\theta_t - \\hat{\\theta}_t))$, where $m_t$ is the exponential moving average (EMA) of gradients and $\\hat{\\theta}_t$ is the bias-corrected EMA of the parameters themselves. The second term is the attraction force: it points from the current parameters toward their own moving average, acting like a self-attraction of bouncing particles so the trajectory stays compact at the bottom of a narrowing valley. The argument is carried by the toy landscape $L(x,y) = \\frac{a}{2} u^2 v^2 - cu$ with multiplicative gradient noise of standard deviation $\\sigma$; the ratio of noise $\\sigma$ to sharpness $a$ decides which optimizer wins, and the phase diagram is the paper's central evidence. The update's sign-based core keeps the step size fixed at $\\eta_t$, avoiding Adam's noise-driven step shrinkage.","core_discovery":"On the paper's own terms, the central discovery is a crossover in optimizer performance: on a valley-shaped loss with multiplicative gradient noise, Adam beats Signum only when noise is small relative to the valley's sharpness, and past a critical noise level Signum wins because Adam's effective step size is $\\eta_t \\hat{m}_t / \\sqrt{\\hat{v}_t}$, which shrinks when noise makes the ratio small even though the curvature does not require it. FOCUS is the proposed remedy: it keeps Signum's fixed-size sign steps and adds the term $-\\gamma\\,\\text{sign}(\\theta_t - \\hat{\\theta}_t)$, which pulls the parameters toward their bias-corrected exponential moving average, letting the trajectory squeeze into a sharp valley without cutting the step size. In GPT-2 (125M) training in float16, the authors report that FOCUS running $5\\times10^4$ steps reaches a loss close to Adam running $10^5$ steps, an approximately twofold speedup. The same update rule carries a worst-case regret bound of $O(\\sqrt{T})$ in online convex optimization, matching Adam's order.","pith_inferences":["The phase-diagram logic extends beyond Adam: other adaptive optimizers that shrink their effective step size in noisy directions, such as Lion and AdEMAMix, should show the same crossover and fall behind fixed-step-size methods when noise dominates sharpness; this is a testable prediction the paper does not run.","The attraction term is closely related to weight averaging, and a natural extension is to schedule or anneal the strength $\\gamma$, or to let $\\hat{\\theta}_t$ track selected past checkpoints; the paper only tests constant $\\gamma$.","The decisive check the paper leaves implicit is an apples-to-apples run: AdamW in the same precision, hardware, and hyperparameter scan as FOCUS would reveal whether the twofold speedup survives a fair comparison, which would markedly strengthen the noise-limitation claim.","If the narrowing-valley picture is right, the loss floor of pre-training may be set by training dynamics (finite step size and noise) rather than model capacity alone, which would connect optimizer design to scaling-law exponents."],"forward_implications":["FOCUS should be tried where gradient noise is largest: small batch sizes, low-precision (float16) training, and the early phase of pre-training before noise averages out.","If gradient noise is a genuine limiting factor, then noise-reducing interventions such as larger batches or variance-reduced gradient estimates should yield speedups comparable to switching optimizers.","The noise-versus-sharpness crossover predicts that the best optimizer depends on batch size: small batches favor fixed-step-size sign updates, large batches favor Adam, a transition the paper demonstrates on MNIST classification.","FOCUS carries a worst-case regret bound of $O(\\sqrt{T})$ in online convex optimization, the same order as Adam, so the practical speedup is not bought by sacrificing the worst-case convergence guarantee."],"supporting_citations":[{"why":"Supplies the Adam update rule, EMA formulas, and bias correction that FOCUS modifies and compares against.","marker":"Kingma & Ba, 2014"},{"why":"Defines sign-based (signSGD) updates; Signum is the base algorithm FOCUS extends with the attraction term.","marker":"Bernstein et al., 2018"},{"why":"Supplies the GPT-2 training codebase, the optimal Adam hyperparameters, and the bfloat16 Adam loss curve used as the speedup baseline.","marker":"Liu et al., 2023a"},{"why":"Motivates the question by reporting that Signum can be comparable to or better than Adam in LLM pre-training.","marker":"Zhao et al., 2024"},{"why":"Supports the narrowing-valley picture and the claim that noise and large learning rates keep optimizers from reaching the valley bottom.","marker":"Wen et al., 2024"},{"why":"Provides the online convex optimization protocol and regret definition used in the convergence analysis.","marker":"Zinkevich, 2003"},{"why":"AdamW and decoupled weight decay, which FOCUS adopts for its weight-decay term.","marker":"Loshchilov & Hutter, 2019"},{"why":"Defines the GPT-2 architecture and tokenizer used in the pre-training experiments.","marker":"Radford et al., 2019"}],"fun_headline_variants":["FOCUS: optimizer that beats Adam 2x on GPT-2","Noise not sharpness: FOCUS optimizer doubles GPT-2 speed","FOCUS optimizer: 2x faster than Adam on noisy LLM training","Signum plus memory: FOCUS speeds GPT-2 2x over Adam","FOCUS: keep step size, attract to EMA, train 2x faster"],"cache_read_input_tokens":20864,"weakest_assumption_plain":"The claimed twofold speedup assumes the Adam baseline copied from a prior paper, trained in bfloat16 on different hardware with its own tuned hyperparameters, is a fair and near-optimal comparison for Adam in FOCUS's own float16 setup.","fun_headline_variants_meta":{"raw":{"variants":["FOCUS: optimizer that beats Adam 2x on GPT-2","Noise not sharpness: FOCUS optimizer doubles GPT-2 speed","FOCUS optimizer: 2x faster than Adam on noisy LLM training","Signum plus memory: FOCUS speeds GPT-2 2x over Adam","FOCUS: keep step size, attract to EMA, train 2x faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1412,"prompt_tokens":938,"completion_tokens":474,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":368}},"tokens_in":554,"tokens_out":474,"duration_ms":4496,"temperature":1.0,"reasoning_tokens":368,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:21:30.691995+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AdamW on the same eight V100 GPUs in float16, with the same batch size and codebase, scanning learning rates around $6\\times10^{-4}$ including the doubled-weight-decay variant, and compare loss curves against FOCUS at equal step counts. If Adam trained in FOCUS's environment matches or beats FOCUS, the twofold speedup collapses. A second check is to measure the gradient signal-to-noise ratio during GPT-2 pre-training: a consistently small ratio would support the large-noise premise, while a large ratio would undercut the explanation.","supporting_citations":[],"review_version":1}