REVIEW 3 major objections 4 minor 3 cited by
FOCUS: First Order Concentrated Updating Scheme
T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read FOCUS, an optimizer that adds a self-attraction term to sign-based updates, trains a 125M-parameter GPT-2 model roughly twice as fast as an Adam baseline while remaining stable in float16.
desk verdict FOCUS is a genuinely new optimizer idea with a coherent toy-model story, but the central '2x faster than Adam' claim rests on a cross-paper, cross-precision baseline and needs a same-environment rerun before it convinces. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the FOCUS update rule $\theta_{t+1} = \theta_t - \eta_t(\text{sign}(m_t) + \gamma\,\text{sign}(\theta_t - \hat{\theta}_t))$, where $m_t$ is the exponential moving average (EMA) of gradients and $\hat{\theta}_t$ is the bias-corrected EMA of the parameters themselves. The second term is the attraction force: it points from the current parameters toward their own moving average, acting like a self-attraction of bouncing particles so the trajectory stays compact at the bottom of a narrowing valley. The argument is carried by the toy landscape $L(x,y) = \frac{a}{2} u^2 v^2 - cu$ with multiplicative gradient noise of standard deviation $\sigma$; the ratio of noise $\sigma$ to sharpness $a$ decides which optimizer wins, and the phase diagram is the paper's central evidence. The update's sign-based core keeps the step size fixed at $\eta_t$, avoiding Adam's noise-driven step shrinkage.
What would settle it
Run AdamW on the same eight V100 GPUs in float16, with the same batch size and codebase, scanning learning rates around $6\times10^{-4}$ including the doubled-weight-decay variant, and compare loss curves against FOCUS at equal step counts. If Adam trained in FOCUS's environment matches or beats FOCUS, the twofold speedup collapses. A second check is to measure the gradient signal-to-noise ratio during GPT-2 pre-training: a consistently small ratio would support the large-noise premise, while a large ratio would undercut the explanation.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is a crossover in optimizer performance: on a valley-shaped loss with multiplicative gradient noise, Adam beats Signum only when noise is small relative to the valley's sharpness, and past a critical noise level Signum wins because Adam's effective step size is $\eta_t \hat{m}_t / \sqrt{\hat{v}_t}$, which shrinks when noise makes the ratio small even though the curvature does not require it. FOCUS is the proposed remedy: it keeps Signum's fixed-size sign steps and adds the term $-\gamma\,\text{sign}(\theta_t - \hat{\theta}_t)$, which pulls the parameters toward their bias-corrected exponential moving average, letting the trajectory squeeze into a sharp valley without cutting the step size. In GPT-2 (125M) training in float16, the authors report that FOCUS running $5\times10^4$ steps reaches a loss close to Adam running $10^5$ steps, an approximately twofold speedup. The same update rule carries a worst-case regret bound of $O(\sqrt{T})$ in online convex optimization, matching Adam's order.
Load-bearing premise
The claimed twofold speedup assumes the Adam baseline copied from a prior paper, trained in bfloat16 on different hardware with its own tuned hyperparameters, is a fair and near-optimal comparison for Adam in FOCUS's own float16 setup.
Editorial extensions
If this is right
- FOCUS should be tried where gradient noise is largest: small batch sizes, low-precision (float16) training, and the early phase of pre-training before noise averages out.
- If gradient noise is a genuine limiting factor, then noise-reducing interventions such as larger batches or variance-reduced gradient estimates should yield speedups comparable to switching optimizers.
- The noise-versus-sharpness crossover predicts that the best optimizer depends on batch size: small batches favor fixed-step-size sign updates, large batches favor Adam, a transition the paper demonstrates on MNIST classification.
- FOCUS carries a worst-case regret bound of $O(\sqrt{T})$ in online convex optimization, the same order as Adam, so the practical speedup is not bought by sacrificing the worst-case convergence guarantee.
Reading between the lines
- The phase-diagram logic extends beyond Adam: other adaptive optimizers that shrink their effective step size in noisy directions, such as Lion and AdEMAMix, should show the same crossover and fall behind fixed-step-size methods when noise dominates sharpness; this is a testable prediction the paper does not run.
- The attraction term is closely related to weight averaging, and a natural extension is to schedule or anneal the strength $\gamma$, or to let $\hat{\theta}_t$ track selected past checkpoints; the paper only tests constant $\gamma$.
- The decisive check the paper leaves implicit is an apples-to-apples run: AdamW in the same precision, hardware, and hyperparameter scan as FOCUS would reveal whether the twofold speedup survives a fair comparison, which would markedly strengthen the noise-limitation claim.
- If the narrowing-valley picture is right, the loss floor of pre-training may be set by training dynamics (finite step size and noise) rather than model capacity alone, which would connect optimizer design to scaling-law exponents.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper hypothesizes that LLM pre-training loss landscapes behave like narrowing valleys with large gradient noise, argues from a toy landscape that Adam becomes too slow when noise is large relative to valley sharpness while Signum keeps a fixed step size, and proposes FOCUS, a Signum-like optimizer augmented by an attraction term toward the exponential moving average of parameters. The authors test FOCUS on GPT-2 small (125M) in float16, reporting that it is more stable than Signum and Adam and approximately twice as fast as a published Adam baseline, and they prove an O(sqrt(T)) regret bound in an online convex setting.
Significance. The proposed mechanism is interesting and the toy-to-real methodology is commendable: the MNIST batch-size transition in Figure 2c is a clean, falsifiable prediction that is independent of the toy model's construction, and the explicit regret analysis in Section 4 is a useful sanity check. If the GPT-2 speedup were established with a controlled baseline, FOCUS would be a practical contribution to LLM pre-training and would provide evidence that gradient noise is an important limiting factor. However, the central empirical claim currently rests on an uncontrolled cross-paper comparison, so the significance cannot be fully assessed yet.
major comments (3)
- [Section 3.3, Figures 3c-d] The central claim that FOCUS is 'faster than Adam' is not established by the reported comparison. The Adam curve used as baseline is copied from Liu et al. (2023a), where training was done in bfloat16 on different hardware with its own tuned hyperparameters; the FOCUS runs are float16 on 8 V100 GPUs with a few manually chosen hyperparameters and no reported seeds or error bars. In the authors' own float16 environment, Adam at lr=6e-4 is unstable (Section 3.2), and the only stable Adam run shown (lr=1e-4) is much slower than FOCUS, so the speedup may reflect precision or hardware differences rather than the optimizer's intrinsic performance. Furthermore, Figure 3d compares FOCUS at 5e4 and 2e4 steps with shortened cosine schedules against Adam at 1e5 and 5e4 steps, so the '2x speedup' conflates optimizer choice with learning-rate schedule. A same-environment float16 Adam run with a tuned stable learning rate and multiple seeds is required before the speedup claim can be evaluated.
- [Section 2.2, Eq. (1), Figure 2a] The toy-model phase diagram partially encodes its own conclusion. The multiplicative gradient noise model and the valley sharpness a enter Adam's normalized update through the signal-to-noise ratio m_t/sqrt(v_t), so by construction Adam's effective step shrinks as sigma or a grows; the toy experiment is therefore an illustration of the proposed mechanism rather than independent evidence for it. The MNIST batch-size transition (Figure 2c) is a genuinely independent and falsifiable test and does support the qualitative claim. The authors should explicitly present the toy result as a consequence of the model and place the evidentiary weight on the MNIST transition and the GPT-2 experiments.
- [Section 4 and Appendix C, Theorem 4.2] The theorem is stated for FOCUS with time-decaying beta_1,t and gamma_t, whereas Algorithm 1 uses constant beta_1 and gamma and includes a weight-decay term. The remark after the theorem that the extra summation terms 'may be small themselves' is not a proof that the bound carries over to Algorithm 1. If the regret bound is intended to cover the proposed algorithm, either the statement must be amended or the theorem should be explicitly labeled as an idealized variant.
minor comments (4)
- [Algorithm 1] The update for the parameter average uses the same symbol theta_t on both sides of the equation ('theta_t = beta_2 theta_{t-1} + (1 - beta_2) theta_t'), which is mathematically self-referential and confusing; introducing a separate symbol such as psi_t or theta_ema would clarify the algorithm.
- [Algorithm 1] There is a typo in the line 'Compute minibach loss', which should read 'minibatch'.
- [Abstract and Section 3.3] The phrase 'approximately a twice speedup' is vague; 'approximately a 2x speedup' or 'roughly half the training steps' would be clearer, especially because the comparison involves different learning-rate schedules.
- [References] In the Morales Brotons et al. reference, the author name is printed as 'V ogels' and should be 'Vogels'; the reference to Pagliardini et al. should be consistently rendered as 'AdEMAMix'.
Circularity Check
No load-bearing circularity: the toy model is an explicit hypothesis-embodying study and the GPT-2 comparison uses an external baseline; the main weaknesses are experimental-validity risks, not definitional circularity.
full rationale
The paper's derivation chain is not circular in the sense prohibited here. The narrowing-valley picture is introduced as an explicit hypothesis ('we hypothesize that the pre-training loss landscape features a narrowing valley structure'), and the toy landscape is deliberately built to instantiate it ('Our toy loss function ... in the spirit of narrowing valleys'). The Adam-vs-Signum phase diagram on that toy landscape is an empirical study of that instantiation, not a theorem derived from the hypothesis, and the MNIST batch-size experiment (Fig 2c) is an independent falsifiable check. The GPT-2 'faster than Adam' claim is a comparison against an externally reported Adam curve ('We copy the optimal performance of Adam (trained in bfloat16) from Liu et al. (2023a)'), so any problem with that baseline (precision, hardware, hyperparameter tuning, no same-environment stable Adam run) is an experimental-validity risk, not a definitional circularity. The paper itself flags the bfloat16 issue ('possibly due to the use of bfloat16') and its inability to tune FOCUS systematically ('Due to our limited computing resources, we cannot scan optimal performance for FOCUS systematically on GPT-2 training'). Self-citations (Liu et al. 2023b; Liu 2024; Large et al. 2024) appear only as supporting asides and are not load-bearing for the central claim. No equation or fitted parameter is reused as a prediction by construction, so no circular step can be exhibited; the score reflects only minor non-load-bearing self-references.
Assumptions & free parameters
free parameters (7)
- toy landscape sharpness a =
varied 0.1 to 1000 in Fig 2a; 10 in Fig 4
- toy landscape slope c =
0.1
- gradient noise std sigma (toy) =
0 to 3, swept
- FOCUS attraction strength gamma =
0.2
- FOCUS EMA decay beta2 (GPT-2) =
0.99
- FOCUS learning rate for fastest run =
1e-3
- weight decay (GPT-2 FOCUS) =
0.2
assumptions (5)
- domain assumption LLM pretraining loss landscape is a narrowing valley with heterogeneous curvature.
- domain assumption Gradient noise in LLM pretraining is large relative to valley sharpness and is captured by multiplicative zero-mean noise.
- standard math Convexity, bounded gradients, and bounded feasible set in regret analysis.
- ad hoc to paper The regret bound extends to Algorithm 1 with constant beta1, gamma and weight decay.
- domain assumption EMA of parameters theta_hat_t is a useful attraction center for staying in the valley.
Cite this review
Pith. "Pith review of FOCUS: First Order Concentrated Updating Scheme." pith.science (2026). https://pith.science/paper/Z3T6UUB5
@misc{pith2026250112243,
author = {Pith},
title = {Pith review of: FOCUS: First Order Concentrated Updating Scheme},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z3T6UUB5}},
note = {Machine review of arXiv:2501.12243}
}
read the original abstract
Large language models (LLMs) demonstrate remarkable performance, and improving their pre-training process appears to be key to enhancing their capabilities further. Based on the documented success of Adam, learning rate decay, and weight decay, we hypothesize that the pre-training loss landscape features a narrowing valley structure. Through experiments with synthetic loss functions, we discover that when gradient query noise is high relative to the valley's sharpness, Adam's performance falls behind that of Signum because Adam reduces the effective step size too drastically. This observation led us to develop FOCUS, an optimizer that enhances Signum by incorporating attraction toward moving averaged parameters, allowing it to handle noise better while maintaining larger step sizes. In training GPT-2, FOCUS proves to be more stable than Signum and faster than Adam. These results suggest that gradient noise may be an underappreciated limiting factor in LLM training, and FOCUS offers promising solutions.
Figures
Figures from the paper (5 more)
Forward citations
Cited by 3 Pith papers
-
Universal One-third Time Scaling in Learning Peaked Distributions
Softmax + cross-entropy on peaked targets yields loss ∼ t^{−1/3}, giving an architecture-driven explanation for power-law LLM training time without power-law data.
-
OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
A meta-pipeline plus LMO four-axis view yields a dual taxonomy of 108 optimizers, and a multi-objective LLM/vision benchmark shows no single family dominates the quality–cost–memory frontier.
-
Mpemba Effect in Large-Language Model Training Dynamics: A Minimal Analysis of the Valley-River model
The authors derive conditions under which a higher plateau learning rate accelerates LLM training convergence via the Mpemba effect, but the central derivative formula is incorrect.
Reference graph
Works this paper leans on
-
[1]
Adam optimizer with β1 = 0.9, β2 = 0.999
-
[2]
FOCUS optimizer with β1 = 0.9, β2 = 0.9, γ = 0.2
-
[3]
Signum optimizer with β1 = 0.9 The experiment shown in Figure 2a systematically explores the following parameter ranges: • Landscape sharpness ( a): 96 values logarithmically spaced between 10−1 and 103 • Noise standard deviation: 100 values linearly spaced between 0 and 3 • Learning rates: 20 values logarithmically spaced between 10−3 and 100 Each config...
-
[4]
Historical minimum loss achieved
-
[5]
12 FOCUS: First Order Concentrated Updating Scheme A.2
Projection onto the valley axis (defined by rotation angle π/6) The implementation includes several important technical considerations: • Gradient computation incorporates multiplicative noise with a controlled standard deviation • Weight decay is implemented as an additive term in parameter updates • Adam uses ϵ = 10−8 for numerical stability in its deno...
-
[6]
AdamW with standard parameters ( β1 = 0.9, β2 = 0.999)
-
[7]
Signum with β1 = 0.9
-
[8]
FOCUS with β1 = 0.9, β2 = 0.99, γ = 0.2
Show all 22 references
-
[9]
FOCUS with β1 = 0.9, β2 = 0.99, γ = 0.4 The experiment systematically explores the following parameter ranges: • Learning rates: 20 values logarithmically spaced between 10−4 and 100 • Batch sizes: 12 values as powers of 2, ranging from 21 to 212 • Weight decay: Fixed at 10−2 ...
2019
-
[10]
Signum with learning rate 6 × 10−4, β1 = 0.9, weight decay 0.2, 105 training steps
-
[11]
Signum with learning rate 10−4, β1 = 0.9, weight decay 0.2, 105 training steps
-
[12]
FOCUS with learning rate 6 × 10−4, β1 = 0.9, β2 = 0.99, γ = 0.2, weight decay 0.2, 105 training steps Figure 3b compares four optimization configurations:
-
[13]
Adam with learning rate 6 × 10−4, β1 = 0.9, β2 = 0.95, weight decay 0.1, 105 training steps
-
[14]
Adam with learning rate 3 × 10−4, β1 = 0.9, β2 = 0.95, weight decay 0.1, 105 training steps
-
[15]
Adam with learning rate 10−4, β1 = 0.9, β2 = 0.95, weight decay 0.1, 105 training steps
-
[16]
FOCUS with learning rate 6 × 10−4, β1 = 0.9, β2 = 0.99, γ = 0.2, weight decay 0.2, 105 training steps Figure 3c compares three optimization configurations:
-
[18]
Training dynamics reaches steady state on the narrowing valley
Adam with learning rate 10−4, β1 = 0.9, β2 = 0.95, weight decay 0.1, 105 training steps 14 FOCUS: First Order Concentrated Updating Scheme Figure 4. Training dynamics reaches steady state on the narrowing valley. The left panel isL(u, v= 0) =−cu showing our toy landscape can g...
-
[19]
FOCUS with learning rate 6 × 10−4, β1 = 0.9, β2 = 0.99, γ = 0.2, weight decay 0.2, 105 training steps Figure 3d compares four optimization configurations:
-
[20]
(2023a))
Adam with learning rate 6 × 10−4, β1 = 0.9, β2 = 0.95, weight decay 0.1, 105 training steps (result copied from Liu et al. (2023a))
2023
-
[21]
FOCUS with learning rate 6 × 10−4, β1 = 0.9, β2 = 0.99, γ = 0.2, weight decay 0.2, 105 training steps
-
[22]
FOCUS with learning rate 6 × 10−4, β1 = 0.9, β2 = 0.99, γ = 0.2, weight decay 0.2, 5 × 104 training steps
-
[23]
entropic
FOCUS with learning rate 10−3, β1 = 0.9, β2 = 0.99, γ = 0.2, weight decay 0.2, 2 × 104 training steps B. Supplementary results The landscape we construct can go to negative infinity (see Figure 4 right panel) which is unrealistic. Our construction serves as a good local approx...
2020
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.