REVIEW 4 major objections 4 minor 14 references
PsiLogic: Chaos-Aware Active Cancellation for Adam with a Fair Cross-Domain Benchmark
T0 review · 4 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read A chaos-gated damping term lets Adam beat AdamW in three of four benchmark arenas.
desk verdict A promising optimizer whose headline results are confounded by its own warmup schedule: the LR sweep tunes AdamW, not PsiLogic. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Carrying the argument is the chaos detector: a fast EMA and a slow EMA of the scale-normalized gradient norm, together with a ratio gate (default threshold 2.0) and a tanh-based chaos score. The score gates an active cancellation term—per-step parameter shrinkage that is scaled by learning rate, chaos gain, and a per-group amplification factor, then clamped by a max-cancel cap and combined with weight decay into a single unified decay coefficient. The unified decay avoids the over-shrinking that would come from multiplying separate decay and cancellation factors, and the whole mechanism acts as an implicit warmup with no hand-tuned schedule.
What would settle it
Run the published harness with, say, 20 seeds and 10,000 steps per arena; if the PsiLogic–AdamW gaps on NLP and ViT collapse into overlapping confidence intervals, or a fixed damping schedule matched to PsiLogic's average cancellation reproduces the gains, the central claim is falsified.
Extended reading notes
Core claim
PsiLogic's claim is that the instability of gradient norms is a usable online signal for when an adaptive optimizer should actively shrink parameters. A fast EMA (10-step horizon) and a slow EMA (100-step horizon) of the normalized gradient norm produce a ratio; when the fast EMA exceeds twice the slow one, a chaos score becomes active, and the score scales an active cancellation term that is folded with weight decay into one unified decay coefficient. During the chaotic early phase this term reduces the effective step; when the slow EMA decays toward zero, the term vanishes and PsiLogic approaches AdamW. In the FairBench protocol the augmented optimizer achieves the best validation metric i
Load-bearing premise
The load-bearing premise is that FairBench's three-seed, 2000-step protocol with a seven-point learning-rate sweep can distinguish genuine optimizer-quality differences from initialization and tuning luck.
Editorial extensions
If this is right
- Practitioners can drop PsiLogic into existing Adam/AdamW code and, under similar tuning, expect improved validation perplexity and accuracy on transformer and CNN training at short horizons.
- The chaos term may remove the need for a separately tuned LR warmup in regimes like those tested, since it damps the same unstable early phase automatically.
- Per-group amplification allows sensitive parameter groups, such as embeddings, to receive stronger cancellation during chaos without globally changing the learning rate.
- The method costs almost no extra memory—two scalar EMAs shared across parameters—so peak GPU memory stays comparable to Adam/AdamW; only step time rises in the current implementation.
- The explicit reporting of ties and non-significant results is itself a contribution: the FairBench protocol offers a template for less biased optimizer comparisons.
Reading between the lines
- If the chaos-gate signal is the active ingredient, the same dual-EMA gate could be bolted onto other base optimizers to test whether chaos-aware damping is a universal stabilizer.
- An obvious control the paper only partially sketches: replace the chaos gate with a fixed damping schedule whose mean cancellation matches PsiLogic's; if the fixed schedule reproduces the gains, the online chaos signal is not essential.
- The reported wall-clock gap is implementation-bound, so a fused rewrite of the elementwise operations should close most of the 1.2–1.8x overhead; this is a testable engineering claim.
- If the implicit warmup truly replaces hand-tuned warmup, PsiLogic could be combined with parameter-free learning-rate methods to produce a schedule-free optimizer, a direction the paper mentions but does not test.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces PsiLogic, an Adam-family optimizer that augments the standard update with a chaos-gated active cancellation term. A dual EMA of scale-normalized gradient norms feeds a chaos detector; when the fast/slow EMA ratio indicates instability, a damping term is applied, and it fades as training stabilizes, providing an implicit warmup. The authors also present FairBench, a two-stage benchmark protocol with per-optimizer learning-rate sweeps, identical per-seed initialization, and Welch t-tests. On an H100 reference run across four arenas (NLP, ViT, ResNet, diffusion), PsiLogic is reported to achieve the best validation metric in three of four arenas, with several comparisons statistically significant and two tied. The paper includes an open-source implementation, harness, raw CSVs, and explicit limitations.
Significance. If the central claim is sustained, PsiLogic would be a simple, drop-in addition to Adam that improves stability and final metrics without a hand-tuned warmup schedule, and FairBench would be a useful template for less biased optimizer evaluation. The paper's strengths are its reproducibility artifacts (code, harness, raw outputs), honest reporting of non-significant and negative results, and the explicit limitation list. However, the current evidence is not yet solid: the protocol has a concrete warmup/sweep mismatch that undermines the fairness premise, the statistical power is low (3 seeds, 2000 steps, small models), and the contribution of the chaos gate versus a longer warmup is not isolated. These issues are fixable within the scope of a revision.
major comments (4)
- [§3.5/§4.1, Table 3, Eq. (7)] The Stage 1 LR sweep runs 500 steps per learning rate. For PsiLogic, chaos_warmup=-1 sets t_warm=max(500,T/20)=500, so w_t=0 throughout the sweep and δ_t=ηλ; the cancellation term is never active and PsiLogic is exactly plain AdamW during LR selection. In Stage 2, T=2000, so t_warm=500 and the chaos term becomes active after step 500. The algorithm whose LR was swept is therefore not the algorithm whose final metrics are reported. This breaks the 'per-optimizer LR sweep' fairness premise. Consistent with this, the selected LRs in §4.2 for PsiLogic are identical to AdamW in NLP, ViT, and ResNet. Re-running the sweep with the gate active (e.g., t_warm < 500 or sweeping warmup) is required.
- [§5/§6] The shared settings in Table 3 give baselines a 100-step warmup, while PsiLogic's auto-warmup produces w_t=0 for 500 steps plus a 125-step ramp in Stage 2, i.e., a much longer effective warmup. The text itself states in §6 that the improvement over hand-tuned warmup is not established, and the ablation in §5 is on a synthetic MLP predating FairBench, not on the four arenas. Since the chaos gate is off for the first 500 steps, the observed gains over Adam/AdamW could be attributable to a longer warmup schedule rather than the chaos-aware cancellation. A matched-warmup comparison or an ablation with the chaos term disabled (while keeping the warmup shape) is needed to support the central claim.
- [§4.2/Appendix B] The statistical support is thin. The headline 'three of four arenas' relies on 3 seeds and 2000 steps; NLP perplexity vs AdamW is p=0.049 (just below 0.05), ResNet vs AdamW is not significant (p=0.44), NLP val loss vs AdamW is borderline (p=0.054), and diffusion is tied. The paper honestly lists these limitations, but the combination of low seed count and short training budget makes the empirical claim fragile. Reporting effect sizes, confidence intervals, or additional seeds would help, and the protocol flaw above compounds this concern.
- [§3.1/§3.4/§4.1] Several hyperparameters that affect the method are not fully specified in the text: the default value of the chaos gain γ is not stated, and the per-arena presets (P_k, gamma_T_max, etc.) are referenced only as fixed config.json. Since reproducibility relies on the code repository, this is not fatal, but the paper should list all defaults and presets to make the optimizer self-contained and to allow independent reimplementation without reading source code.
minor comments (4)
- [§3.3, Eq. (5)] The relation between the gate condition fast_t > τ_scale·slow_t and the ratio term in Eq. (5) is not fully explained. It would help to define when s_t=1 explicitly and to state whether the relu(ratio-1) term is redundant given the gate.
- [§4.1, Table 3] The symbol T is used for training steps in §3.5, but Table 3 reports Stage 1 as 500 steps and Stage 2 as 2000 steps without naming which value T refers to. Clarify to avoid ambiguity.
- [§5] The mirror ablation is described briefly and reported to predate FairBench. In a revision, please include those results in the Appendix or provide a reference to the repository location so readers can verify the claim that the chaos signal is not equivalent to a global weight-decay schedule.
- [§6] The sentence 'The cancellation term reduces effective step size during chaotic phases' is intuitive, but a precise account of how δ_t interacts with the Adam step size would be useful, especially because the unified-decay term is applied before the Adam update.
Circularity Check
No significant circularity: the chaos gate is an open-loop gradient statistic and the benchmark results are measured, not derived from the fit; the LR-sweep/warmup overlap is a protocol concern, not a circular reduction.
full rationale
PhiLogic's derivation chain is an explicitly specified update rule plus an empirical benchmark. The chaos detector (Eqs. 2-5) is an open-loop function of current gradient norms - fast/slow EMAs of ||g_t||_2/sqrt(numel) and a ratio gate - with no parameter fitted to the reported validation metrics. The cancellation term enters the update as multiplicative damping (Eqs. 6-7), and the paper's ablation states that mirroring its magnitude as AdamW weight decay does not reproduce its per-parameter behavior, so the term is not merely a renamed decay schedule. Table 2 metrics are measured on the actual optimizer in Stage 2 with identical initialization per seed, and ties/non-significant results are disclosed. The only self-referential evidence is the archived v0.3.x synthetic MLP ablation, which is not load-bearing for the main benchmark claims. One protocol-level issue does exist: Stage 1's LR sweep runs exactly 500 steps (Table 3), and for chaos_warmup=-1 the auto-warmup horizon is max(500,T/20) (Sec. 3.5), so w_t=0 throughout the sweep and Eq. (7) reduces to delta_t=eta*lambda, making the tuned configuration AdamW-equivalent. This is a fairness/validity concern about whether the selected LR is optimal for the full PsiLogic algorithm, but it does not make the reported PsiLogic metrics equivalent to fitted inputs by construction; the final numbers are measured, not algebraically forced. Therefore no circularity step is scored.
Assumptions & free parameters
free parameters (5)
- tau_scale =
2.0 in adaptive mode
- max_cancel (cmax) =
0.05
- gamma (chaos gain) =
not reported in the text
- chaos_warmup =
auto max(500, T/20) by default
- EMA rates 0.10 and 0.01 =
0.10 and 0.01
assumptions (2)
- domain assumption Logging in log-space produces Gaussian noise so Welch t-tests are valid.
- domain assumption The MSE for the two seeds in the diffusion rows has std 0.00006 despite an apparent difference of ~0.0002 in the means, which yields a non-significant (p=0.49) test.
Cite this review
Pith. "Pith review of PsiLogic: Chaos-Aware Active Cancellation for Adam with a Fair Cross-Domain Benchmark." pith.science (2026). https://pith.science/paper/5YXHS5VL
@misc{pith2026260716268,
author = {Pith},
title = {Pith review of: PsiLogic: Chaos-Aware Active Cancellation for Adam with a Fair Cross-Domain Benchmark},
year = {2026},
howpublished = {\url{https://pith.science/paper/5YXHS5VL}},
note = {Machine review of arXiv:2607.16268}
}
read the original abstract
Adaptive optimizers such as Adam and AdamW apply the same update rule regardless of whether training is in a chaotic early phase or near convergence. We introduce PsiLogic, an optimizer that augments Adam with a dynamic Active Cancellation Term gated by a dual exponential moving average (EMA) of scale-normalized gradient norms. The resulting chaos detector strengthens damping when gradient statistics are unstable and fades to zero as training stabilizes, providing an implicit warmup without a hand-tuned schedule. We evaluate PsiLogic against Adam, AdamW, and Lion using FairBench -- a reproducible benchmark protocol with per-optimizer learning-rate sweeps, identical initialization per seed, and Welch t-tests. On an NVIDIA H100 80GB reference run (4 arenas, 3 seeds, 2000 steps, bf16 AMP), PsiLogic achieves the best validation metric in three of four arenas: NLP perplexity 7.79 +/- 0.18 vs. 8.17 +/- 0.08 (AdamW, p = 0.049), ViT top-1 accuracy 0.244 +/- 0.006 vs. 0.223 +/- 0.002 (AdamW, p = 0.015), and ResNet top-1 accuracy 0.222 +/- 0.001 vs. 0.172 +/- 0.004 (Adam, p = 0.001). On diffusion, validation MSE is statistically tied with Adam/AdamW (p = 0.49). ResNet accuracy vs. AdamW is a numerical tie without significance at three seeds (p = 0.44). Peak GPU memory is comparable across optimizers; PsiLogic incurs 1.2--1.8x wall-clock overhead on transformer-heavy arenas (implementation-bound). We release an open-source PyTorch implementation, the full FairBench harness, and all raw CSV outputs to support independent verification.
Figures
Reference graph
Works this paper leans on
-
[1]
A. G. Baydin, R. Cornish, M. Rubinstein, and D. M. Wood. Online learning rate adaptation with hypergradient descent.ICLR, 2018
2018
-
[2]
Brock, et al
A. Brock, et al. High-performance large-scale image recognition without normalization.ICML, 2021
2021
-
[3]
X. Chen, C. Liang, D. Huang, E. Real, K. Wang, Y. Liu, et al. Symbolic discovery of optimization algorithms.NeurIPS, 2023
2023
-
[4]
Defazio and K
M. Defazio and K. Mishchenko. Learning-rate-free learning by D-Adaptation.ICML, 2023
2023
-
[5]
P. Goyal, et al. Accurate, large minibatch SGD.arXiv preprintarXiv:1706.02677, 2017
arXiv 2017
-
[6]
Gupta, T
V. Gupta, T. Koren, and Y. Singer. Shampoo: Preconditioned stochastic tensor optimization. ICML, 2018
2018
-
[7]
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization.ICLR, 2015
2015
-
[8]
H. Liu, Z. Shen, Y. Li, S. Lin, K. Wang, and L. Ma. Sophia: A scalable stochastic second-order optimizer.ICLR, 2024
2024
Show all 14 references
-
[9]
Loshchilov and F
I. Loshchilov and F. Hutter. Decoupled weight decay regularization.ICLR, 2019
2019
-
[10]
Mishchenko and M
K. Mishchenko and M. Defazio. Prodigy: An expeditiously adaptive parameter-free learner. arXiv preprintarXiv:2306.06169, 2023. 7
2023 arXiv
-
[11]
Shazeer and M
N. Shazeer and M. Stern. Adafactor: Adaptive learning rates with sublinear memory cost. ICML, 2018
2018
-
[12]
H. Yong, J. Huang, X. Hua, and L. Zhang. Gradient centralization.ECCV, 2020
2020
-
[13]
Y. You, I. Gitman, and B. Ginsburg. Large batch training of convolutional networks with layer-wise adaptive rate scaling.arXiv preprintarXiv:1708.03888, 2017
2017 arXiv
-
[14]
Y. You, J. Li, S. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. Song, J. Demmel, and C.- J. Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes.ICLR, 2020. A FairBench Protocol and Arenas Table 3: FairBench protocol stages. Stage Description Stage 1—...
2020
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.