REVIEW 3 major objections 6 minor 24 references
Recovering Plasticity of Neural Networks via Soft Weight Rescaling
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Soft Weight Rescaling recovers neural-network plasticity by scaling weights toward their initial norms every update step, a move the paper proves bounds weight growth and balances layers.
desk verdict A cheap regularization trick with real empirical promise, but the boundedness proof assumes what it should prove; needs a revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the proportionality of feed-forward networks with homogeneous activations (Theorem 1): for any set of positive layer scalers $c_l$ with product $C$, rescaling $W_l$ by $c_l$ and $b_l$ by the cumulative product of preceding $c_i$ multiplies the whole network function by $C$. SWR uses this identity to shrink weights every step while preserving model behavior. The per-layer scaling factor $c_l = (\lambda \|W_0\| + (1-\lambda) \|W_l\|)/\|W_l\|$ acts as an EMA-like pull toward the initial norm, and the mediant structure of the rescaled norm ratios is what Theorem 3 exploits to prove improved balancedness.
What would settle it
Track $\max_l \left| \|W_{t,l}\|^2 - \|W_{t-1,l}^{c}\|^2 \right|$ across training on a standard benchmark; if this quantity grows without bound as $t$ increases, the fixed constant $B$ in Theorem 2 does not exist and the claimed bounded-weight guarantee fails. Alternatively, find a dataset or architecture where SWR's weight norms still diverge despite the algorithm, which would refute the method's central mechanism.
Extended reading notes
Core claim
The paper's central claim is that unbounded weight growth, not just the loss of representational capacity, is a primary driver of plasticity loss, and that a simple per-step rescaling operation can arrest that growth without erasing learned knowledge. The scaling factor for layer l is $c_l = (\lambda \|W^{\mathrm{init}}_l\| + (1-\lambda) \|W_l\|) / \|W_l\|$, chosen so the layer norm moves toward initialization at a rate controlled by $\lambda$. Because homogeneous activations make the network proportional under simultaneous weight and bias rescaling (Theorem 1), this shrinkage does not distort classification behavior. Theorem 2 shows that if per-step squared Frobenius norm changes are bounded, SWR bounds each weight norm; Corollary 2.1 gives a Lipschitz bound, and Theorem 3 shows norms become more balanced across layers. Empirically, SWR improves test accuracy in warm-start, continual (full- and limited-access), and single-task settings, notably exceeding re-initialization methods on VGG-16 without the information loss they incur.
Load-bearing premise
The theoretical bound on weight growth assumes that the change in squared Frobenius norm caused by a single gradient update is uniformly bounded by a constant across all layers and all training steps, an assumption the paper neither derives nor verifies empirically.
Editorial extensions
If this is right
- Continual learning can be done from a single network without re-initialization events, because SWR preserves learned features while keeping the model plastic.
- On larger architectures such as VGG-16, where weight decay brings little improvement, SWR raises test accuracy by more than four points in the single-task experiments.
- The per-step cost is one Frobenius norm and a scalar multiply per layer, so the method is nearly free to add to an existing training loop.
- The Lipschitz bound in Corollary 2.1 caps output sensitivity, which gives SWR-trained networks a formal robustness property.
- With learning-rate decay, resetting the reference initial norm at each decay (the paper's SWR + re-init variant) keeps the regularization effective and is competitive with baselines.
Reading between the lines
- The proportionality proof could generalize to other architectures with homogeneous components, so SWR may transfer to attention-based or recurrent models, though the paper does not test this.
- Because Theorem 2's constant $B$ is never measured, a practical check is to log the per-step squared Frobenius norm changes during training; if that maximum grows with training time, the theoretical guarantee is vacuous even if the method still helps.
- The mediant argument suggests that the optimal $\lambda$ should scale with layer growth rates, so tuning $\lambda$ per layer or per depth could yield larger gains than the single shared coefficient used here.
- The authors' split of $\lambda_f$ and $\lambda_c$ implies that regularizing the feature extractor and classifier have different roles; one could design schedules where the classifier is regularized harder early and the feature extractor later.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes Soft Weight Rescaling (SWR), a per-layer post-update rescaling method that scales each weight matrix toward its initial norm by a coefficient λ, while scaling biases by a cumulative factor to preserve the network's function up to a global multiplicative constant (a property the authors call proportionality). The authors prove a proportionality construction (Theorem 1), claim a boundedness guarantee for SWR (Theorem 2), claim that SWR improves the balance of weight norms across layers (Theorem 3), and report experiments on warm-start, continual, and single-task image classification showing that SWR often outperforms L2, L2 Init, S&P, and head reset.
Significance. The proportionality observation is elegant and correctly proven for homogeneous networks without normalization; it gives a principled reason why scaling weights at every step need not destroy the information accumulated by gradient descent. The empirical results are reasonably broad (five architectures/datasets, three learning scenarios) and the gains on VGG-16/TinyImageNet are substantial. However, the paper's main theoretical contribution — the boundedness theorem — rests on a premise that is not verified and is plausibly equivalent to the conclusion, and the proof as written contains gaps. Because the boundedness guarantee is announced in the abstract and introduction as a proven result, the manuscript requires revision to either supply a meaningful, verifiable premise or replace the theorem with a correct proof under standard optimizer assumptions.
major comments (3)
- [Appendix B, Theorem 2 (Eq. (6))] The premise that |∥W_t∥² − ∥W^c_{t−1}∥²| is bounded by a global constant B for every step and every layer is not established and is plausibly circular. Writing W_t = W^c_{t−1} + δ_t, this quantity equals |2⟨W^c_{t−1}, δ_t⟩ + ∥δ_t∥²|; for any optimizer with bounded update norm (e.g., Adam), it can be as large as 2D∥W^c_{t−1}∥ + D², so a uniform B exists only if the post-SWR norms are already bounded, which is exactly the conclusion the theorem aims to prove. The O(√t) remark in Appendix B does not validate the premise for the SWR trajectory, and no empirical measurement of B is reported. To make the result non-circular, replace the assumption with a bound on ∥δ_t∥ (satisfied, for example, by Adam's per-coordinate step size) and prove boundedness via a contraction argument on the post-SWR norm.
- [Appendix B, proof of Theorem 2 (Eqs. (10)–(12))] The proof does not justify the stated bound ∥W_t∥ ≤ B′. The argument that once the norm exceeds B′ it will no longer increase does not preclude the norm from crossing B′ from below. A correct proof would need an induction showing that the post-SWR norm at each step is at most B′ − √B, which is not supplied. The proof also restricts attention to the case where the gradient update increases the norm, even though the assumption in Eq. (6) bounds the absolute change; the decreasing case is not addressed.
- [Section 3.4 and Appendix C.2 (Theorem 3)] Theorem 3 is proved for a single coefficient λ applied to both layers, but in the experiments the scaling coefficient is split into λ_f and λ_c for the feature extractor and classifier (Section 3.3 and Tables 2–4). Equation (19) is only a generalized mediant when both layers share the same λ; with different coefficients, the proof does not apply. The abstract's unqualified claim that SWR balances weight magnitude between layers therefore exceeds what is proven for the algorithm as evaluated.
minor comments (6)
- [Abstract and Section 4.3] The claim that SWR improves performance in single-task learning is stronger than Table 1 supports for CNN-BN on CIFAR-100, where L2 achieves higher mean accuracy (0.4222 vs 0.4129); please qualify the claim as holding in most settings.
- [Section 3.3 and Algorithm 1] The extension of Theorem 1 to networks with normalization layers is stated without proof; please provide a formal argument that the bias scaling in Algorithm 1 preserves proportionality when normalization layers are present.
- [Figures 2–5] No error bars or confidence intervals are shown despite results being averaged over seeds; please add variability information.
- [Appendix C.1 vs C.2] Appendix C.1 uses a different balancedness metric (ratio of the entry-wise ℓ_p,q norm to its global minimum) from the one defined in Definition 2 and used in Theorem 3; please clarify the relationship or unify the metrics.
- [Section 4.2 and Table 3] Hyperparameters are separately tuned for full-access and limited-access continual settings, which effectively selects hyperparameters on the evaluation scenario; please state this limitation or use a validation-based selection procedure.
- [Appendix A] The proof of Theorem 1 contains small notational slips in the unfolding of the recursion; a cleaner derivation would improve readability.
Circularity Check
No significant circularity: SWR's theoretical claims are conditional and do not reduce to their inputs, and the empirical results are not derived from the fitted lambda values.
full rationale
The paper's derivation chain is self-contained with respect to the circularity patterns in scope. Theorem 1 proves a general proportionality identity for homogeneous networks from explicit scaling rules, and it does not assume SWR. Theorem 2 is explicitly conditional: it assumes a uniform bound B on the per-step change of squared Frobenius norm and then derives a norm bound; the assumption is not defined in terms of the conclusion, and the theorem does not rename a fitted parameter as a prediction. Theorem 3 shows that the norm-growth ratio moves toward the initial ratio under the SWR update, which is a direct algebraic consequence of the mediant form rather than a restatement of the desired balancedness. The experiments tune lambda per dataset, but lambda is a free hyperparameter and the theoretical claims do not depend on the tuned values. The citations are to external prior work; there is no load-bearing self-citation chain. The genuine weakness is that the uniform bound B in Theorem 2 is never measured, and the proof's step (12) only controls post-SWR norms while writing a pre-SWR bound; however, this is a proof-gap and empirical-verification concern, not a circular reduction, so it does not increase the circularity score.
Assumptions & free parameters
free parameters (2)
- lambda (scaling strength) =
0.0001 to 0.1; depends on dataset and model (Appendix D Tables 2-4)
- lambda_f and lambda_c (feature and classifier coefficients) =
e.g., lambda_f=1e-2, lambda_c=1e-1 for VGG-16 generalization; varies
assumptions (3)
- domain assumption Homogeneous activation functions (e.g., ReLU, Leaky ReLU) for the proportionality theorem.
- ad hoc to paper Bounded per-step change of squared Frobenius norm by a global constant B.
- domain assumption 1-Lipschitz activation functions for the Lipschitz bound in Corollary 2.1.
Cite this review
Pith. "Pith review of Recovering Plasticity of Neural Networks via Soft Weight Rescaling." pith.science (2026). https://pith.science/paper/F2O53OKB
@misc{pith2026250704683,
author = {Pith},
title = {Pith review of: Recovering Plasticity of Neural Networks via Soft Weight Rescaling},
year = {2026},
howpublished = {\url{https://pith.science/paper/F2O53OKB}},
note = {Machine review of arXiv:2507.04683}
}
read the original abstract
Recent studies have shown that as training progresses, neural networks gradually lose their capacity to learn new information, a phenomenon known as plasticity loss. An unbounded weight growth is one of the main causes of plasticity loss. Furthermore, it harms generalization capability and disrupts optimization dynamics. Re-initializing the network can be a solution, but it results in the loss of learned information, leading to performance drops. In this paper, we propose Soft Weight Rescaling (SWR), a novel approach that prevents unbounded weight growth without losing information. SWR recovers the plasticity of the network by simply scaling down the weight at each step of the learning process. We theoretically prove that SWR bounds weight magnitude and balances weight magnitude between layers. Our experiment shows that SWR improves performance on warm-start learning, continual learning, and single-task learning setups on standard image classification benchmarks.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
The impact of reinitialization on generalization in convolutional neural networks
Ibrahim Alabdulmohsin, Hartmut Maennel, and Daniel Keysers. The impact of reinitialization on generalization in convolutional neural networks. arXiv preprint arXiv:2109.00267,
-
[2]
If the change of squared Frobenius norm of the weight matrix, resulting from the single gradient update, is bounded by a constant for all weight matrices in the neural network, then SWR for every update step with fixed coefficient λ bounds the Frobenius norm of the weight matrix. Proof. It is enough to show the case where the gradient update increases the...
work page 2021
-
[6]
Dsd: Dense-sparse-dense training for deep neural networks
Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Enhao Gong, Shijian Tang, Erich Elsen, Peter Va- jda, Manohar Paluri, John Tran, et al. Dsd: Dense-sparse-dense training for deep neural networks. arXiv preprint arXiv:1607.04381,
-
[11]
Maintaining plasticity via regenerative regularization
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity via regenerative regularization. arXiv preprint arXiv:2308.11958,
-
[12]
Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks
Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jaegul Choo, and Clare Lyle. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. arXiv preprint arXiv:2406.02596,
-
[14]
Normalization and effective learning rates in reinforcement learning
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, James Martens, Hado van Hasselt, Razvan Pascanu, and Will Dabney. Normalization and effective learning rates in reinforcement learning. arXiv preprint arXiv:2407.01800, 2024a. Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of pla...
arXiv 2010
-
[16]
Step Out and Seek Around: On Warm-Start Training with Incremental Data
ISSN 2835-8856. URL https://openreview.net/forum? id=uaHyXxyp2r. Maying Shen, Hongxu Yin, Pavlo Molchanov, Lei Mao, and Jose M Alvarez. Step out and seek around: On warm-start training with incremental data. arXiv preprint arXiv:2406.04484,
-
[18]
Very deep convolutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556,
Show all 24 references
-
[19]
Four things everyone should know to improve batch normalization
Cecilia Summers and Michael J Dinneen. Four things everyone should know to improve batch normalization. arXiv preprint arXiv:1906.03548,
1906 arXiv
-
[20]
L2 regularization versus batch and weight normalization
Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05350,
-
[21]
Spectral norm regularization for improving the generalizability of deep learning
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv:1705.10941,
-
[22]
Three mechanisms of weight decay regularization
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. arXiv preprint arXiv:1810.12281,
-
[28]
CNN: We employed a Convolutional Neural Network (CNN), which is used in relatively small image classification
input size and 10 output size are fixed since MLP is only trained in the MNIST dataset. CNN: We employed a Convolutional Neural Network (CNN), which is used in relatively small image classification. The model includes two convolutional layers with a 5 × 5 kernel and 16 channel...
1991
-
[1991]
Implicit under-parameterization inhibits data-efficient deep reinforcement learning
9 Preprint Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parameterization inhibits data-efficient deep reinforcement learning. arXiv preprint arXiv:2010.14498,
2010 arXiv
-
[1996]
A study on the plasticity of neural networks
Tudor Berariu, Wojciech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pas- canu, and Claudia Clopath. A study on the plasticity of neural networks. arXiv preprint arXiv:2106.00042,
-
[2015]
Layer-wise weight decay for deep neural networks
Masato Ishii and Atsushi Sato. Layer-wise weight decay for deep neural networks. In Image and Video Technology: 8th Pacific-Rim Symposium, PSIVT 2017, Wuhan, China, November 20-24, 2017, Revised Selected Papers 8, pp. 276–289. Springer,
2017
-
[2016]
Projection based weight normalization for deep neural networks
Lei Huang, Xianglong Liu, Bo Lang, and Bo Li. Projection based weight normalization for deep neural networks. arXiv preprint arXiv:1710.02338,
-
[2017]
Batch normalization: Accelerating deep network training by reducing internal covari- ate shift
Sergey Ioffe. Batch normalization: Accelerating deep network training by reducing internal covari- ate shift. arXiv preprint arXiv:1502.03167,
-
[2018]
Weight clipping for deep continual and reinforcement learning
Mohamed Elsayed, Qingfeng Lan, Clare Lyle, and A Rupam Mahmood. Weight clipping for deep continual and reinforcement learning. arXiv preprint arXiv:2407.01704,
- [2020]
-
[2021]
Why do we need weight decay in modern deep learning? arXiv preprint arXiv:2310.04415,
Maksym Andriushchenko, Francesco D’Angelo, Aditya Varre, and Nicolas Flammarion. Why do we need weight decay in modern deep learning? arXiv preprint arXiv:2310.04415,
-
[2022]
Learn, unlearn and relearn: An online learning paradigm for deep neural networks
Vijaya Raghavan T Ramkumar, Elahe Arani, and Bahram Zonooz. Learn, unlearn and relearn: An online learning paradigm for deep neural networks. arXiv preprint arXiv:2303.10455,
-
[2023]
Understanding the disharmony between weight normaliza- tion family and weight decay
Xiang Li, Shuo Chen, and Jian Yang. Understanding the disharmony between weight normaliza- tion family and weight decay. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pp. 4715–4722, 2020a. Xingjian Li, Haoyi Xiong, Haozhe An, Cheng-Zhong Xu, and...
2021
-
[2024]
Dash: Warm-starting neural network training without loss of plasticity under stationarity
10 Preprint Baekrok Shin, Junsoo Oh, Hanseul Cho, and Chulhee Yun. Dash: Warm-starting neural network training without loss of plasticity under stationarity. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization ...
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.