Pith. sign in

REVIEW 5 major objections 3 minor 19 references

A Simple Baseline for Stable and Plastic Neural Networks

T0 review · 5 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that RDBP—a static activation tweak plus a gradient-annealing schedule—keeps a network learning for 5,000 ImageNet tasks without forgetting, matching leader methods with less computation and no replay buffer.

desk verdict A simple rehearsal-free baseline worth knowing, but the stability metric only looks at the last ten tasks and the DBP layer numbering doesn't add up to the claimed 25%. read the letter →

arxiv 2507.10637 v2 pith:EFIFSUBZ submitted 2025-07-14 cs.LG cs.AIcs.CV

Étienne Künzel , Achref Jaziri , Visvanathan Ramesh This is my paper
classification cs.LGcs.AIcs.CV
keywords continuallearningstability-plasticitydilemmaReLUDowndecreasingbackpropagationplasticitylosscatastrophicforgettingImageNetactivationfunction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a simple, low-overhead baseline, RDBP, can hold the stability-plasticity balance on a long continual-learning stream. The baseline pairs ReLUDown, a static activation that keeps neurons responsive by giving the negative side of ReLU a small linear slope, with Decreasing Backpropagation (DBP), a schedule that shrinks gradient flow into early layers as tasks accumulate. On Continual ImageNet, where each task is binary classification between two classes and the stream runs for 5,000 tasks, RDBP keeps current-task accuracy near the level of a per-task reset network and raises the average accuracy over the last ten tasks as training proceeds. The reported result is that this matches or exceeds the plasticity and stability of Continual Backpropagation and Generative Replay with lower training-time overhead than the adaptive-activation and generative alternatives, and with no replay buffers or architectural changes. If the claim holds, RDBP gives the field a cheap, reproducible yardstick against which more complex continual-learning methods can be measured.

What carries the argument

The central mechanism is the joint action of the activation function ReLUDown and the gradient schedule DBP. ReLUDown is $f(x)=\max(0,x)-\max(0,-x+d)$ with hinge $d<0$ (the experiments use $d=-3$); the extra linear slope on the negative side prevents the preactivation distribution from drifting into the region where ReLU has zero gradients, which the paper identifies as a main cause of plasticity loss. DBP multiplies the backpropagated gradient into layer $l$ at task $n$ by $(1-lf)+lf a^{-n}$, where $f=0.15$ is a decrease factor and $a=1.005$ is a speed factor; the multiplier starts at one and anneals toward $1-lf$, so early layers receive a shrinking share of the update while the classification head remains unrestricted. The paper's plasticity result is carried by the activation; its stability result is carried by the annealing.

What would settle it

Run RDBP on a stream where early-layer features do not transfer, such as tasks with permuted input statistics, and track current-task accuracy after task 1000: if accuracy falls monotonically as DBP anneals the first-layer gradient, the stability gain is bought at the cost of plasticity and the central claim fails for that setting.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that plasticity and stability do not have to be purchased with extra machinery: two modest changes to training are enough. ReLUDown modifies only the activation, adding a linear tail below a negative hinge point ($d=-3$) so that neurons whose preactivations drift negative still receive gradient; the paper shows the preactivation distribution then converges toward a near-normal shape across tasks instead of drifting negative as it does under ReLU. DBP scales the gradient reaching layer $l$ at task $n$ by a factor that starts at 1 and anneals toward $1-lf$, protecting early layers from catastrophic updates while leaving the classification head fully adaptable. In the 5,000-task Continual ImageNet stream, current-task accuracy stays in the band of a network retrained per task, and last-ten-task average accuracy climbs with training, which the paper presents as evidence that the two mechanisms preserve plasticity and stability together.

Load-bearing premise

The load-bearing premise is that the features learned by the early layers during the first tasks remain useful across all 5,000 tasks, so reducing gradient flow into those layers preserves what matters without capping the network's ability to learn the new tasks.

Editorial extensions

If this is right

  • RDBP is a drop-in baseline: swap the activation and add the gradient schedule to an existing convolutional network, with no replay buffer, no architecture growth, and no reset rule.
  • Current-task accuracy stays near the per-task reset level for the full 5,000-task stream, so in this setting plasticity is not traded away for stability.
  • Average accuracy on the last ten tasks rises as training progresses, showing that early-layer protection increasingly helps retention.
  • Training time is below the adaptive-activation (PAU) and generative-replay alternatives: ReLUDown adds about 34% over plain ReLU, and DBP adds almost nothing on top.
  • RDBP offers a fixed simple baseline against which future continual-learning methods can be compared, making it easier to see whether added machinery actually improves stability or plasticity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves untested whether the same mechanisms transfer to transformer architectures; since ReLUDown is static and parameter-free, replacing ReLU with it and applying DBP's layer-wise annealing is a direct low-cost experiment.
  • A domain-incremental stream with shifting input statistics would test the scope limit the paper itself flags: if early-layer features stop transferring, DBP's annealing should eventually cap plasticity.
  • The reported operating point fixes the hinge at $d=-3$ and the annealing constants at $f=0.15$, $a=1.005$; sweeping these parameters would trace the stability-plasticity trade-off rather than living at one point.
  • The stability metric averages the last ten tasks; full per-task accuracy curves would show whether stability comes from genuinely preserving old tasks or from the recent-ten-task window.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. This paper proposes RDBP, a rehearsal-free continual learning baseline that combines two mechanisms: ReLUDown, a static activation function that adds a non-zero-slope tail for negative preactivations below a hinge, and Decreasing Backpropagation (DBP), which multiplies per-layer gradient signals by an annealed factor that decays with task number. The method is evaluated on Continual ImageNet (5000 binary tasks) with a small CNN, against ReLU/Tanh/PAU ablations, Continual Backpropagation, and Generative Replay. Plasticity is measured as current-task accuracy; stability is measured as mean accuracy over the last ten tasks. The paper claims that RDBP matches or exceeds the plasticity and stability of state-of-the-art methods at reduced computational cost.

Significance. RDBP is attractive as a baseline: it needs no memory buffer, no architectural expansion, and introduces only two small modifications; the 5000-task benchmark and the preactivation-distribution analysis are informative, and the appendix reports hyperparameters and per-task training time in detail. If the central claim held, the method would be a useful reference point for the stability-plasticity literature. However, the evaluation as reported is not yet sufficient to establish that claim: the stability metric has a short window, no variance information accompanies the five runs, and the comparison set is too small to justify 'state-of-the-art.' The central idea is defensible and the issues are addressable, so this is a major-revision situation rather than a rejection.

major comments (5)
  1. [Section 4.1, Figure 3 (right)] Stability is defined as the mean accuracy over the previous ten tasks (Section 4.1) and reported in Figure 3 (right). This short-window measure cannot support the abstract's long-term stability claim: a model that forgets all tasks before the last ten would still receive a high stability score. Because the DBP annealing factor has essentially converged by task 1000 (Appendix A.4), the last ten tasks are trained on nearly fixed early features, so the increasing stability curve may reflect convergence to a stationary feature extractor rather than genuine retention. Please add an all-task measure (e.g., average accuracy over all 5000 tasks, per-task forgetting, or probe accuracy on early tasks at the end of training) and show the stability curve over the full stream.
  2. [Section 3.2, Appendix A.4, Figure 7] The DBP specification is internally inconsistent. Section 3.2 defines the layer index l as numbered from front to back and gives the multiplier as (1 - l*f + l*f*a^{-n}); with f = 0.15 and l = 1, the first convolutional layer would anneal to a factor of about 0.85, not 0.25. Appendix A.4 and Figure 7 state that after task 1000 the first convolutional layer receives only 25% of the backpropagation signal, which corresponds to l = 5 (back-to-front numbering) or to a different value of f. Please clarify the layer indexing, state the exact multiplier used for each layer (including the fully connected layers), and make Figure 7 consistent with the formula.
  3. [Appendix A.1, Figure 3] The paper reports five runs (Appendix A.1) but the figures show no error bars, confidence intervals, or statistical tests. The central claim 'matches or exceeds' is a comparative statement, and without inter-run variance the reader cannot determine whether the plasticity curves of RDBP and Continual Backpropagation are statistically equivalent or whether the stability advantage is meaningful. Please report run-level statistics (standard deviations or shaded error regions) and, if possible, paired significance tests for the key comparisons.
  4. [Section 4.1, Abstract] The comparison set is too narrow to support the abstract's 'state-of-the-art methods' claim. The evaluated baselines are Continual Backpropagation, Generative Replay, and activation-function ablations; no modern rehearsal-free regularization or distillation baselines (e.g., EWC, LwF, SI, or recent class-incremental methods) are included, and Generative Replay is a comparatively old approach. Please either restrict the claims to the methods actually compared or include representative state-of-the-art baselines on the same benchmark.
  5. [Section 4.1, Appendix A.4] The three hand-set hyperparameters (d = -3, f = 0.15, a = 1.005) are used for all experiments without a sensitivity analysis. Because the DBP annealing schedule is the entire stability mechanism, the reported stability-plasticity balance may be tied to these particular values. Please add a sensitivity study over f and a (and ideally d), or justify the choices from first principles, so that the baseline is not fitted to the single evaluation setting.
minor comments (3)
  1. [Section 4.2, Appendix A.5.3] The statement that ReLUDown 'reduces training time by approximately 20% compared to PAU' is inconsistent with Figure 10: PAU adds 60% and ReLUDown adds 34% relative to ReLU, so the reduction in overhead is 26 percentage points (about 43% relative). Please correct the claim or the table.
  2. [Section 3.1, Appendix A.6] The sign convention for the hinge point is inconsistent: the main text defines ReLUDown with d < 0, but the hyperparameter table lists 'Hinge Point d 3'. Clarify the sign so the implementation matches the formula.
  3. [Section 4.1, Appendix A.6] The text says the network uses two fully connected layers, while the architecture table lists three linear layers (512->128, 128->128, 128->2); please reconcile this and clarify which layers are subject to DBP and which are reset at each task.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found; RDBP is an engineered empirical baseline whose claims rest on direct benchmark comparisons rather than on reduction to its inputs.

full rationale

The paper does not derive RDBP's performance from its own inputs. ReLUDown is defined explicitly as f(x)=max(0,x)-max(0,-x+d) with d<0, and DBP is defined explicitly as bpdecrease(n,l,f,a)=bpstandard*(1-(l*f)+(l*f)*a^-n); both are proposed mechanisms, not predictions derived from the Continual ImageNet results. The hyperparameters d=-3, f=0.15, and a=1.005 are stated as chosen constants, and no equation in the paper uses the reported accuracy curves to set these values, so there is no fitted parameter presented as a prediction. The only self-citation (Jaziri et al. 2024) appears in related work and is not load-bearing for the method's design or evaluation, and no uniqueness theorem or prior result is invoked to force the approach. The conclusion explicitly labels the transferability assumption as a hypothesis, which is a stated limitation rather than a circular step. The stability metric being the mean accuracy over the previous ten tasks and the absence of error bars are legitimate evaluation concerns, but they concern construct validity and statistical reporting, not circularity of the argument. The central claims are empirical comparisons against Continual Backpropagation and Generative Replay on a public benchmark, making the paper self-contained and externally testable. No circularity is present.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three hand-chosen hyperparameters (d, f, a) and two domain assumptions: that early layers are transferable and that preactivation shift is the main plasticity-loss driver. No new entities are introduced; the biological memory framing is metaphorical, not a mechanistic postulation.

free parameters (3)
  • d (ReLUDown hinge point) = -3
    Hand-chosen; controls the negative-slope onset point of the activation. No sensitivity analysis is provided.
  • f (DBP decrease factor) = 0.15
    Hand-chosen; sets the asymptotic gradient attenuation per layer. The 25% first-conv-layer claim only works with back-to-front layer numbering.
  • a (DBP speed factor) = 1.005
    Hand-chosen; controls how quickly the gradient factor anneals with task number n.
assumptions (3)
  • domain assumption Later layers are more task-specific while early layers encode transferable features
    The core premise of DBP; stated as a hypothesis in the conclusion ('We hypothesize that this algorithm will perform effectively in settings where the underlying task structure remains consistent'). If false, attenuating early-layer gradients would hurt plasticity.
  • domain assumption Plasticity loss is primarily caused by preactivation distribution shift and neuron dormancy
    ReLUDown is designed to counter this specific mechanism, citing Lyle et al. (2024). The paper does not test alternative plasticity-loss causes.
  • domain assumption The Continual ImageNet binary-task benchmark is a representative proxy for continual learning
    All conclusions are drawn from this single benchmark; no other CL benchmark or paradigm is evaluated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Simple Baseline for Stable and Plastic Neural Networks." pith.science (2026). https://pith.science/paper/EFIFSUBZ

@misc{pith2026250710637,
  author       = {Pith},
  title        = {Pith review of: A Simple Baseline for Stable and Plastic Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EFIFSUBZ}},
  note         = {Machine review of arXiv:2507.10637}
}
read the original abstract

Continual learning in computer vision requires that models adapt to a continuous stream of tasks without forgetting prior knowledge, yet existing approaches often tip the balance heavily toward either plasticity or stability. We introduce RDBP, a simple, low-overhead baseline that unites two complementary mechanisms: ReLUDown, a lightweight activation modification that preserves feature sensitivity while preventing neuron dormancy, and Decreasing Backpropagation, a biologically inspired gradient-scheduling scheme that progressively shields early layers from catastrophic updates. Evaluated on the Continual ImageNet benchmark, RDBP matches or exceeds the plasticity and stability of state-of-the-art methods while reducing computational cost. RDBP thus provides both a practical solution for real-world continual learning and a clear benchmark against which future continual learning strategies can be measured.

Figures

Figures reproduced from arXiv: 2507.10637 by the authors.

Figure 1
Figure 1. Schematic illustration of the DBP algorithm [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Left: Current Task Performance of a CNN with different Activation Functions: ReLU, Tanh, ReLUDown, PAU and the Average Performance of a CNN fully reset at each task Right: Preactivation Distribution for Task 100, 1000, 3000, 5000 of the CNN with the ReLU(top) and ReLUDown(bottom) 4.2 RESULTS AND DISCUSSION In [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance of a CNN with a ReLU, RDBP, Continual Backpropagation and Generative Replay. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Continual ImageNet (Dohare et al., 2024) A.2 EXAMPLE IMAGES [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Example Images of the downsampled ImageNet Dataset. : Baby, Dog, Bridge, Bird, Oven [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Activation Function used in the Experiments: ReLU, Tanh, PAU, ReLUDown [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Gradient Factor for the different Layers of the [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Current Task Performance of a CNN to evaluate plasticity with different Activation Functions: ReLU, Tanh, [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Average Performance in the last 10 Tasks to evaluate the Stability of a CNN with different Activation [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Training time per task for different algorithms. [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 10 canonical work pages

  1. [1]

    128 × 4 × 4 ReLU Flatten 2048 – FC → µ, log σ2 2 × 128 – z + Label Embedding 130 – FC → 2048 2048 – Unflatten 128 × 4 × 4 – ConvT (128 →

    work page

  2. [4]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pp. 248–255. Ieee,

    work page 2009

  3. [8]

    New Insights for the Stability-Plasticity Dilemma in Online Continual Learning

    Dahuin Jung, Dongjin Lee, Sunwon Hong, Hyemi Jang, Ho Bae, and Sungroh Yoon. New insights for the stability- plasticity dilemma in online continual learning. arXiv preprint arXiv:2302.08741,

    work page Pith review arXiv

  4. [10]

    Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks

    Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jaegul Choo, and Clare Lyle. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. arXiv preprint arXiv:2406.02596 ,

    arXiv

  5. [11]

    How far can we go without convolution: Improving fully-connected networks

    Zhouhan Lin, Roland Memisevic, and Kishore Konda. How far can we go without convolution: Improving fully- connected networks. arXiv preprint arXiv:1511.02580,

    work page Pith review arXiv

  6. [12]

    Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney

    URL https://arxiv.org/abs/2410.07994. Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402.18762,

    arXiv

  7. [15]

    Lifelong learning with dynamically expandable networks

    Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547,

    arXiv

  8. [17]

    : Baby, Dog, Bridge, Bird, Oven A.3 A CTIVATION FUNCTIONS Figure 6 illustrates the activation functions used in our experiments

    A.2 E XAMPLE IMAGES Figure 5: Example Images of the downsampled ImageNet Dataset. : Baby, Dog, Bridge, Bird, Oven A.3 A CTIVATION FUNCTIONS Figure 6 illustrates the activation functions used in our experiments. The first is the standard Rectified Linear Unit, followed by the hyperbolic tangent function. The third is the initialization of the Pade Activati...

    work page 2025

Show all 19 references
  1. [100]

    6 Published at 4th Conference on Lifelong Learning Agents (CoLLAs), 2025 Figure 4: Continual ImageNet (Dohare et al.,

    We trained for 5 runs of 5000 tasks each. 6 Published at 4th Conference on Lifelong Learning Agents (CoLLAs), 2025 Figure 4: Continual ImageNet (Dohare et al.,

    2025

  2. [1999]

    Mitigating the stability-plasticity dilemma in adaptive train scheduling with curriculum-driven continual dqn expansion

    Achref Jaziri, Etienne K ¨unzel, and Visvanathan Ramesh. Mitigating the stability-plasticity dilemma in adaptive train scheduling with curriculum-driven continual dqn expansion. arXiv preprint arXiv:2408.09838,

    arXiv

  3. [2016]

    A study on the plasticity of neural networks

    Tudor Berariu, Wojciech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath. A study on the plasticity of neural networks. arXiv preprint arXiv:2106.00042,

    arXiv

  4. [2018]

    Fine-tuned language models are continual learners

    Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. arXiv preprint arXiv:2205.12393,

    arXiv

  5. [2019]

    Learning to learn without forgetting by maximizing transfer and minimizing interference

    Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910 ,

    arXiv

  6. [2021]

    Self-stabilization: The implicit bias of gradient descent at the edge of stability

    Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. arXiv preprint arXiv:2209.15594,

    arXiv

  7. [2022]

    Adaptive rational activations to boost deep reinforcement learning

    Quentin Delfosse, Patrick Schramowski, Martin Mundt, Alejandro Molina, and Kristian Kersting. Adaptive rational activations to boost deep reinforcement learning. arXiv preprint arXiv:2102.09407,

    arXiv

  8. [2023]

    Plastic and stable gated classifiers for continual learning

    5 Published at 4th Conference on Lifelong Learning Agents (CoLLAs), 2025 Nicholas I Kuo, Mehrtash Harandi, Nicolas Fourrier, Christian Walder, Gabriela Ferraro, Hanna Suominen, et al. Plastic and stable gated classifiers for continual learning. InProceedings of the IEEE/CVF Co...

    2025

  9. [2024]

    Addressing loss of plasticity and catastrophic forgetting in continual learning

    Mohamed Elsayed and A Rupam Mahmood. Addressing loss of plasticity and catastrophic forgetting in continual learning. arXiv preprint arXiv:2404.00781,

    arXiv

  10. [2025]

    Iv´an Izquierdo, Jorge H Medina, M ˆonica RM Vianna, Luciana A Izquierdo, and Daniela M Barros

    URL https://arxiv.org/abs/2504.09906. Iv´an Izquierdo, Jorge H Medina, M ˆonica RM Vianna, Luciana A Izquierdo, and Daniela M Barros. Separate mecha- nisms for short-and long-term memory. Behavioural Brain Research, 103(1):1–11,

  11. [5000]

    As a result, most activations are zero with zero gradients

    When using the ReLU activation function, the preactivation distribution exhibits a consistent shift toward negative values and becomes increasingly dispersed. As a result, most activations are zero with zero gradients. This worsens adaptation to new tasks and learning performa...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.