Pith. sign in

REVIEW 4 cited by

A study on the plasticity of neural networks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2106.00042 v2 pith:XSVVBJ3S submitted 2021-05-31 cs.LG

Tudor Berariu , Wojciech Czarnecki , Soham De , Jorg Bornschein , Samuel Smith , Razvan Pascanu , Claudia Clopath This is my paper
classification cs.LG
keywords learningplasticitycontinualpretrainedpreviouslyreachsametask
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

One aim shared by multiple settings, such as continual learning or transfer learning, is to leverage previously acquired knowledge to converge faster on the current task. Usually this is done through fine-tuning, where an implicit assumption is that the network maintains its plasticity, meaning that the performance it can reach on any given task is not affected negatively by previously seen tasks. It has been observed recently that a pretrained model on data from the same distribution as the one it is fine-tuned on might not reach the same generalisation as a freshly initialised one. We build and extend this observation, providing a hypothesis for the mechanics behind it. We discuss the implication of losing plasticity for continual learning which heavily relies on optimising pretrained models.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Weight Decay Improves Language Model Plasticity

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Pretrained models trained with larger weight decay fine-tune better on downstream tasks, so the best pretraining checkpoint by loss is not always the best starting point for later training.

  2. The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks

    cs.LG 2025-06 conditional novelty 6.0 of 10

    In PPO, scaling data by adding parallel environments improves final performance and network stability more than scaling rollout length, across Atari, Procgen, and Isaac Gym.

  3. A Simple Baseline for Stable and Plastic Neural Networks

    cs.LG 2025-07 conditional novelty 4.0 of 10

    RDBP combines a new activation (ReLUDown) with decaying backpropagation to achieve strong stability and plasticity on Continual ImageNet with low overhead.

  4. Recovering Plasticity of Neural Networks via Soft Weight Rescaling

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Soft Weight Rescaling shrinks each layer's weights toward their initialization at every step, bounding weight norms and improving plasticity and test accuracy in continual, warm-start, and single-task learning.

Pith tools