REVIEW 9 cited by
Continual Learning via Neural Pruning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce Continual Learning via Neural Pruning (CLNP), a new method aimed at lifelong learning in fixed capacity models based on neuronal model sparsification. In this method, subsequent tasks are trained using the inactive neurons and filters of the sparsified network and cause zero deterioration to the performance of previous tasks. In order to deal with the possible compromise between model sparsity and performance, we formalize and incorporate the concept of graceful forgetting: the idea that it is preferable to suffer a small amount of forgetting in a controlled manner if it helps regain network capacity and prevents uncontrolled loss of performance during the training of future tasks. CLNP also provides simple continual learning diagnostic tools in terms of the number of free neurons left for the training of future tasks as well as the number of neurons that are being reused. In particular, we see in experiments that CLNP verifies and automatically takes advantage of the fact that the features of earlier layers are more transferable. We show empirically that CLNP leads to significantly improved results over current weight elasticity based methods.
Forward citations
Cited by 9 Pith papers
-
Learning in Deep Networks under Dale's Constraint
An on-off two-channel network with fixed-sign synapses and local Hebbian learning is claimed to recover backpropagation exactly under symmetric weights and to beat comparable vanilla networks on Tiny ImageNet.
-
When Does Continual Learning Require Learning
Different patterns of environmental change (space vs time) require different LLM update behaviors; no single family of methods—prompts, distillation, RL, or compression—handles all regimes.
-
Learning without Isolation: Pathway Protection for Continual Learning
LwI fuses old and new models with graph matching, matching similar channels in shallow layers and dissimilar channels in deep layers, to reduce catastrophic forgetting without storing old data.
-
Continual Learning Beyond Experience Rehearsal and Full Model Surrogates
SPARC achieves strong continual learning accuracy with a fraction of the parameters of surrogate-based methods by combining task-specific depthwise filters with shared pointwise filters updated by exponential averaging.
-
Eidetic Learning: an Efficient and Provable Solution to Catastrophic Forgetting
Eidetic Learning freezes each task's important neurons and prunes connections from recycled neurons, making catastrophic forgetting impossible for the retained subnetwork.
-
Make Domain Shift a Catastrophic Forgetting Alleviator in Class-Incremental Learning
Domain shift reduces catastrophic forgetting in class-incremental learning, and the DisCo module transfers that benefit to ordinary benchmarks by enforcing task-separated features with contrastive losses.
-
Listen, Look, and Learn: Learning Without Forgetting through SAM-Audio
Integrates SAM-Audio dense representations with guided attention and dual distillation for audio-visual class-incremental learning, reporting consistent outperformance on benchmarks.
-
Understanding and Analyzing Model Robustness and Knowledge-Transfer in Multilingual Neural Machine Translation using TX-Ray
Sequential transfer with English-English pre-training yields a small BLEU gain for English-Spanish only, and neuron pruning consistently hurts low-resource NMT.
-
Adaptive Reorganization of Neural Pathways for Continual Learning with Spiking Neural Networks
SOR-SNN employs Self-Organizing Regulation networks to reorganize a single SNN into sparse pathways, achieving better performance, energy efficiency, memory use, backward transfer, and self-repair on continual learnin...
Discussion (0). Continue with ORCID to comment.