REVIEW 5 cited by
Towards Robust Evaluations of Continual Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Experiments used in current continual learning research do not faithfully assess fundamental challenges of learning continually. Instead of assessing performance on challenging and representative experiment designs, recent research has focused on increased dataset difficulty, while still using flawed experiment set-ups. We examine standard evaluations and show why these evaluations make some continual learning approaches look better than they are. We introduce desiderata for continual learning evaluations and explain why their absence creates misleading comparisons. Based on our desiderata we then propose new experiment designs which we demonstrate with various continual learning approaches and datasets. Our analysis calls for a reprioritization of research effort by the community.
Forward citations
Cited by 5 Pith papers
-
Online Continual Learning with Maximally Interfered Retrieval
Selecting replay samples by estimated loss increase after a virtual update improves online continual learning performance over random replay.
-
Fine-Tuning Regimes Define Distinct Continual Learning Problems
The relative rankings of continual learning methods are not preserved across different fine-tuning regimes defined by trainable parameter depth.
-
Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis
A fixed-feature CNN plus streaming linear discriminant analysis achieves strong continual classification performance with orders of magnitude less memory and compute than rehearsal-based incremental learners.
-
Toward Understanding Catastrophic Forgetting in Continual Learning
Error rates of continual learning algorithms on a task sequence correlate strongly with the sum of individual task complexities, but not with the dissimilarity between consecutive tasks.
-
Autoencoder-Based Hybrid Replay for Class-Incremental Learning
AHR combines an autoencoder with a repulsive-force class centroid layout to replay compressed latent exemplars, reporting higher accuracy than selected CIL baselines.
Discussion (0). Continue with ORCID to comment.