RECORD
Bilevel Continual Learning
Machine Learning · Submitted 30 Jul 2020
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Our extensive experiments on continual learning benchmarks demonstrate the efficacy of the proposed BCL compared to many state-of-the-art methods.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Continual learning aims to learn continuously from a stream of tasks and data in an online-learning fashion, being capable of exploiting what was learned previously to improve current and future tasks while still being able to perform well on the previous tasks. One common limitation of many existing continual learning methods is that they often train a model directly on all available training data without validation due to the nature of continual learning, thus suffering poor generalization at test time. In this work, we present a novel framework of continual learning named "Bilevel Continual Learning" (BCL) by unifying a {\it bilevel optimization} objective and a {\it dual memory management} strategy comprising both episodic memory and generalization memory to achieve effective knowledge transfer to future tasks and alleviate catastrophic forgetting on old tasks simultaneously. Our extensive experiments on continual learning benchmarks demonstrate the efficacy of the proposed BCL compared to many state-of-the-art methods. Our implementation is available at https://github.com/phquang/bilevel-continual-learning.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Continual learning aims to learn continuously from a stream of tasks and data in an online-learning fashion, being capable of exploiting what was learned previously to improve current and future tasks while still being able to perform well on the previous tasks. | background 1.00 | 0.04 | 0.03 | asserted 0.99 |
| s1 | One common limitation of many existing continual learning methods is that they often train a model directly on all available training data without validation due to the nature of continual learning, thus suffering poor generalization at test time. | background 0.60 | 0.06 | 0.03 | asserted 0.92 |
| s2 | In this work, we present a novel framework of continual learning named "Bilevel Continual Learning" (BCL) by unifying a {\it bilevel optimization} objective and a {\it dual memory management} strategy comprising both episodic memory and generalization memory to achieve effective knowledge transfer to future tasks and alleviate catastrophic forgetting on old tasks simultaneously. | contribution 0.84 | 0.88 | 0.21 | asserted 1.00 |
| s3 | Our extensive experiments on continual learning benchmarks demonstrate the efficacy of the proposed BCL compared to many state-of-the-art methods. | result 1.00 | 0.96 | 0.72 | asserted 1.00 |
| s4 | Our implementation is available at https://github.com/phquang/bilevel-continual-learning. | contribution 0.70 | 0.48 | 0.02 | asserted 0.97 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Continual learning aims to learn continuously from a stream of tasks and data in an online-learning fashion, being capable of exploiting what was learned previously to improve current and future tasks while still being able to perform well on the previous tasks. One common limitation of many existing continual learning methods is that they often train a model directly on all available training data without validation due to the nature of continual learning, thus suffering poor generalization at test time. In this work, we present a novel framework of continual learning named "Bilevel Continual Learning" (BCL) by unifying a {\it bilevel optimization} objective and a {\it dual memory management} strategy comprising both episodic memory and generalization memory to achieve effective knowledge transfer to future tasks and alleviate catastrophic forgetting on old tasks simultaneously. Our extensive experiments on continual learning benchmarks demonstrate the efficacy of the proposed BCL compared to many state-of-the-art methods. Our implementation is available at https://github.com/phquang/bilevel-continual-learning.
Human review
References
How the literature uses it
5 arXiv papers cite this in 12 sentences
Papers citing it · 8 sentences without a confident label
Uses its method or tool 2 papers
-
We follow the implementation details in the literatures [ref] , in all of our experiments.
Online Continual Learning via the Meta-learning Update with Multi-scale Knowledge Distillation and Data Augmentation · Implementation Details · 0.99
-
In particular, during the training process, we also use the cross-validation principle via bilevel optimization [ref] to cross-validate the parameter of our model and perform online continual learning [ref] .
Online Continual Learning via the Knowledge Invariant and Spread-out Properties · Experiment Setting · 1.00
Cites it as background 1 paper
-
Other works selectively save some examples to prevent forgetting [ref] .
Online Continual Learning via the Knowledge Invariant and Spread-out Properties · Introduction · 0.89
What it cites · 4 sentences without a confident label
Contests 1 paper
-
Despite the initial success of existing works [ref] , there is still a huge performance gap because they struggle to balance between knowledge transfer and preventing catastrophic forgetting.
On Tiny Episodic Memories in Continual Learning · Introduction · contest 0.77
Uses the method or tool of 2 papers
-
To alleviate the computational cost, we use the first-order approximation proposed in [ref] .
On First-Order Meta-Learning Algorithms · First-Order Approximation · 1.00
-
This can be achieved by employing a knowledge distillation regularizer [ref] on the inner objective as follows: [equation] where $τ$ denotes the softmax's temperature, which is usually set to be greater than 1.
Distilling the Knowledge in a Neural Network · Preventing The Inner Optimization Bias · 1.00
Compares with 1 paper
-
Particularly, we consider the following continual learning methods: LwF [ref] , EWC [ref] , ICARL [ref] , GEM [ref] , KDR [ref] , ER [ref] , MER [ref] , FTML [ref] .
On Tiny Episodic Memories in Continual Learning · Benchmarks and Baselines · 0.97
Cites as background 3 papers
-
Old data can be used as a constraint to optimize the model [ref] , representation learning [ref] , or just perform joint training with current data [ref] .
On Tiny Episodic Memories in Continual Learning · Related Work · 0.87
-
(3) Dynamic architecture approaches address catastrophic forgetting by having a separate network for each task and can grow its structure over time [ref] .
Progressive Neural Networks · Related Work · 0.98
-
(3) Dynamic architecture approaches address catastrophic forgetting by having a separate network for each task and can grow its structure over time [ref] .
PathNet: Evolution Channels Gradient Descent in Super Neural Networks · Related Work · 0.98
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.