Continual Learning by Asymmetric Loss Approximation with Single-Side Overestimation

Bohyung Han; Dongmin Park; Kyoung Mu Lee; Seokil Hong

REVIEW 3 major objections 6 minor 37 references

Continual Learning by Asymmetric Loss Approximation with Single-Side Overestimation

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ALASSO claims that overestimating the unobserved side of an asymmetric quadratic loss lets a fixed network learn many tasks in sequence with near-upper-bound accuracy.

desk verdict A promising SI variant with strong reported results, but the key derivation in Section 4.3 is internally inconsistent and the algorithm as written is under-specified. read the letter →

arxiv 1908.02984 v2 pith:WVHDQIAS submitted 2019-08-08 cs.LG stat.ML

Dongmin Park , Seokil Hong , Bohyung Han , Kyoung Mu Lee This is my paper · ORCID

classification cs.LGstat.ML

keywords continuallearningcatastrophicforgettingasymmetriclossapproximationquadraticsurrogatestructuralregularizationsynapticintelligencepermutedMNISToverestimation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes ALASSO, a continual learning regularizer that keeps a network from forgetting old tasks as new tasks arrive. It replaces the symmetric quadratic penalty used by earlier methods with an asymmetric quadratic surrogate: on the side of each parameter where the old loss was never observed, the penalty is deliberately overestimated. The paper also introduces a new formula for the quadratic coefficient, claimed to be exact whenever the surrogate is truly quadratic. If the claim holds, a single fixed network can learn dozens of tasks in sequence with accuracy close to what a model trained on all tasks at once would achieve.

What carries the argument

The central object is the asymmetric quadratic surrogate loss $L_s^n(\theta_k)$ with side-dependent curvature, gated by $\alpha(\theta_k)>0$ (observed side) versus $\alpha(\theta_k)\le 0$ (unobserved side, multiplied by $a>1$ plus $\epsilon$). The load-bearing identity is Eq. (7), which expresses the coefficient $\hat\Omega_k^n$ as the ratio of the sum of loss differences $\omega_k^n+\omega_k^{1:(n-1)}$ to the squared parameter displacement $(\hat\theta_k^n-\hat\theta_k^{n-1})^2$. This identity converts the asymmetry idea into a computable regularizer; it is derived from the assumption that the surrogate is quadratic and vanishes at the new optimum. A second component, parameter decoupling, uses distinct hyperparameters $a',c'$ when computing $\omega_k^{1:(n-1)}$ to avoid contradictory effects on the current-task gradient.

What would settle it

Construct a synthetic continual learning problem where the true per-parameter loss is known to be quadratic with a known coefficient, run ALASSO, and compare the coefficient recovered by Eq. (7) with the true value; any mismatch would falsify the exactness claim. Alternatively, on a real permuted MNIST run, measure $L_s^n(\hat\theta_k^n)$ for many parameters: if these values are systematically nonzero, then Eq. (7) is not the exact quadratic coefficient.

Watch

Extended reading notes

Core claim

The paper's central claim is that per-parameter loss surfaces in trained networks are asymmetric around their optima, so symmetric quadratic approximations systematically underestimate the danger of moving in the unobserved direction. ALASSO models the loss as $L_s^n(\theta_k)=\hat\Omega_k^n(\theta_k-\hat\theta_k^n)^2$ on the observed side but $(a\hat\Omega_k^n+\epsilon)(\theta_k-\hat\theta_k^n)^2$ on the unobserved side, where the side is decided by the sign of $\alpha(\theta_k)=(\theta_k-\hat\theta_k^n)(\hat\theta_k^{n-1}-\hat\theta_k^n)$. The quadratic coefficient is computed as $\hat\Omega_k^n = (\omega_k^n+\omega_k^{1:(n-1)})/(\hat\theta_k^n-\hat\theta_k^{n-1})^2$, summing the loss drops of the current and previous tasks; this is claimed to be the exact quadratic coefficient, unlike the fixed coefficient used in synaptic intelligence. On permuted MNIST, split CIFAR-10/100, and split Tiny ImageNet, ALASSO reports accuracy close to the single-task upper bound and substantially above existing regularizers.

Load-bearing premise

The derivation of the 'accurate' quadratic coefficient assumes that, for every parameter, the total surrogate loss is exactly quadratic and equals zero at the newly found optimum; if the true loss at that optimum is not zero, the formula is not exact and the computed importance can even have the wrong sign.

Editorial extensions

If this is right

A fixed architecture can learn 30 to 100 sequential tasks while retaining earlier tasks, so continual learning would not require growing networks or replay buffers.
Accuracy on permuted MNIST reaches about 0.944 at 30 tasks, within a few points of the multi-task upper bound, and strong retention is reported on split image benchmarks.
The quadratic-coefficient formula gives an online estimate of per-parameter importance that updates with each new optimum rather than staying fixed.
The method is a drop-in regularizer: only per-parameter scalars $\hat\Omega_k^n$, $\hat\theta_k^n$, and $\omega_k^n$ need to be stored.
The overestimation factor $a$ is reported to be robust over a wide range of values (2 to 5), suggesting the method does not require delicate tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the asymmetry is motivated by an empirical observation, one testable extension is to measure whether the optimal $a$ correlates with the actual curvature ratio of the true loss on the two sides; if it does, $a$ could be set per layer or per parameter rather than globally.
The exactness of Eq. (7) depends on the surrogate vanishing at the new optimum; in practice the true loss at the new optimum is generally nonzero, so the 'accurate' coefficient is an approximation that could be corrected by subtracting the measured residual $L_s^n(\hat\theta_k^n)$ when it is available.
The method's success on permuted MNIST suggests the same asymmetric surrogate idea could be applied to other parameter-space regularizers, such as Fisher-based penalties, by replacing symmetric quadratic penalties with side-dependent ones.
Parameter decoupling hints that the two roles of hyperparameters—gradient shaping versus importance estimation—should be tuned separately in any quadratic surrogate method, a lesson that may transfer to other continual learning algorithms.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Plainly: this is a paper worth reading but not in the form on the page. The idea—asymmetric quadratic surrogate with single-side overestimation and decoupled hyperparameters—is a real variation on SI, and the reported gains are large. The ablations show both components contribute, and the variance plots are a nice addition. But the central derivation in Section 4.3 doesn't hold up. The claim of an 'exact quadratic approximation' rests on assuming the surrogate L_s^n(theta_hat^n) = 0, which is definitional once you've declared the quadratic, but then Eq. (9) says that same quantity equals L^n(theta_hat^n) + c L_s^{n-1}(theta_hat^n). The only way both hold is if the total loss at the new optimum is zero, which essentially never happens with cross-entropy. The stress-test note is right: Eq. (7) can produce negative Omega^n_k even though Eq. (5) asserts Omega^n_k >= 0. The paper never says how non-negativity is enforced, and epsilon is never specified. So the published algorithm is under-specified: either the implementation clips or re-projects Omega somewhere, or it silently uses negative importance, which contradicts the asymmetric loss model.

The empirical claims are plausible but not fully verifiable. No code is released. The overestimation factor a is chosen on permuted MNIST (Table 1) and then applied to other datasets; that's a tuned hyperparameter, not a prediction. The Tiny ImageNet result slightly above single-task is a red flag, though not impossible.

What's genuinely valuable: the observed asymmetry in per-parameter losses (Figure 2) is a useful piece of evidence, and the ablation showing decoupling helps convergence is interesting. Continual learning researchers, especially those working on SI-style regularizers, will find the empirical comparison useful. If the authors can fix the derivation—probably by treating Eq. (7) as a heuristic and stating the clipping—and release code, the paper could be a solid contribution to the structural-regularization line.

For peer review: it deserves referee time, but only with the expectation of major revision. I would not cite it as a reliable method until the code and epsilon/clipping details are out.

Referee Report

3 major / 6 minor

Summary. The paper proposes ALASSO, a continual learning method based on an asymmetric quadratic surrogate loss with single-side overestimation. The algorithm overestimates the unobserved side of the loss for previous tasks, estimates the observed side via a claimed "exact quadratic approximation" in Eq. (7), and introduces a hyperparameter decoupling trick in Section 4.4. The authors report state-of-the-art results on permuted MNIST (30 and 100 tasks), split CIFAR-10/100, and split Tiny ImageNet, including accuracy close to upper bounds and substantially better forgetting than SI and EWC. Ablation studies are presented for the overestimation, the accurate approximation, and the decoupling components.

Significance. If the method is as effective as reported, this is a practically significant contribution: a simple structural regularizer that avoids additional memory, network components, and batch processing, and that dramatically improves over SI on several continual learning benchmarks. The empirical results, especially the 100-task permuted MNIST experiment and the stability analysis, are interesting and would be valuable to the community. The paper also contains a clear conceptual motivation based on empirically observed asymmetric loss surfaces. However, the theoretical derivation of the core coefficient equation is not sound as written, and the manuscript leaves unspecified several implementation details that are necessary to reproduce the reported numbers. The claim of an "exact quadratic approximation" is currently not justified, and the algorithm actually evaluated may differ from the one described by the equations.

major comments (3)

[Section 4.3, Eqs. (7)-(11)] The derivation of Eq. (7) relies on assumptions that are not stated in the main text and are generally false. Eq. (8) asserts L_s^n(theta_hat^n_k)=0, which holds by definition if L_s^n is the quadratic surrogate. Eq. (9) asserts that the same quantity equals L^n(theta_hat^n_k)+c L_s^{n-1}(theta_hat^n_k). For both to hold, the total loss at the new optimum must be zero. For cross-entropy classification losses, L^n(theta_hat^n_k) is positive and L_s^{n-1}(theta_hat^n_k) is generally positive after the parameter has moved, so the equality is not satisfied. In the supplement (Eq. A.14), the equality L_s^n(theta_k)=L^n(theta_k)+cL_s^{n-1}(theta_k) is assumed for all theta_k; under that assumption Eq. (7) is a restatement of the quadratic assumption rather than an independent derivation. The paper should either prove when this strong assumption holds or reframe Eq. (7) as a heuristic approximation rather than an exact one.
[Section 4.3, Eq. (7) with Eq. (11)] The coefficient Omega_hat^n_k computed by Eq. (7) can be negative, contradicting the requirement Omega_hat^n_k >= 0 stated in Eq. (5). Since omega^{1:(n-1)}_k = -c L_s^{n-1}(theta_hat^n_k) and L_s^{n-1} is nonnegative, this term is nonpositive; once the new optimum theta_hat^n_k has moved away from theta_hat^{n-1}_k, the term is strictly negative and can exceed omega^n_k in magnitude. The paper does not specify a clipping, absolute-value, or projection step, nor does it give the value of epsilon in Eq. (5). If the implementation silently enforces non-negativity, the evaluated algorithm differs from the published equations. If it does not, the asymmetric surrogate can have a negative coefficient, invalidating the claimed quadratic approximation. Please specify the exact update rule used in the experiments, including the treatment of non-positive Omega_hat^n_k.
[Section 4.4, Eq. (12)] The hyperparameter decoupling is not fully specified. The paper replaces (a,c) with (a',c') in the computation of omega^{1:(n-1)}_k, but the values of a' and c' are never reported in Section 5; only c=1.0 and the sensitivity of a are discussed. If a' and c' differ from a and c, ALASSO has two additional free hyperparameters, and the claim that the method is controlled by a single overestimation factor a is inaccurate. If a' and c' are equal to a and c in the experiments, that should be stated explicitly. As written, the decoupling is presented as conceptually necessary but its practical realization is left undefined, which makes the experimental evaluation difficult to reproduce.

minor comments (6)

[Section 5.2] The statement "We plan to release our source code and raw results for better reproducibility" is not a substitute for the experimental details needed now; please provide the exact handling of Omega_hat^n_k, the value of epsilon, and the values of a' and c' used for each benchmark.
[Section 5.5, Figure 9] The average validation accuracy of ALASSO on split Tiny ImageNet is reported as 59.4%, which is higher than the single-task upper bound of 58.9%; this is surprising and should be explained, for example by reporting variance across runs or clarifying the single-task training protocol.
[Section 4.3, Eq. (9)] The notation L_s^n is used ambiguously: Eq. (8) treats it as the quadratic surrogate, while Eq. (9) treats it as the sum of the current loss and the previous surrogate. The two roles should be distinguished notationally and their relationship stated clearly.
[Table 1] Table 1 reports results for a=0.8 on permuted MNIST, but Eq. (5) requires a>1; please clarify whether this row is included only to demonstrate degradation and whether a<1 is ever used in the actual algorithm.
[Section 5.3] The claim that ALASSO outperforms all compared methods by at least about 15 percentage points is not directly supported by a table of exact average accuracies for all methods; including such a table would improve the verification of the claimed margins.
[Section 2.3] There is a typo in the first paragraph: "categorize… and and their characteristics" should read "categorize… and their characteristics."

Circularity Check

1 steps flagged · score 4.0 of 10

Eq. (7) is the defining quadratic ansatz rearranged, so the claimed 'exact quadratic approximation' is self-definitional; the benchmark gains remain empirical evidence and are not forced by construction.

self definitional [Section 4.3, Eq. (7); supplementary Section H.2, Eq. (A.16)]
"we present a new derivation that leads to the exact quadratic approximation, which is given by Ω̂nk = Lns(θk)/(θk−θ̂nk)^2 = Lns(θ̂n−1k)/(θ̂n−1k−θ̂nk)^2 = ... = (ωnk + ω1:(n−1)k)/(θ̂nk − θ̂n−1k)^2."

The first equality of Eq. (7) is the literal rearrangement of the quadratic ansatz in Eq. (2)/(A.14): Lns(θk)=Ω̂nk(θk−θ̂nk)^2. Therefore Ω̂nk is not estimated from independent loss geometry; it is the coefficient forced by assuming the surrogate is quadratic and vanishes at θ̂nk. The claim that this yields 'the exact quadratic approximation' is a restatement of the assumption, so the derivation is self-definitional. Nothing in Eq. (7) tests the quadratic hypothesis; any predictive claim for the surrogate's shape is built in rather than derived.

full rationale

The only circularity I can exhibit by the paper's own equations is in the derivation of Eq. (7): the 'accurate quadratic approximation' coefficient Ω̂nk is obtained by dividing the defining quadratic surrogate Lns(θk)=Ω̂nk(θk−θ̂nk)^2 by (θk−θ̂nk)^2 and evaluating at θ̂n−1k. That is an algebraic identity of the ansatz, not an independent derivation, so this component is self-definitional rather than empirically inferred. The reported state-of-the-art accuracies on permuted MNIST, split CIFAR-10/100, and split Tiny ImageNet are empirical benchmarks; they do not reduce to Eq. (7) by construction, so the central performance claim retains independent content. I do not classify the hyperparameter a as a fitted-input-called-prediction: the paper transparently states it is chosen empirically on permuted MNIST and then fixed, which is ordinary hyperparameter selection, not a disguised fit. There is no load-bearing self-citation or imported uniqueness theorem; the cited SI/EWC/VCL works are external prior art. A separate correctness risk, noted for completeness rather than as circularity, is that Eq. (11) makes ω1:(n−1)k = −c Lns−1(θ̂nk) ≤ 0, which can make Eq. (7) negative despite Eq. (5)'s assumption Ω̂nk ≥ 0; the text never specifies clipping or projection, and the paper's Eq. (8) plus Eq. (9) would require Ln(θ̂nk)+cLns−1(θ̂nk)=0, which cross-entropy losses do not satisfy in general. These are internal-consistency threats to the 'exact' claim, but they are not reductions of a prediction to an input. Overall, the derivation-level circularity is real but localized, so the score is 4 rather than 6 or above.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

All free parameters, axioms, and entities are listed above. The main burden is the quadratic-and-zero assumption behind Eq. (7).

free parameters (3)

overestimation factor a = 2 to 4 (best for 100 tasks gives 0.79 accuracy for a=2 or 3; a=0.8 drops to 0.59)
Chosen empirically on permuted MNIST (Section 5.6, Table 1) and fixed to similar values for other datasets.
balance hyperparameter c = 1.0
Set to 1.0 throughout experiments (Section 5.6); conceptually coupled to a but decoupled as c and c' in Eq. (12).
epsilon in Eq. (5) = unspecified
Small positive constant to keep overestimation when Ω=0; no value is reported in the paper.

assumptions (3)

domain assumption True loss functions are asymmetric in parameter space, and overestimating the unobserved side is safe.
Motivated by Figure 2, which shows asymmetric loss curves for permuted MNIST, but no general proof is given.
ad hoc to paper The per-parameter surrogate loss L^n_s(θ_k) is exactly quadratic and equals zero at the current optimum θhat^n.
Used in the derivation of Eq. (7) in Section 4.3 and Supplementary H.2; combined with Eq. (9), this implies the total loss at the new optimum is zero, which is generally false.
ad hoc to paper The hyperparameters controlling gradient updates and importance accumulation can be decoupled into (a,c) and (a',c') without changing the conceptual objective.
Introduced in Section 4.4 to accelerate convergence; no theoretical justification beyond Figure 7.

invented entities (1)

Asymmetric quadratic surrogate loss L^n_s(θ_k,a)
purpose: Represents the loss of previous tasks in the regularizer, with overestimated unobserved side.
A new piecewise quadratic function introduced in Eq. (5); its validity is only demonstrated on benchmarks, with no external falsifiable handle.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Continual Learning by Asymmetric Loss Approximation with Single-Side Overestimation." pith.science (2026). https://pith.science/paper/WVHDQIAS

@misc{pith2026190802984,
  author       = {Pith},
  title        = {Pith review of: Continual Learning by Asymmetric Loss Approximation with Single-Side Overestimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WVHDQIAS}},
  note         = {Machine review of arXiv:1908.02984}
}

read the original abstract

Catastrophic forgetting is a critical challenge in training deep neural networks. Although continual learning has been investigated as a countermeasure to the problem, it often suffers from the requirements of additional network components and the limited scalability to a large number of tasks. We propose a novel approach to continual learning by approximating a true loss function using an asymmetric quadratic function with one of its sides overestimated. Our algorithm is motivated by the empirical observation that the network parameter updates affect the target loss functions asymmetrically. In the proposed continual learning framework, we estimate an asymmetric loss function for the tasks considered in the past through a proper overestimation of its unobserved sides in training new tasks, while deriving the accurate model parameter for the observable sides. In contrast to existing approaches, our method is free from the side effects and achieves the state-of-the-art accuracy that is even close to the upper-bound performance on several challenging benchmark datasets.

Figures

Figures reproduced from arXiv: 1908.02984 by the authors.

**Figure 2.** Illustration of asymmetric characteristics of a loss func [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Visualization of quadratic surrogate loss functions based [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Analysis of results from ALASSO in comparison to [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Test accuracy on the permuted MNIST dataset with 30 (left) and 100 (right) tasks. The results from several continual learning [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: The standard deviation of per-task test accuracy on the permuted MNIST dataset [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 8.** Figure 8: Per-task accuracy at every 5 task on the split CIFAR [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

**Figure 9.** Figure 9: Per-task accuracy of every 5 task on the split Tiny Im [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 32 canonical work pages

[1]

Available at tiny-imagenet.herokuapp.com

Tiny ImageNet Visual Recognition Challenge. Available at tiny-imagenet.herokuapp.com

work page
[2]

Memory aware synapses: Learning what (not) to forget

Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision, 2018

work page 2018
[3]

Expert gate: Lifelong learning with a network of experts

Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017

work page 2017
[4]

Weight uncertainty in neural network

Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, 2015

2015
[5]

Streaming varia- tional bayes

Tamara Broderick, Nicholas Boyd, Andre Wibisono, Ashia C Wilson, and Michael I Jordan. Streaming varia- tional bayes. In Advances in Neural Information Processing Systems, 2013

work page 2013
[6]

Riemannian walk for incremen- tal learning: Understanding forgetting and intransigence

Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajan- than, and Philip HS Torr. Riemannian walk for incremen- tal learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vi- sion, 2018

work page 2018
[7]

Pathnet: Evolution channels gradient descent in super neural networks

Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734, 2017

arXiv 2017
[8]

Catastrophic forgetting in connectionist networks

Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 1999

work page 1999

Show all 37 references

[9]

Online variational bayesian learning

Zoubin Ghahramani and H Attias. Online variational bayesian learning. In NIPS workshop on Online Learning , 2000

2000
[10]

An empirical investigation of catas- trophic forgetting in gradient-based neural networks

Ian Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catas- trophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013

2013 arXiv
[11]

Generative adversarial nets

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014

2014
[12]

Overcoming catastrophic inter- ference using conceptor-aided backpropagation

Xu He and Herbert Jaeger. Overcoming catastrophic inter- ference using conceptor-aided backpropagation. In Interna- tional Conference on Learning Representations, 2018

2018
[13]

Over- coming catastrophic forgetting via model adaptation

Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zheng- wei Tao, Jinwen Ma, Dongyan Zhao, and Rui Yan. Over- coming catastrophic forgetting via model adaptation. In In- ternational Conference on Learning Representations, 2019

2019
[14]

Less-forgetful learning for domain expansion in deep neu- ral networks

Heechul Jung, Jeongwoo Ju, Minju Jung, and Junmo Kim. Less-forgetful learning for domain expansion in deep neu- ral networks. In AAAI Conference on Artiﬁcial Intelligence, 2018

2018
[15]

Fearnet: Brain- inspired model for incremental learning

Ronald Kemker and Christopher Kanan. Fearnet: Brain- inspired model for incremental learning. In International Conference on Learning Representations, 2018

2018
[16]

Stochastic estimation of the maximum of a regression function

Jack Kiefer, Jacob Wolfowitz, et al. Stochastic estimation of the maximum of a regression function. The Annals of Mathematical Statistics, 1952

1952
[17]

Overcoming catastrophic forgetting in neu- ral networks

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neu- ral networks. Proceedings of the National Academy of Sc...

2017
[18]

The CIFAR-10 and CIFAR-100 datasets

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 and CIFAR-100 datasets. Available at www.cs.toronto.edu/ kriz/cifar.html
[19]

Yann LeCun, Cortes Cortes, and Christopher J.C. Burges. The MNIST database of handwritten digits. Available at yann.lecun.com/exdb/mnist/
[20]

Lifelong learning with dynamically expandable net- works

Jeongtae Lee, Jaehong Yun, Sungju Hwang, and Eunho Yang. Lifelong learning with dynamically expandable net- works. In International Conference on Learning Represen- tations, 2018

2018
[21]

Overcoming catastrophic forgetting by incremental moment matching

Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In Advances in Neural Information Processing Systems, 2017

2017
[22]

Dual-memory deep learning architectures for lifelong learning of everyday human behaviors

Sang-Woo Lee, Chung-Yeon Lee, Dong-Hyun Kwak, Jiwon Kim, Jeonghee Kim, and Byoung-Tak Zhang. Dual-memory deep learning architectures for lifelong learning of everyday human behaviors. In International Joint Conference on Arti- ﬁcial Intelligence, 2016

2016
[23]

Learning without forgetting

Zhizhong Li and Derek Hoiem. Learning without forgetting. In Proceedings of the European Conference on Computer Vi- sion. Springer, 2016

2016
[24]

Gradient episodic memory for contin- ual learning

David Lopez-Paz et al. Gradient episodic memory for contin- ual learning. In Advances in Neural Information Processing Systems, 2017

2017
[25]

Packnet: Adding mul- tiple tasks to a single network by iterative pruning

Arun Mallya and Svetlana Lazebnik. Packnet: Adding mul- tiple tasks to a single network by iterative pruning. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018

2018
[26]

Catastrophic inter- ference in connectionist networks: The sequential learning problem

Michael McCloskey and Neal J Cohen. Catastrophic inter- ference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation . Else- vier, 1989

1989
[27]

Variational continual learning

Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. InInternational Con- ference on Learning Representations, 2018

2018
[28]

icarl: Incremental classiﬁer and representation learning

Sylvestre-Alvise Rebufﬁ, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classiﬁer and representation learning. In Proceedings of the IEEE con- ference on Computer Vision and Pattern Recognition, 2017

2017
[29]

A stochastic approxima- tion method

Herbert Robbins and Sutton Monro. A stochastic approxima- tion method. The Annals of Mathematical Statistics, 1951

1951
[30]

Progressive neural networks

Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Raz- van Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016

2016 arXiv
[31]

Online model selection based on the varia- tional bayes

Masa-Aki Sato. Online model selection based on the varia- tional bayes. Neural computation, 2001

2001
[32]

Progress compress: A scalable framework for continual learning

Jonathan Schwarz, Jelena Luketina, Wojciech Czarnecki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pas- canu, and Raia Hadsell. Progress compress: A scalable framework for continual learning. In International Confer- ence on Machine Learning, 2018

2018
[33]

Overcoming catastrophic forgetting with hard attention to the task

Joan Serr `a, D ´ıdac Sur´ıs, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. InInternational Conference on Machine Learning, 2018

2018
[34]

Continual learning with deep generative replay

Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, 2017

2017
[35]

Memory-based parameter adaptation

Pablo Sprechmann, Siddhant M Jayakumar, Jack W Rae, Alexander Pritzel, Adri `a Puigdom `enech Badia, Benigno Uria, Oriol Vinyals, Demis Hassabis, Razvan Pascanu, and Charles Blundell. Memory-based parameter adaptation. In International Conference on Learning Representations , 2018

2018
[36]

Compete to com- pute

Rupesh K Srivastava, Jonathan Masci, Sohrob Kazerounian, Faustino Gomez, and J¨urgen Schmidhuber. Compete to com- pute. In Advances in Neural Information Processing Sys- tems, 2013

2013
[37]

Contin- ual learning through synaptic intelligence

Friedemann Zenke, Ben Poole, and Surya Ganguli. Contin- ual learning through synaptic intelligence. In International Conference on Machine Learning, 2017. SUPPLEMENTARY MATERIAL This document discusses intuitive interpretation ofLn(θk) in Section G. Section H presents the deri...

2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Available at tiny-imagenet.herokuapp.com

Tiny ImageNet Visual Recognition Challenge. Available at tiny-imagenet.herokuapp.com

work page

[2] [2]

Memory aware synapses: Learning what (not) to forget

Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision, 2018

work page 2018

[3] [3]

Expert gate: Lifelong learning with a network of experts

Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017

work page 2017

[4] [4]

Weight uncertainty in neural network

Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, 2015

2015

[5] [5]

Streaming varia- tional bayes

Tamara Broderick, Nicholas Boyd, Andre Wibisono, Ashia C Wilson, and Michael I Jordan. Streaming varia- tional bayes. In Advances in Neural Information Processing Systems, 2013

work page 2013

[6] [6]

Riemannian walk for incremen- tal learning: Understanding forgetting and intransigence

Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajan- than, and Philip HS Torr. Riemannian walk for incremen- tal learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vi- sion, 2018

work page 2018

[7] [7]

Pathnet: Evolution channels gradient descent in super neural networks

Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734, 2017

arXiv 2017

[8] [8]

Catastrophic forgetting in connectionist networks

Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 1999

work page 1999