Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent
read the original abstract
In Continual Learning settings, deep neural networks are prone to Catastrophic Forgetting. Orthogonal Gradient Descent was proposed to tackle the challenge. However, no theoretical guarantees have been proven yet. We present a theoretical framework to study Continual Learning algorithms in the Neural Tangent Kernel regime. This framework comprises closed form expression of the model through tasks and proxies for Transfer Learning, generalisation and tasks similarity. In this framework, we prove that OGD is robust to Catastrophic Forgetting then derive the first generalisation bound for SGD and OGD for Continual Learning. Finally, we study the limits of this framework in practice for OGD and highlight the importance of the Neural Tangent Kernel variation for Continual Learning with OGD.
This paper has not been read by Pith yet.
Forward citations
Cited by 5 Pith papers
-
Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformers
Infinite-width transformers exhibit an inductive bias against high-complexity polynomial-time algorithms, with derived upper bounds on capturable tasks like sorting and string matching.
-
Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation
In the NTK regime, new-task training induces old-task prediction drift through the cross-task kernel, yielding an exact closed-form forgetting predictor under frozen linear heads and a low-rank concentration result.
-
Architecture-driven Shift: towards a lightweight selector for capturing the trends of logit shift
Proposes Architecture-driven Shift (ADS) as an architecture-based proxy for logit shift in continual learning, derived from spectral norm scaling, optimization path length and task conflict, with monotonic correlation...
-
KAN-CL: Per-Knot Importance Regularization for Continual Learning with Kolmogorov-Arnold Networks
KAN-CL cuts catastrophic forgetting by 88-93% on Split-CIFAR-10/5T and Split-CIFAR-100/10T by anchoring KAN parameters at per-knot granularity while matching baseline accuracy.
-
Characterizing and Correcting Effective Target Shift in Online Learning
Online kernel regression equals offline regression with shifted targets; correcting the targets lets online learning match offline performance and outperform true targets in continual image classification.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.