Pith. sign in

REVIEW 2 major objections 4 minor 52 references

Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning

T0 review · 2 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Replay-based continual learning splits its generalization gap at every layer into a finite-memory drift term and an optimization-dependence term, and derives concrete diagnostics from the split.

desk verdict A genuinely new layer-wise decomposition of replay generalization with real controlled validation, but the SGLD diagnostic branch is derived for a frozen-prefix regime the benchmarks do not run, and Corollary IV.2 overclaims independence at the input layer. read the letter →

arxiv 2608.11690 v1 pith:ZTJE7QHW submitted 2026-08-12 cs.LG stat.ML

Tieliang Gong , Zhongbo Zhang , Wen Wen , Yong-Jin Liu This is my paper · ORCID
classification cs.LGstat.ML
keywords continuallearningexperiencereplaycatastrophicforgettinginformation-theoreticgeneralizationboundsWassersteindistancestochasticgradientLangevindynamicslayer-wiseanalysisalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Replay-based continual learning stores a small buffer of past examples and mixes them into current training; the paper argues that the resulting generalization gap is governed by two coupled but separable effects that existing theory folds together. Finite memory replaces each past task distribution with an empirical proxy, producing a representation drift that is reshaped by every layer of the network, while repeated reuse couples the buffer, the current data, and the final hypothesis through a shared optimization trajectory. The paper's main theorem (Theorem IV.1) bounds the expected gap by a drift term plus an optimization-dependence term, and shows the latter decomposes into stability, plasticity, interaction, and residual coupling at every layer. If this decomposition is right, it would let practitioners identify which interior layer to stabilize and monitor forgetting online through gradient alignment, and the experiments report the predicted memory scaling, an interior funnel layer, and an alignment signal that tracks forgetting on standard replay benchmarks.

What carries the argument

The load-bearing machinery is a layer split of the network at depth $l\in\{0,\dots,L\}$. The paper defines three layer-$l$ laws: the population law $P_{A_l,Y|i,W^{1:l}}$, the empirical proxy $\hat P_{A_l,Y|S_i,W^{1:l}}$ on the stored buffer, and the replay centroid $Q_{A_l,Y|i,W^{1:l}} = \mathbb{E}_{S_i}[\hat P_{A_l,Y|S_i,W^{1:l}}\mid W^{1:l}]$, which averages over buffer construction while conditioning on the learned features. Comparing the centroid to the population isolates finite-memory drift; comparing the actual training sequence to a decoupled reference $\tilde Q$ (independent coordinates with the same marginals) isolates reuse coupling. The optimization dependence is then rewritten through the interaction-information identity as $S^{(l)}+P^{(l)}-R^{(l)} = I(U_{\mathrm{old}};U_{\mathrm{new}};W^{l+1:L}\mid W^{1:l})$, and the geometric relaxation replaces KL by Wasserstein-1 distance with a suffix Lipschitz factor $\bar\rho_l(W)=\rho_0(1\vee\prod_{h>l}\rho_h\|W_h\|_{\mathrm{op}})$, while the SGLD refinement bounds the incremental dependence by $\frac{1}{2}\log\det(I+(\eta^2/\tau^2)M)$ per step and reads it through the sensitivity metric $H=(I+\alpha V)^{-1}$.

What would settle it

In the paper's controlled Gaussian–linear setting, train two agents with identical current-task data but independently resampled equal-size buffers while keeping the feature prefix fixed; if the difference in their generalization gaps is not explained by the change in the estimated drift term $K^{(l)}$ (with the estimated dependence term $S^{(l)}+P^{(l)}-R^{(l)}+C^{(l)}$ held fixed), the layer-wise separation claimed in Theorem IV.1 would be falsified.

Watch

Extended reading notes

Core claim

The central discovery is Theorem IV.1: for a $\sigma$-subgaussian loss and any split layer $l$, the expected generalization gap is bounded by $(T-1)\sqrt{2\sigma^2 K^{(l)}} + \sqrt{2\sigma^2/N_{\mathrm{eff}}\,(S^{(l)}+P^{(l)}-R^{(l)}+C^{(l)})}$, where $N_{\mathrm{eff}} = 1/((T-1)/m + 1/n)$. Here $K^{(l)}$ is the average KL divergence between the layer-$l$ replay centroid and the population distribution of each old task, and $S^{(l)},P^{(l)},R^{(l)},C^{(l)}$ are layer-wise mutual-information terms measuring, respectively, how much the suffix parameters retain about replayed old representations, how much they encode the current task, the signed synergy/interference between old and new representations, and residual coupling of the training sequence. The drift term does not vanish as current-task size grows: it is the cost of compressing memory to $m$ exemplars per past task, and it survives even when the buffer contents are i.i.d. because the learned feature map is itself fitted to the stored samples. A Wasserstein relaxation of the drift gives a depth-dependent drift–sensitivity product whose minimizer the paper calls a generalization funnel layer, and an SGLD instantiation of the optimization term turns the bound into a trajectory-level log-determinant budget that separates gradient-covariance instability from a curvature-aware gradient-alignment interaction cost.

Load-bearing premise

The optimization-branch analysis fixes the bottom $l$ layers and analyzes only the suffix under SGLD, while the benchmark experiments update all parameters jointly; the claim that the log-determinant budget and alignment diagnostic remain valid under joint training is therefore an assumption, not a proven consequence.

Editorial extensions

If this is right

  • The variance term obeys an effective-sample-size scaling: as the number of current examples $n\to\infty$, the bound does not shrink to zero but settles at a finite-memory floor of order $\sqrt{(T-1)/m}$; controlled experiments confirm a fitted exponent near $-1/2$ for the gap versus buffer size $m$.
  • At the input layer the bound reduces to the standard mutual-information bound with effective sample size $N_{\mathrm{eff}}$, while at the output layer only drift and residual coupling remain; the dominant source of error therefore shifts from parameter-information complexity at shallow layers to representation mismatch at deep layers.
  • Minimizing the drift–sensitivity product selects an interior generalization funnel layer; the paper argues that feature distillation or partial freezing should target that interior basin rather than either end of the network, and reports that the interior minimizer appears in every tested depth cell and moves deeper as networks deepen.
  • For gradient-replay methods (experience replay and DER++), the curvature-aware alignment $\cos_H$ of old and new mean gradients negatively tracks pairwise forgetting, with benchmark partial correlations between $-0.65$ and $-0.95$, several times stronger than task-order baselines.
  • The same SGLD analysis yields a local replay/current mixing coefficient that minimizes the sensitivity-weighted mean-update energy; replay-heavy values are warnings of interference rather than a globally optimal replay policy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit: the same layer-wise decomposition offers a testable criterion for buffer design—buffer selection policies should be judged by how much they reduce $K^{(l)}$ and $C^{(l)}$ at the funnel layer, not just by end-task accuracy.
  • Because the theorems condition on a frozen prefix $W^{1:l}$ while the benchmark experiments train all parameters jointly, the operational force of the log-determinant budget in standard training is an extrapolation; comparing the alignment diagnostic under frozen versus jointly trained prefixes would delimit when the bound's optimization branch applies.
  • The signed interaction term $R^{(l)}$ suggests a curriculum prediction not tested in the paper: if old and new tasks are ordered so that their sensitivity-metric gradient alignment is positive, the accumulated dependence budget should be lower, so task orderings with high $\cos_H$ among adjacent tasks should forget less.
  • The funnel-layer and alignment quantities could in principle be combined into an online intervention—freeze or lightly regularize the layer with the smallest drift–sensitivity product and down-weight replay gradients that anti-align in the $H$ metric—but the paper only provides the diagnostics, not the combined controller.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper develops a layer-wise information-theoretic framework for replay-based continual learning. Theorem IV.1 decomposes the expected generalization gap into a replay-centroid drift term K^(l) and an optimization-dependence term that is further resolved into stability S^(l), plasticity P^(l), interaction R^(l), and residual coupling C^(l), with an effective sample size N_eff governing the variance scale. A Wasserstein relaxation (Theorem IV.4) replaces the KL-based drift by a drift--sensitivity product and predicts an interior "generalization funnel" layer. An SGLD instantiation (Theorem V.3) upper-bounds the optimization branch by a cumulative trajectory-level log-determinant budget, which is then expanded into instability and interaction costs and leads to a curvature-aware gradient-alignment statistic cos_H. Controlled Gaussian and MNIST experiments plus Split-CIFAR-100 and Split-TinyImageNet benchmarks test three predictions: the m^{-1/2} variance-floor scaling, the interior funnel location and its depth trend, and the negative correlation between the alignment diagnostic and task-wise forgetting.

Significance. If the main theorem holds, the paper is a genuine step beyond hypothesis-level information-theoretic bounds: it isolates a finite-memory representation drift from reuse-induced optimization dependence at every layer, and the two refinements yield falsifiable structural predictions. The manuscript is unusually honest: it reports the fitted m-exponent as -0.443 with CI [-0.498, -0.391] rather than claiming the idealized -1/2, it explicitly presents the funnel proxy as weak for DER++ and iCaRL, it includes iCaRL as a deliberate scope boundary, and it acknowledges that the certified Lipschitz product is too loose to localize a funnel in deep networks. The Appendix B proofs are detailed, and the core decomposition in Theorem IV.1 does not depend on the SGLD or frozen-prefix assumptions. The main weakness is that the SGLD-derived diagnostics are rigorously justified only under a fixed prefix and injected noise, while the experiments that carry the operational claims use joint training and, on the benchmarks, SGD without injected noise.

major comments (2)
  1. [Section V-A / Theorem V.3 / Corollary V.2] The SGLD analysis conditions on a fixed prefix W^{1:l} and updates only the suffix parameters; Corollary V.2's Markov chain U_T -> M_{1:T-1} -> Theta_{T-1}^R requires the old-task component of U_T to be a deterministic function of the buffer and the frozen prefix. The experiments in Section VI-A, however, train all parameters jointly: the benchmark pipelines update every layer with SGD, and even the controlled MNIST SGLD stream does not freeze a prefix. Consequently, the log-determinant budget of Theorem V.3 and the alignment statistic cos_H of Corollary V.7, which are presented as bound-derived diagnostics, are not theoretically justified for the dynamics actually executed. This is load-bearing for the paper's operational claim that the alignment diagnostic is a reading of the bound rather than a heuristic. I would ask the authors either to extend the analysis to joint prefix updates, or to explicitly reclassify the SGLD diagnostics as heuristics motivated by the frozen-prefix analysis and to validate them in a controlled experiment with a genuinely frozen prefix.
  2. [Section VI-A / Table I / Figure 6] The benchmark experiments use SGD with learning rate 0.03 and no injected Gaussian noise, so the SGLD signal-to-noise ratio alpha = eta^2/tau^2 entering the sensitivity metric H = (I + alpha V)^{-1} is undefined unless an ad hoc regularization is introduced. The paper does not state how alpha is set when computing cos_H for the SGD-trained ER, DER++, and iCaRL models in Table I, nor does it specify the diagonal approximation details beyond saying it is used. Since the central empirical claim -- that the alignment diagnostic tracks forgetting -- is made on these SGD benchmarks, the absence of a well-defined alpha and the lack of a theoretical bridge from SGLD to SGD leave the benchmark validation of the optimization branch incomplete. The paper should either provide the SGD analogues of Theorem V.3 or clearly report the heuristic choice of alpha and argue its insensitivity.
minor comments (4)
  1. [Section V / References] The citation [47] is given for stochastic gradient Langevin dynamics, but the listed reference is Chen et al., "On the stability-plasticity dilemma in continual meta-learning," which is not the standard SGLD source; please cite Welling and Teh (2011) or an equivalent canonical SGLD reference.
  2. [Figure 1 caption] The caption contains the fragment "MNIST-100×y-axis," which appears to be a formatting artifact or missing explanation; please clarify what the y-axis scaling refers to.
  3. [Appendix B, proof of Corollary IV.2] The proof states that the training sequence is block-wise i.i.d. and therefore C^(0)=0; this is correct, but the argument would be clearer if it explicitly noted that the random subset index set is independent of the i.i.d. task data, so without-replacement sampling does not create dependence at the input layer.
  4. [Section VI-C / Figure 4] The funnel study uses the on-distribution Jacobian norm as a lower bound for the suffix Lipschitz constant, while Theorem IV.4 and Corollary IV.5 concern the certified upper bound product; the paper acknowledges this in Section VII, but a sentence in Section VI-C stating that the empirical funnel is a surrogate-location result, not a certified one, would improve precision.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: the main Synergy–Drift bound is proved from sub-Gaussian and chain-rule identities, and the paper's self-citations are not used to force its predictions.

full rationale

Theorem IV.1 is derived in Appendix B.A from the exact decomposition L(W)-Δ(W)=Δ_drift+Δ_old,var+Δ_new,var, followed by Donsker-Varadhan (Lemma A.6), the interaction-information identity, and Cauchy–Schwarz/Jensen aggregation. The quantities K^(l), S^(l), P^(l), R^(l), and C^(l) are defined information measures, not fitted constants; no experimental gap value is inserted into the proof. The m^{-1/2} scaling, the plateau in n, and the Spearman 0.94 are tests of the N_eff prefactor on independently simulated closed-form data, and Section VI-B explicitly acknowledges that the fitted exponent −0.443 is a composite of the prefactor and the information content rather than a hidden fit. The funnel layer is defined in Corollary IV.5 as the minimizer of the bound's product; the experiments then estimate that same product and observe an interior minimizer and a depth trend, so the definition does not force the empirical outcome. The SGLD branch is proved from the chain rule and the entropy bound Lemma A.10; Lemma A.9 is cited to the authors' prior [51], but it is a standard parameter-free Gaussian maximum-entropy fact whose assumptions do not include the target result, so it counts as independent support and does not create load-bearing self-citation. References [36] and [38] are related-work positioning and are not used in the proofs. The frozen-prefix conditioning in Section V (fixing W^{1:l}) versus joint SGD training in Section VI is a genuine scope/validity gap for the operational diagnostics, but it is not circularity: no prediction is equal to an input by construction, and the main decomposition remains self-contained.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim introduces no fitted constants. The novel mathematical objects (replay centroid, decoupled reference, sensitivity metric, funnel layer) are constructions with empirical support, not postulated physical entities. The assumptions listed are the standard and domain-specific premises the proofs invoke.

assumptions (5)
  • domain assumption Loss is sigma-subgaussian for all weights and inputs
    Invoked in Theorem IV.1 and Theorem V.3 through Donsker-Varadhan; standard in information-theoretic generalization bounds, restricts unbounded losses.
  • domain assumption Replay buffer indices drawn uniformly without replacement from each task
    Defines the replay centroid and the input-layer equality of centroid and population; the without-replacement dependence is ignored when the corollary sets C^(0)=0.
  • domain assumption SGLD noise is isotropic Gaussian and independent across steps
    Needed for the differential entropy comparison in Theorem V.3 that yields the log-determinant budget.
  • domain assumption Within a task, the final suffix parameters are conditionally independent of past training data given the initialization and current task data
    Used in Corollary V.2 to telescope the heritage term; valid for SGLD with independent noise, not for optimizers with momentum or adaptive steps.
  • domain assumption Loss and activations are Lipschitz for the Wasserstein bound
    Required by Kantorovich-Rubinstein duality in Theorem IV.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning." pith.science (2026). https://pith.science/paper/ZTJE7QHW

@misc{pith2026260811690,
  author       = {Pith},
  title        = {Pith review of: Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZTJE7QHW}},
  note         = {Machine review of arXiv:2608.11690}
}
read the original abstract

Continual learning must absorb new tasks without erasing old ones, and replay---mixing a small buffer of past examples into current training---is among the most effective remedies for catastrophic forgetting. Yet its generalization behavior is shaped by two coupled effects that existing analyses fold into a single hypothesis-level quantity: finite memory replaces each past distribution with an empirical proxy, and repeated reuse couples the buffer, the current data, and the final hypothesis through a shared optimization trajectory. We develop a layer-wise information-theoretic framework that separates these effects at every depth. Our main result decomposes the expected generalization gap into a replay-induced representation drift and an optimization-dependence term, the latter further resolved into stability, plasticity, interaction, and residual-coupling components. Two refinements make the framework operational. A Wasserstein relaxation of the drift term, valid under support mismatch, yields a depth-dependent drift--sensitivity trade-off whose minimizer identifies which interior layer to stabilize. An SGLD instantiation of the optimization term reduces it to a trajectory-level log-determinant budget, exposing a curvature-aware gradient-alignment statistic that serves as an online diagnostic of task-wise forgetting. Controlled and benchmark experiments confirm the predicted memory scaling, the interior funnel, and the alignment signal's link to forgetting.

Figures

Figures reproduced from arXiv: 2608.11690 by the authors.

Figure 1
Figure 1. Decoupling Generalization Dynamics: Synergy vs. Interference on MNIST. Vertical dashed lines indicate task transitions. [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Scaling laws of the generalization gap for fixed [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. The bound is loose in constant but order-correct, and its rate prefactor separates cleanly from its information content. [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The drift–sensitivity trade-off and the empirical funnel. (a) On the controlled Gaussian setting with closed-form [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: The sensitivity-aware alignment cosH improves on the Euclidean gradient cosine cosE (the OGD/GEM quantity), both logged from identical gradients. (a) cosH separates synergy from interference more sharply (larger between-regime effect size). (b) cosH is a stronger predi…
Figure 6
Figure 6. Figure 6: Task-controlled partial Pearson correlation of [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 34 canonical work pages

  1. [1]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinskaet al., “Overcoming catastrophic forgetting in neural networks,”Proceedings of the national academy of sciences, vol. 114, no. 13, pp. 3521–3526, 2017

    2017

  2. [2]

    Continual lifelong learning with neural networks: A review,

    G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual lifelong learning with neural networks: A review,”Neural networks, vol. 113, pp. 54–71, 2019

    2019

  3. [3]

    A continual learning survey: Defying forgetting in classification tasks,

    M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars, “A continual learning survey: Defying forgetting in classification tasks,”IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 7, pp. 3366–3385, 2021

    2021

  4. [4]

    Catastrophic interference in connectionist networks: The sequential learning problem,

    M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” inPsychology of learning and motivation. Elsevier, 1989, vol. 24, pp. 109–165

    work page 1989

  5. [5]

    An empirical investigation of catastrophic forgetting in gradient-based neural networks,

    I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y . Bengio, “An empirical investigation of catastrophic forgetting in gradient-based neural networks,”ArXiv Preprint ArXiv:1312.6211, 2013

    arXiv 2013

  6. [6]

    Natural continual learning: success is a journey, not (just) a destination,

    T.-C. Kao, K. Jensen, G. van de Ven, A. Bernacchia, and G. Hennequin, “Natural continual learning: success is a journey, not (just) a destination,” Advances in Neural Information Processing Systems, vol. 34, pp. 28 067–28 079, 2021

    work page 2021

  7. [7]

    Adaptive plasticity improvement for continual learning,

    Y .-S. Liang and W.-J. Li, “Adaptive plasticity improvement for continual learning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 7816–7825

    work page 2023

  8. [8]

    Experience replay for continual learning,

    D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne, “Experience replay for continual learning,”Advances in neural information processing systems, vol. 32, 2019

    work page 2019

Show all 52 references
  1. [9]

    icarl: Incremental classifier and representation learning,

    S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” inProceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017, pp. 2001–2010

    2017

  2. [10]

    Online continual learning through mutual information maximization,

    Y . Guo, B. Liu, and D. Zhao, “Online continual learning through mutual information maximization,” inInternational Conference on Machine Learning. PMLR, 2022, pp. 8109–8126

    2022

  3. [11]

    Learning to learn without forgetting by maximizing transfer and minimizing interference,

    M. Riemer, I. Cases, R. Ajemian, M. Liu, I. Rish, Y . Tu, and G. Tesauro, “Learning to learn without forgetting by maximizing transfer and minimizing interference,” inInternational Conference on Learning Representations, 2019

    2019

  4. [12]

    A unified approach to domain incremental learning with memory: Theory and algorithm,

    H. Shi and H. Wang, “A unified approach to domain incremental learning with memory: Theory and algorithm,”Advances in Neural Information Processing Systems, vol. 36, pp. 15 027–15 059, 2023

    2023

  5. [13]

    Stability analysis for incremental adaptive dynamic programming with approximation errors,

    Y . Li and E.-J. van Kampen, “Stability analysis for incremental adaptive dynamic programming with approximation errors,”Journal of Aerospace Engineering, vol. 37, no. 1, p. 04023097, 2024. 17

    2024

  6. [14]

    Optimal continual learning has perfect memory and is np-hard,

    J. Knoblauch, H. Husain, and T. Diethe, “Optimal continual learning has perfect memory and is np-hard,” inInternational Conference on Machine Learning. PMLR, 2020, pp. 5327–5337

    2020

  7. [15]

    Information-theoretic analysis of generalization capability of learning algorithms,

    A. Xu and M. Raginsky, “Information-theoretic analysis of generalization capability of learning algorithms,”Advances in neural information processing systems, vol. 30, 2017

    2017

  8. [16]

    Reasoning about generalization via conditional mutual information,

    T. Steinke and L. Zakynthinou, “Reasoning about generalization via conditional mutual information,” inConference on Learning Theory. PMLR, 2020, pp. 3437–3452

    2020

  9. [17]

    Sharpened generalization bounds based on conditional mutual information and an application to noisy, iterative algorithms,

    M. Haghifam, J. Negrea, A. Khisti, D. M. Roy, and G. K. Dziugaite, “Sharpened generalization bounds based on conditional mutual information and an application to noisy, iterative algorithms,”Advances in Neural Information Processing Systems, vol. 33, pp. 9925–9935, 2020

    2020

  10. [18]

    Hierarchical generalization bounds for deep neural networks,

    H. He, C. L. Yu, and Z. Goldfeld, “Hierarchical generalization bounds for deep neural networks,” in2024 IEEE International Symposium on Information Theory (ISIT). IEEE, 2024, pp. 2688–2693

    2024

  11. [19]

    Catastrophic forgetting in connectionist networks,

    R. M. French, “Catastrophic forgetting in connectionist networks,”Trends in cognitive sciences, vol. 3, no. 4, pp. 128–135, 1999

    1999

  12. [20]

    Continual learning and catastrophic forgetting,

    G. M. van de Ven, N. Soures, and D. Kudithipudi, “Continual learning and catastrophic forgetting,”arXiv preprint arXiv:2403.05175, 2024

    2024 arXiv

  13. [21]

    A comprehensive survey of continual learning: Theory, method and application,

    L. Wang, X. Zhang, H. Su, and J. Zhu, “A comprehensive survey of continual learning: Theory, method and application,”IEEE transactions on pattern analysis and machine intelligence, vol. 46, no. 8, pp. 5362–5383, 2024

    2024

  14. [22]

    Continual learning through synaptic intelligence,

    F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,” inInternational conference on machine learning. PMLR, 2017, pp. 3987–3995

    2017

  15. [23]

    Memory aware synapses: Learning what (not) to forget,

    R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, “Memory aware synapses: Learning what (not) to forget,” inProceedings of the European conference on computer vision (ECCV), 2018, pp. 139–154

    2018

  16. [24]

    Progressive neural networks,

    A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, “Progressive neural networks,” arXiv preprint arXiv:1606.04671, 2016

    2016 arXiv

  17. [25]

    Packnet: Adding multiple tasks to a single network by iterative pruning,

    A. Mallya and S. Lazebnik, “Packnet: Adding multiple tasks to a single network by iterative pruning,” inProceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2018, pp. 7765–7773

    2018

  18. [26]

    Continual learning with deep generative replay,

    H. Shin, J. K. Lee, J. Kim, and J. Kim, “Continual learning with deep generative replay,”Advances in neural information processing systems, vol. 30, 2017

    2017

  19. [27]

    Online continual learning from imbalanced data,

    A. Chrysakis and M.-F. Moens, “Online continual learning from imbalanced data,” inInternational Conference on Machine Learning. PMLR, 2020, pp. 1952–1961

    2020

  20. [28]

    Information-theoretic online memory selection for continual learning,

    S. Sun, D. Calandriello, H. Hu, A. Li, and M. Titsias, “Information-theoretic online memory selection for continual learning,”arXiv preprint arXiv:2204.04763, 2022

    2022 arXiv

  21. [29]

    New insights on reducing abrupt representation change in online continual learning,

    L. Caccia, R. Aljundi, N. Asadi, T. Tuytelaars, J. Pineau, and E. Belilovsky, “New insights on reducing abrupt representation change in online continual learning,” inInternational Conference on Learning Representations, 2022

    2022

  22. [30]

    Gradient based sample selection for online continual learning,

    R. Aljundi, M. Lin, B. Goujaud, and Y . Bengio, “Gradient based sample selection for online continual learning,”Advances in neural information processing systems, vol. 32, 2019

    2019

  23. [31]

    Gradient episodic memory for continual learning,

    D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,”Advances in neural information processing systems, vol. 30, 2017

    2017

  24. [32]

    Efficient lifelong learning with a-gem,

    A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny, “Efficient lifelong learning with a-gem,”arXiv preprint arXiv:1812.00420, 2018

    2018 arXiv

  25. [33]

    Gradient projection memory for continual learning,

    G. Saha, I. Garg, and K. Roy, “Gradient projection memory for continual learning,”arXiv preprint arXiv:2103.09762, 2021

    2021 arXiv

  26. [34]

    Theory on forgetting and generalization of continual learning,

    S. Lin, P. Ju, Y . Liang, and N. Shroff, “Theory on forgetting and generalization of continual learning,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 21 078–21 100

    2023

  27. [35]

    Understanding forgetting in continual learning with linear regression,

    M. Ding, K. Ji, D. Wang, and J. Xu, “Understanding forgetting in continual learning with linear regression,”arXiv preprint arXiv:2405.17583, 2024

    2024 arXiv

  28. [36]

    Information-theoretic generalization bounds of replay-based continual learning,

    W. Wen, T. Gong, Z. Gao, Y . Zhang, W. Zhang, and Y .-J. Liu, “Information-theoretic generalization bounds of replay-based continual learning,” 2026. [Online]. Available: https://arxiv.org/abs/2507.12043

    2026

  29. [37]

    Information-theoretic generalization bounds for sgld via data-dependent estimates,

    J. Negrea, M. Haghifam, G. K. Dziugaite, A. Khisti, and D. M. Roy, “Information-theoretic generalization bounds for sgld via data-dependent estimates,” Advances in Neural Information Processing Systems, vol. 32, 2019

    2019

  30. [38]

    Understanding the generalization ability of deep learning algorithms: a kernelized r ´enyi’s entropy perspective,

    Y . Dong, T. Gong, H. Chen, and C. Li, “Understanding the generalization ability of deep learning algorithms: a kernelized r ´enyi’s entropy perspective,” arXiv preprint arXiv:2305.01143, 2023

    2023 arXiv

  31. [39]

    Why and when deep is better than shallow: An implementation-agnostic state-transition view of depth supremacy,

    S. Sonoda, Y . Hashimoto, I. Ishikawa, and M. Ikeda, “Why and when deep is better than shallow: An implementation-agnostic state-transition view of depth supremacy,” 2025. [Online]. Available: https://arxiv.org/abs/2505.15064

    2025 arXiv

  32. [40]

    Computational optimal transport: With applications to data science,

    G. Peyr ´e, M. Cuturiet al., “Computational optimal transport: With applications to data science,”Foundations and Trends® in Machine Learning, vol. 11, no. 5-6, pp. 355–607, 2019

    2019

  33. [41]

    Theoretical analysis of domain adaptation with optimal transport,

    I. Redko, A. Habrard, and M. Sebban, “Theoretical analysis of domain adaptation with optimal transport,” inJoint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 2017, pp. 737–753

    2017

  34. [42]

    Generalization error bounds using wasserstein distances,

    A. T. Lopez and V . Jog, “Generalization error bounds using wasserstein distances,” in2018 ieee information theory workshop (ITW). IEEE, 2018, pp. 1–5

    2018

  35. [43]

    An information-theoretic view of generalization via wasserstein distance,

    H. Wang, M. Diaz, J. C. S. Santos Filho, and F. P. Calmon, “An information-theoretic view of generalization via wasserstein distance,” in2019 IEEE international symposium on information theory (ISIT). IEEE, 2019, pp. 577–581

    2019

  36. [44]

    Podnet: Pooled outputs distillation for small-tasks incremental learning,

    A. Douillard, M. Cord, C. Ollion, T. Robert, and E. Valle, “Podnet: Pooled outputs distillation for small-tasks incremental learning,” inEuropean Conference on Computer Vision. Springer, 2020, pp. 86–102

    2020

  37. [45]

    Reducing catastrophic forgetting with associative learning: a lesson from fruit flies,

    Y . Shen, S. Dasgupta, and S. Navlakha, “Reducing catastrophic forgetting with associative learning: a lesson from fruit flies,”Neural Computation, vol. 35, no. 11, pp. 1797–1819, 2023

    2023

  38. [46]

    Selective freezing for efficient continual learning,

    A. Sorrenti, G. Bellitto, F. P. Salanitri, M. Pennisi, C. Spampinato, and S. Palazzo, “Selective freezing for efficient continual learning,” in2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). IEEE Computer Society, 2023, pp. 3542–3551

    2023

  39. [47]

    On the stability-plasticity dilemma in continual meta-learning: Theory and algorithm,

    Q. Chen, C. Shui, L. Han, and M. Marchand, “On the stability-plasticity dilemma in continual meta-learning: Theory and algorithm,”Advances in Neural Information Processing Systems, vol. 36, pp. 27 414–27 468, 2023

    2023

  40. [48]

    Dark experience for general continual learning: a strong, simple baseline,

    P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara, “Dark experience for general continual learning: a strong, simple baseline,”Advances in Neural Information Processing Systems, vol. 33, pp. 15 920–15 930, 2020

    2020

  41. [49]

    R. M. Gray,Entropy and information theory. Springer Science & Business Media, 2011

    2011

  42. [50]

    Information-theoretic generalization bounds for black-box learning algorithms,

    H. Harutyunyan, M. Raginsky, G. Ver Steeg, and A. Galstyan, “Information-theoretic generalization bounds for black-box learning algorithms,”Advances in Neural Information Processing Systems, vol. 34, pp. 24 670–24 682, 2021

    2021

  43. [51]

    Towards generalization beyond pointwise learning: A unified information-theoretic perspective,

    Y . Dong, T. Gong, H. Chen, Z. He, M. Li, S. Song, and C. Li, “Towards generalization beyond pointwise learning: A unified information-theoretic perspective,” inForty-first International Conference on Machine Learning, 2024. 18 APPENDIXA PREREQUISITEDEFINITIONS ANDLEMMAS Defin...

    2024

  44. [52]

    ¯ρl(W)· TX i=1 W1 ˆPAl,Y|S i,W1:l, PAl,Y|i,W 1:l # .(56) Since the bound (56) holds for everyl∈{0,...,L}, taking the minimum overlgives, genW ≤min l∈{0,...,L} E

    Ifα l(W)≥1, thenα l(W) 2∥a−a′∥2 2 +∥y−y ′∥2 2≤ αl(W) 2(∥a−a ′∥2 2 +∥y−y ′∥2 2). In both cases, q αl(W) 2∥a−a ′∥2 2 +∥y−y ′∥2 2≤(1∨α l(W)) q ∥a−a ′∥2 2 +∥y−y ′∥2 2 = (1∨α l(W))d (a,y),(a′,y′) . Therefore, |fW,l(a,y)−f W,l(a′,y′)|≤ρ 0(1∨α l(W))d (a,y),(a′,y′) = ¯ρl(W)d (a,y),(a′...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.