Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that representation forgetting in continual learning follows a two-phase curve and that this curve, along with its dependence on layer depth and network width, follows from a new linear-alignment metric called…

desk verdict A genuinely new metric and a first theoretical handle on representation forgetting, but the main proof has a repairable gap and the two-phase time course rests on an unproven lower-bound assumption. read the letter →

arxiv 2505.20970 v3 pith:X3YHI43W submitted 2025-05-27 cs.LG

Joonkyu Kim , Yejin Kim , Jy-yong Sohn This is my paper
classification cs.LG
keywords representationforgettingcontinuallearningdiscrepancycatastrophiclineartransformationalignmentspacelayer-wiseprobing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that representation forgetting in continual learning—the degradation of a model's hidden-layer features for an earlier task as it learns later tasks—follows a two-phase pattern that can be derived, not just observed: an initial forgetting phase in which drift grows, then a saturation phase in which drift plateaus. It introduces the representation discrepancy, the minimum worst-case distance between the layer-$k$ representation spaces of two model snapshots after the best linear transformation aligns them, and argues this metric is a tractable surrogate for linear-probing forgetting because a small discrepancy preserves the power of any linear classifier. The central result is an upper bound $U^k_t(\Delta t)=\mu_t c_t \|R^k_t(h_t)\|\,f(\omega^{k-1}_t(\Delta t))$ with $f(\omega)=(\omega^2+\omega)/(\omega^2+1)$, whose shape predicts that deeper layers forget faster and wider networks forget slower. Experiments on Split-CIFAR100 and downsampled ImageNet1K show the predicted curve, a linear relation between forgetting and representation-space size, and the predicted dependence on layer index and width. This matters because representation-level forgetting is the relevant object for settings like unsupervised continual learning, where the hidden features are the deliverable.

What carries the argument

The load-bearing object is the representation discrepancy $D^k_t(h_t,\Delta t)=\min_T d(R^k_t(h_t),T(R^k_t(h_{t+\Delta t})))$—the minimum worst-case misalignment between two representation spaces after the best linear map $T$ is allowed to align the later snapshot to the earlier one. The argument then runs through a single-variable shape function $f(\omega)=(\omega^2+\omega)/(\omega^2+1)$: because the distance at layer $k-1$ enters only through the normalized ratio $\omega^{k-1}_t(\Delta t)$, the upper bound factorizes into a task- and layer-dependent scale $\mu_t c_t\|R^k_t(h_t)\|$ and this shape function. The derivation uses two auxiliary data-dependent constants, the layer cushion and activation contraction, to replace weight norms by activation norms, and an assumption that the weight matrices of the two snapshots are linearly equivalent. The further assumption that the layer-$(k-1)$ representation distance grows as $\Theta(\Delta t)$ is what turns $f$ into a rising-then-saturating curve and defines the saturation time $\Delta t_{\mathrm{sat}}$ used in the convergence-rate bound.

What would settle it

Measure $d(R^{k-1}_t(h_t),R^{k-1}_t(h_{t+\Delta t}))$ over many task sequences on a fixed architecture and check whether it scales as $c\,\Delta t$ with $c>0$ across layers; if the fitted growth exponent is not close to 1 over the relevant horizon, the lower half of Assumption 2 fails and the predicted $\Delta t_{\mathrm{sat}}$ positions and two-phase curve would not be expected to hold.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that the representation discrepancy $D^k_t(h_t,\Delta t)$ is both an effective surrogate for representation forgetting and analytically tractable. The effectiveness comes from the implication that if $D^k_t(h_t,\Delta t)$ is small, then for any linear classifier $C_1$ on the old features there is a linear classifier $C_2$ on the new features with $C_1\circ h^k_t \approx C_2\circ h^k_{t+\Delta t}$. The tractability comes from the bound $D^k_t(h_t,\Delta t)\le U^k_t(\Delta t)=\mu_t c_t \|R^k_t(h_t)\|\,(\omega^2+\omega)/(\omega^2+1)$ with $\omega=\omega^{k-1}_t(\Delta t)$, which rises monotonically and then saturates to $\mu_t c_t \|R^k_t(h_t)\|$. Under a weight-update scaling assumption, the convergence rate $r^k_t=1/\Delta t_{\mathrm{sat}}$ is bounded by an expression increasing in layer index $k$ and decreasing in width $m$; combined with a linear empirical relation between layer index and representation-space size, the paper concludes that asymptotic representation forgetting is larger in deeper layers and that wider networks slow forgetting.

Load-bearing premise

Assumption 2 is the load-bearing premise: the distance between the representation spaces at layer $k-1$ before and after $\Delta t$ more tasks grows linearly with $\Delta t$, and the paper states that only the linear upper bound is established in prior work, so the unproven linear lower bound is what turns the curve into a rise-then-plateau.

Editorial extensions

If this is right

  • For a fixed task and layer, representation forgetting should not grow without bound: the derived upper bound saturates at $\mu_t c_t\|R^k_t(h_t)\|$ after a finite number of additional tasks.
  • Deeper layers should enter the saturation phase sooner, so protecting or aligning higher-layer features is the key operation for retaining old-task representations.
  • Wider networks should delay saturation, giving a concrete mechanism by which width acts as a buffer against representation forgetting.
  • The amount of asymptotic forgetting should scale linearly with $\|R^k_t(h_t)\|$, so measuring feature norms across layers predicts which layers are most at risk.
  • The two-phase shape of the upper bound should be mirrored by linear-probing accuracy drops on the old tasks, which the paper observes on both Split-CIFAR100 and ImageNet1K.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct consequence the paper leaves implicit is that the two-phase prediction rests on the unproven lower half of Assumption 2; if layer-$(k-1)$ drift grows sublinearly with $\Delta t$, the plateau would arrive later or disappear within practical task horizons, so the growth exponent is the testable hinge.
  • Because the metric allows an arbitrary linear transformation, it measures whether the old feature space is recoverable by a linear map, not whether the features are identically preserved; a task's information could be destroyed nonlinearly while $D^k_t$ stays small.
  • Because only the ratio $\omega=d/\|R\|$ enters the bound, the theory suggests a scale-invariant reading of forgetting: larger-norm layers tolerate proportionally larger drift before the same forgetting level, which could be tested by normalizing feature norms across architectures.
  • If the empirical linear relation between layer index and representation-space size holds in other architectures, the convergence-rate bound implies a testable ordering of layer vulnerability that could guide where to schedule replay or regularization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a new metric, representation discrepancy D^k_t(h_t, Δt), defined as the minimum over linear transformations T of the worst-case distance between the task-t representation spaces of an L-layer ReLU network at layer k before and after learning Δt additional tasks. It derives an upper bound U^k_t(Δt) = μ_t c_t ||R^k_t(h_t)|| f(ω^{k-1}_t(Δt)) with f(x) = (x^2+x)/(x^2+1), under a linear-alignment assumption on weight matrices (Assumption 1), using the layer cushion and activation contraction from Arora et al. (2018). The paper then claims that, under an assumption that the representation-space distance grows linearly with Δt (Assumption 2), the representation forgetting curve has two phases (forgetting and saturation), that the asymptotic discrepancy is proportional to the size of the representation space (Corollary 1), and that deeper layers forget faster while wider networks forget more slowly (Theorem 2). Experiments on Split-CIFAR100 and ImageNet1K measure linear-probing accuracy drops, representation-space norms, and saturation times to support these claims.

Significance. If the results hold, this is a useful and much-needed first theoretical step for representation forgetting in continual learning. The proposed metric is interpretable, and the empirical correlation between D and linear-probing forgetting is strong (Fig. 8: R²=0.88 for Split-CIFAR100, R²=0.74 for ImageNet1K). The empirical support for Assumption 1 across both CNNs and a Vision Transformer (Fig. 11) is also a valuable contribution. However, the theoretical core is weaker than the presentation suggests: the asymptotic proportionality in Corollary 1 is definitional (Eq. 7), the temporal two-phase claim depends on an explicitly unproven linear lower bound (Assumption 2), and the proof of the main theorem has a repairable but real gap in Appendix A.1.2. The paper should be publishable after these issues are fixed and the claims are rephrased to match what is actually proved.

major comments (4)
  1. [Appendix A.1.2, proof of Theorem 1] The substitution in the derivation of Corollary 2 is incorrect as printed. The minimization in Eq. (12) is over T of c1 ||T W^k_{t+Δt}||_2 + c2 ||T W^k_{t+Δt} - W^k_t||_2 with c1 = d(R^{k-1}_t(h_t), R^{k-1}_t(h_{t+Δt})) and c2 = ||R^{k-1}_t(h_t)||; identifying A = W^k_{t+Δt} and X = T W^k_t changes the second term into T W^k_t - W^k_{t+Δt}, which does not match the expression being bounded. The correct identification is A = W^k_t and X = T W^k_{t+Δt}. Moreover, Lemma 2 minimizes over all X ∈ R^{n×n}, whereas the admissible X in Eq. (12) is restricted to the set {T W^k_{t+Δt} : T ∈ R^{w_k×w_k}}; the proof does not justify that the unconstrained minimizer X* = (c_2^2/(c_1^2+c_2^2)) A is feasible. A simple repair exists: Assumption 1 supplies T0 with T0 W^k_{t+Δt} = W^k_t, so taking T = (c_2^2/(c_1^2+c_2^2)) T0 achieves the claimed value. This gap should be fixed before Theorem 1 can be considered proved.
  2. [Sec. 5.2, Assumption 2] The temporal content of the paper rests on Assumption 2, which asserts d(R^{k-1}_t(h_t), R^{k-1}_t(h_{t+Δt})) = Θ(Δt). As the paper itself notes, the lower bound is not established in the literature; only the upper bound follows from Theorem 4.1 of Guha & Lakshman (2024). This is load-bearing: without the lower bound, ω^{k-1}_t(Δt) need not reach the critical value 1+√2 at any finite Δt, so the two-phase curve in Proposition 1, the definition of Δt_sat in Eq. (9), and the proof of Theorem 2 (which uses ω^k_t(Δt_sat)=1+√2) lose their justification. A direct empirical measurement of d(R^{k-1}_t(h_t), R^{k-1}_t(h_{t+Δt})) versus Δt, or a proof under Assumptions 1 and 3, is needed; currently the central qualitative claims are conditional on an unverified assumption.
  3. [Sec. 5.2, Proposition 1] Proposition 1 contains two mathematical errors. First, f(x) = (x^2+x)/(x^2+1) is not monotone in x: it increases on [0, 1+√2), attains its maximum at x=1+√2, and then decreases toward 1 as x→∞. The statement that U^k_t(Δt) 'monotonically increases with respect to Δt' is therefore false; the correct description is an increase followed by an overshoot and a gradual decline to the asymptotic value. Second, the claimed peak value (1+√2/4)μ_t c_t ||R^k_t(h_t)|| is arithmetically wrong: f(1+√2) = (1+√2)/2, so the peak is ((1+√2)/2) μ_t c_t ||R^k_t(h_t)|| ≈ 1.207 μ_t c_t ||R^k_t(h_t)||, not 1.354. These errors do not destroy the two-phase qualitative picture, but they must be corrected.
  4. [Appendix A.2, proof of Theorem 4] The derivation of the cumulative weight-drift bound in Theorem 4 omits a factor of λ_t. From Assumption 3, ||W^k_{t+1} - W^k_t||_F / ||W^k_t||_2 ≤ γ m^{-β}; summing over Δt steps yields ||W^k_t - W^k_{t+Δt}||_2 / ||W^k_t||_2 ≤ γ m^{-β} Σ_{i=1}^{Δt} ||W^k_{t+i-1}||_2 / ||W^k_t||_2, which is only ≤ λ_t γ m^{-β} Δt, not γ m^{-β} Δt as written in Eq. (22). Inserting the missing λ_t changes the final bound in Eq. (23) from Σ_{i=1}^k (λ_t μ_t c_t)^i / λ_t to Σ_{i=1}^k (λ_t μ_t c_t)^i. The qualitative conclusions (increase in k, decrease in m) are unaffected, but the stated bound is not proved as written.
minor comments (5)
  1. [Sec. 4.2] The argument that small D implies small representation forgetting is valid only in one direction: it shows that for any linear classifier on h_t, a corresponding classifier on h_{t+Δt} approximately matches its performance. It does not establish the converse, so the paper should state that the surrogate claim is supported empirically (Fig. 8) rather than derived.
  2. [Sec. 6.3 and Appendix C.3] The saturation time Δt_sat is estimated as the first local maximum of a 4th-degree polynomial fitted to the measured curve; this heuristic should be described as a finite-horizon proxy, and its sensitivity to the polynomial degree should be discussed at least briefly.
  3. [Appendix A.1.1, Lemma 2] In the proof of Lemma 2, the minimizer X* is computed for the squared Frobenius norm objective, but the original objective uses the spectral norm; the inequality direction is correct, but the text should state explicitly that evaluating the original objective at the Frobenius minimizer yields an upper bound to avoid confusion. There is also a notation typo: 'X−T' should be 'X−A'.
  4. [Appendix A.2, proof of Theorem 4] The sentence 'Equation (21) comes from Assumption 1 and the definition of λ_t' is inaccurate: the bound on ||W^k_t - W^k_{t+Δt}|| comes from Assumption 3, not Assumption 1. Also, the first term in the ratio inside the displayed derivation should be ||h^{k-1}_t(x) - h^{k-1}_{t+Δt}(x)||, with the subscript t on the first term.
  5. [Sec. 1 and Abstract] The claim of being the 'first theoretical analysis of representation forgetting' should be qualified, since prior work such as Davari et al. (2022) and Ramasesh et al. (2020) provides empirical analyses and the paper's novelty is better stated as the first analysis of a linear-alignment-based discrepancy metric under the stated assumptions.

Circularity Check

1 steps flagged · score 4.0 of 10

One definitional identity (U∞ ∝ ||R||) is presented as a derived finding, but the surrogate claim and rate/depth/width theorems retain independent empirical and conditional content.

  1. self definitional [Def. 7, Eq. (7), and Corollary 1 (Sec. 5.2)]
    "U^k_{t,∞} := lim_{Δt→∞} U^k_t(Δt) = μ_t c_t ∥R^k_t(h_t)∥, and Corollary 1: 'the k-dependency of the asymptotic representation discrepancy U^k_{t,∞} is fully captured by ∥R^k_t(h_t)∥ ... U^k_{t,∞} is linearly proportional to ∥R^k_t(h_t)∥.'"

    By Def. 7, U^k_t(Δt) = μ_t c_t ∥R^k_t(h_t)∥ · (ω²+ω)/(ω²+1), and (ω²+ω)/(ω²+1) → 1 as Δt→∞ with ω defined as a distance ratio. Hence the 'asymptotic representation discrepancy' is, by definition, exactly the prefactor μ_t c_t ∥R^k_t(h_t)∥. Corollary 1 merely unpacks this definition; the claimed proportionality contains no dynamical measurement and is true by construction. The paper then converts the empirical linearity of ∥R∥ in k (Fig. 5b) plus this identity into the conclusion that forgetting grows with layer depth, so the corollary is load-bearing for that qualitative claim even though the surrogate correlation (Fig. 8) is an independent empirical check.

full rationale

The derivation chain is mostly conditional and externally sourced rather than circular: Theorem 1 holds under the explicit Assumption 1 and uses cushion/contraction constants from Arora et al.; the rate result (Theorem 2) follows from Assumption 3 imported from Guha and Lakshman (2024); there are no author self-citations carrying the argument. The main genuine circularity is the asymptotic proportionality U_{t,∞} ∝ ∥R^k_t(h_t)∥, which is an identity inherited from the definition of U rather than a derived prediction. Separately, Sec. 5.2's Assumption 2 asserts d(R^{k-1}_t(h_t), R^{k-1}_t(h_{t+Δt})) = Θ(Δt) and explicitly notes that the lower bound 'has not yet been established in the literature'; the two-phase shape and Δt_sat definitions depend on this linear growth, so the time-course claims are conditional on an unproven (though transparently acknowledged) premise. That is a correctness/robustness gap, not circularity, since it is stated as an assumption rather than smuggled in as a conclusion. Empirical validation of the surrogate against ΔP (R²=0.88 and 0.74) and of the depth/width trends is performed on real datasets, giving independent content; hence partial, not total, circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The theoretical framework uses no fitted constants: µ_t, c_t, λ_t, γ, and β are defined or borrowed from prior work and are not adjusted to match experiments. The paper introduces no new physical or mechanistic entities; the representation discrepancy is a mathematical definition, not an entity requiring independent falsifiable evidence. The main load-bearing assumptions are Assumptions 2 and 3, both explicitly stated, with Assumption 2 acknowledged as partially unproven.

assumptions (5)
  • ad hoc to paper For each layer k and task indices t < t', there exists a linear transformation T such that T W^k_{t'} = W^k_t (Assumption 1, Sec. 5.1).
    Stated as needed for the analysis and empirically supported in Fig. 2 and Fig. 11. It is strong as an exact equality, and its role in the derivations is not clearly demonstrated; Appendix A.2 attributes a step to it that follows from norm inequalities instead.
  • ad hoc to paper The representation distance d(R^{k-1}_t(h_t), R^{k-1}_t(h_{t+Δt})) grows linearly with Δt, i.e., Θ(Δt) (Assumption 2, Sec. 5.2).
    The authors explicitly note that the linear lower bound is not established in the literature. This assumption is load-bearing for the two-phase saturation structure and for the convergence-rate bound in Thm 2.
  • domain assumption Per-layer weight updates satisfy ||W^k_{s+1} - W^k_s||_F / ||W^k_s||_2 ≤ γ m^{-β} (Assumption 3, Sec. 5.2).
    Borrowed from Assumption 4.3 of Guha & Lakshman (2024), where it is empirically justified. Used in the proof of Thm 2 to bound accumulated weight drift over Δt tasks.
  • domain assumption The layer cushion µ_t and activation contraction c_t (Defs 5 and 6, from Arora et al. 2018) are finite and well-defined for the given model and dataset.
    Standard from the compression-based generalization bound literature; the existence of finite µ and c is assumed without additional justification.
  • standard math ReLU is 1-Lipschitz and matrix norms satisfy sub-multiplicativity.
    Used repeatedly in the proof chain in Appendix A to bound activation differences and operator norms.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective." pith.science (2026). https://pith.science/paper/X3YHI43W

@misc{pith2026250520970,
  author       = {Pith},
  title        = {Pith review of: Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X3YHI43W}},
  note         = {Machine review of arXiv:2505.20970}
}
read the original abstract

In continual learning scenarios, catastrophic forgetting of previously learned tasks is a critical issue, making it essential to effectively measure such forgetting. Recently, there has been growing interest in focusing on representation forgetting, the forgetting measured at the hidden layer. In this paper, we provide the first theoretical analysis of representation forgetting and use this analysis to better understand the behavior of continual learning. First, we introduce a new metric called representation discrepancy, which measures the difference between representation spaces constructed by two snapshots of a model trained through continual learning. We demonstrate that our proposed metric serves as an effective surrogate for the representation forgetting while remaining analytically tractable. Second, through mathematical analysis of our metric, we derive several key findings about the dynamics of representation forgetting: the forgetting occurs more rapidly to a higher degree as the layer index increases, while increasing the width of the network slows down the forgetting process. Third, we support our theoretical findings through experiments on real image datasets, including Split-CIFAR100 and ImageNet1K.

Figures

Figures reproduced from arXiv: 2505.20970 by the authors.

Figure 1
Figure 1. Visual interpretation of the representation discrepancy D k t (ht, ∆t) = min T d(Rk t (ht), T (Rk t (ht+∆t))) defined by us in Def. 4. Here, the linear transformation T is optimized to re￾duce the distance between the representation space Rk t (ht) = {h k t (x)}x∈Xt and Rk t (ht+∆t) = {h k t+∆t (x)}x∈Xt . We argue that Dk t (ht, ∆t) serves as an effective surrogate for the representation forgetting, following the lo… view at source ↗
Figure 3
Figure 3. Illustration of Proposition 1. The graph of U k t (∆t) (shown as black solid line) can be divided into two regions: (1) the forgetting region (∆t < ∆tsat), where U k t (∆t) consis￾tently increases as a function of ∆t and (2) the saturation region (∆t ≥ ∆tsat), where U k t (∆t) saturates to the asymptotic value U k t,∞. Note that U k t (∆t) can be factored out into two terms: the first term µtct∥Rk t (ht)∥ that does … view at source ↗
Figure 4
Figure 4. The evolution of representation forgetting ∆P k t (∆t) measured on Split-CIFAR100 and ImageNet1K datasets. We mea￾sure ∆P k t (∆t) for t = 1 and k = L. The solid line shows ∆P k t (∆t), while the red dashed line shows the 4-th degree poly￾nomial that best fits to the solid line, to check the overall tendency of the representation forgetting. Similar to the plot in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (12 more)
Figure 6
Figure 6. Figure 6: The impact of # channels (Left) and k (Right) on ∆tsat. For each k and # channels, we report ∆t when the best fitted 4-th degree polynomial achieves its first local maximum, similar to the point indicated by the black dashed line in [PITH_FULL_IMAGE:figures/full_fig_p…
Figure 7
Figure 7. Figure 7: ResNet model architecture illustration. We illustrate the ResNet model used in this paper [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Correlation between representation discrepancy D k t (ht, ∆t) and representation forgetting ∆P k t (∆t) on (a) Split-CIFAR100 and (b) ImageNet1K. The strong linear trends (with R 2 values of 0.88 and 0.74 respectively) demonstrate that the proposed representation discr…
Figure 9
Figure 9. Figure 9: Empirical comparison between the theoretical upper bound U k t and the measured representation discrepancy D k t across ReLU networks of width 64, 128, and 256 on Split-CIFAR100 [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Discrepancy between the task-1 model and the final model (∆t = N−1) remains far below that of a randomly initialized model (∆t = −1), indicating that the model retains nontrivial task-1 information and has not undergone maximal forgetting. B.4. Additional Supplementar…
Figure 11
Figure 11. Figure 11: Difference between Wk t and TWk t ′ across epochs for each transformer layer k in a Vision Transformer (ViT) trained on Split-CIFAR100. The results show rapid convergence of the linear transformation T , validating Assumption 1 in the transformer setting. B.6. Analyzi…
Figure 12
Figure 12. Figure 12: ∥Rk t (ht)∥ explains the representation forgetting ∆P k t (∆t). Each point represents each layer from the ResNet. In Fig 12a, the red line shows the linear regression fit between ∥Rk t (ht)∥ and ∆P k t (∆t), demonstrating a strong correlation with R 2 = 0.84. In Fig. …
Figure 13
Figure 13. Figure 13: The impact of k on the convergence of representation discrepancy [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: The impact of m on the convergence of representation discrepancy. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Evolution of the empirically measured upper bound U k t (∆t) on the representation discrepancy for t = 1 and k = L, evaluated on (a) Split-CIFAR100 and (b) ImageNet1K. The solid blue line shows U k t (∆t), and the red dashed line indicates the 4-th degree polynomial f…
Figure 16
Figure 16. Figure 16: Relationship between the U k t and ∥Rk t (ht)∥, across layers k = 1, . . . , 9, for Split-CIFAR100 (left) and ImageNet1K (right). Each point corresponds to a specific layer. The fitted red lines and high R 2 values (0.97 and 0.99) confirm a strong linear relationship,…
Figure 17
Figure 17. Figure 17: Saturation time ∆tsat measured using U k t for different layer indices k. Following the same methodology as in [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Forgetting is Everywhere

    cs.LG 2025-11 conditional novelty 6.0 of 10

    Forgetting is defined as violation of predictive self-consistency under self-generated updates, yielding the measure Γ_k(t); exact Bayesian learners are shown to have Γ = 0.

Reference graph

Works this paper leans on

48 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    and Yasaei Sekeh, S

    Andle, J. and Yasaei Sekeh, S. Theoretical understanding of the information flow on continual learning performance. In European Conference on Computer Vision, pp.\ 86--101. Springer, 2022

    work page 2022

  3. [3]

    Stronger generalization bounds for deep nets via a compression approach

    Arora, S., Ge, R., Neyshabur, B., and Zhang, Y. Stronger generalization bounds for deep nets via a compression approach. In International conference on machine learning, pp.\ 254--263. PMLR, 2018

    work page 2018

  4. [4]

    Statistical mechanical analysis of catastrophic forgetting in continual learning with teacher and student networks

    Asanuma, H., Takagi, S., Nagano, Y., Yoshida, Y., Igarashi, Y., and Okada, M. Statistical mechanical analysis of catastrophic forgetting in continual learning with teacher and student networks. Journal of the Physical Society of Japan, 90 0 (10): 0 104001, 2021

    work page 2021

  5. [5]

    A., Doan, T., and Sugiyama, M

    Bennani, M. A., Doan, T., and Sugiyama, M. Generalisation guarantees for continual learning with orthogonal gradient descent. arXiv preprint arXiv:2006.11942, 2020

    arXiv 2006

  6. [6]

    New insights on reducing abrupt representation change in online continual learning

    Caccia, L., Aljundi, R., Asadi, N., Tuytelaars, T., Pineau, J., and Belilovsky, E. New insights on reducing abrupt representation change in online continual learning. arXiv preprint arXiv:2104.05025, 2021

    arXiv 2021

  7. [7]

    Provable lifelong learning of representations

    Cao, X., Liu, W., and Vempala, S. Provable lifelong learning of representations. In International Conference on Artificial Intelligence and Statistics, pp.\ 6334--6356. PMLR, 2022

    work page 2022

  8. [8]

    Memory bounds for continual learning

    Chen, X., Papadimitriou, C., and Peng, B. Memory bounds for continual learning. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS), pp.\ 519--530. IEEE, 2022

    work page 2022

Show all 48 references
  1. [9]

    and Liu, B

    Chen, Z. and Liu, B. Lifelong machine learning. Morgan & Claypool Publishers, 2018

    2018

  2. [10]

    A downsampled variant of imagenet as an alternative to the cifar datasets

    Chrabaszcz, P., Loshchilov, I., and Hutter, F. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819, 2017

    2017 arXiv

  3. [11]

    Probing representation forgetting in supervised and unsupervised continual learning

    Davari, M., Asadi, N., Mudur, S., Aljundi, R., and Belilovsky, E. Probing representation forgetting in supervised and unsupervised continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 16712--16721, 2022

    2022

  4. [12]

    Understanding forgetting in continual learning with linear regression

    Ding, M., Ji, K., Wang, D., and Xu, J. Understanding forgetting in continual learning with linear regression. arXiv preprint arXiv:2405.17583, 2024

    2024 arXiv

  5. [13]

    A., Mazoure, B., Rabusseau, G., and Alquier, P

    Doan, T., Bennani, M. A., Mazoure, B., Rabusseau, G., and Alquier, P. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In International Conference on Artificial Intelligence and Statistics, pp.\ 1072--1080. PMLR, 2021

    2021

  6. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  7. [15]

    How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory, pp.\ 4028--4079

    Evron, I., Moroshko, E., Ward, R., Srebro, N., and Soudry, D. How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory, pp.\ 4028--4079. PMLR, 2022

    2022

  8. [16]

    Fini, E., Da Costa, V. G. T., Alameda-Pineda, X., Ricci, E., Alahari, K., and Mairal, J. Self-supervised models are continual learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 9621--9630, 2022

    2022

  9. [17]

    M., Krzakala, F., and Zdeborov \'a , L

    Goldt, S., Advani, M., Saxe, A. M., Krzakala, F., and Zdeborov \'a , L. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. Advances in neural information processing systems, 32, 2019

    2019

  10. [18]

    and Lakshman, V

    Guha, E. and Lakshman, V. On the diminishing returns of width for continual learning. arXiv preprint arXiv:2403.06398, 2024

    2024 arXiv

  11. [19]

    A., and Pascanu, R

    Hadsell, R., Rao, D., Rusu, A. A., and Pascanu, R. Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24 0 (12): 0 1028--1040, 2020

    2020

  12. [20]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  13. [21]

    Provable continual learning via sketched jacobian approximations

    Heckel, R. Provable continual learning via sketched jacobian approximations. In International Conference on Artificial Intelligence and Statistics, pp.\ 10448--10470. PMLR, 2022

    2022

  14. [22]

    M., and Tuytelaars, T

    Hess, T., Verwimp, E., van de Ven, G. M., and Tuytelaars, T. Knowledge accumulation in continually learned representations and the issue of feature forgetting. arXiv preprint arXiv:2304.00933, 2023

    2023 arXiv

  15. [23]

    and Akaho, S

    Karakida, R. and Akaho, S. Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting. arXiv preprint arXiv:2112.01653, 2021

    2021 arXiv

  16. [24]

    A theoretical study on solving continual learning

    Kim, G., Xiao, C., Konishi, T., Ke, Z., and Liu, B. A theoretical study on solving continual learning. Advances in neural information processing systems, 35: 0 5065--5079, 2022

    2022

  17. [25]

    Similarity of neural network representations revisited

    Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Similarity of neural network representations revisited. In International conference on machine learning, pp.\ 3519--3529. PMLR, 2019

    2019

  18. [26]

    P., Chakravarthi Raja, S., Cheney, N., Clune, J., et al

    Kudithipudi, D., Aguilar-Simon, M., Babb, J., Bazhenov, M., Blackiston, D., Bongard, J., Brna, A. P., Chakravarthi Raja, S., Cheney, N., Clune, J., et al. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4 0 (3): 0 196--210, 2022

    2022

  19. [27]

    Continual learning in the teacher-student setup: Impact of task similarity

    Lee, S., Goldt, S., and Saxe, A. Continual learning in the teacher-student setup: Impact of task similarity. In International Conference on Machine Learning, pp.\ 6109--6119. PMLR, 2021

    2021

  20. [28]

    Fixed design analysis of regularization-based continual learning

    Li, H., Wu, J., and Braverman, V. Fixed design analysis of regularization-based continual learning. In Conference on Lifelong Learning Agents, pp.\ 513--533. PMLR, 2023

    2023

  21. [29]

    S., and Oymak, S

    Li, Y., Li, M., Asif, M. S., and Oymak, S. Provable and efficient continual representation learning. arXiv preprint arXiv:2203.02026, 2022

    2022 arXiv

  22. [30]

    Theory on forgetting and generalization of continual learning

    Lin, S., Ju, P., Liang, Y., and Shroff, N. Theory on forgetting and generalization of continual learning. In International Conference on Machine Learning, pp.\ 21078--21100. PMLR, 2023

    2023

  23. [31]

    Investigating forgetting in pre-trained representations through continual learning

    Luo, Y., Yang, Z., Bai, X., Meng, F., Zhou, J., and Zhang, Y. Investigating forgetting in pre-trained representations through continual learning. arXiv preprint arXiv:2305.05968, 2023

    2023 arXiv

  24. [32]

    Domain adaptation: Learning bounds and algorithms

    Mansour, Y., Mohri, M., and Rostamizadeh, A. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.3430, 2009

    2009 arXiv

  25. [33]

    L., McNaughton, B

    McClelland, J. L., McNaughton, B. L., and O'Reilly, R. C. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102 0 (3): 0 419, 1995

    1995

  26. [34]

    and Cohen, N

    McCloskey, M. and Cohen, N. J. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp.\ 109--165. Elsevier, 1989

    1989

  27. [35]

    What is happening inside a continual learning model? a representation-based evaluation of representational forgetting

    Murata, K., Toyota, T., and Ohara, K. What is happening inside a continual learning model? a representation-based evaluation of representational forgetting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp.\ 234--235, 2020

    2020

  28. [36]

    The ideal continual learner: An agent that never forgets

    Peng, L., Giampouras, P., and Vidal, R. The ideal continual learner: An agent that never forgets. In International Conference on Machine Learning, pp.\ 27585--27610. PMLR, 2023

    2023

  29. [37]

    and Lampert, C

    Pentina, A. and Lampert, C. A pac-bayesian bound for lifelong learning. In International Conference on Machine Learning, pp.\ 991--999. PMLR, 2014

    2014

  30. [38]

    V., Dyer, E., and Raghu, M

    Ramasesh, V. V., Dyer, E., and Raghu, M. Anatomy of catastrophic forgetting: Hidden representations and task semantics. arXiv preprint arXiv:2007.07400, 2020

    2007 arXiv

  31. [39]

    W., and Hadsell, R

    Rao, D., Visin, F., Rusu, A., Pascanu, R., Teh, Y. W., and Hadsell, R. Continual unsupervised representation learning. Advances in neural information processing systems, 32, 2019

    2019

  32. [40]

    and Solla, S

    Saad, D. and Solla, S. A. Exact solution for on-line learning in multilayer neural networks. Physical Review Letters, 74 0 (21): 0 4337, 1995 a

    1995

  33. [41]

    and Solla, S

    Saad, D. and Solla, S. A. On-line learning in soft committee machines. Physical Review E, 52 0 (4): 0 4225, 1995 b

    1995

  34. [42]

    Supervised learning in the presence of concept drift: a modelling framework

    Straat, M., Abadi, F., Kan, Z., G \"o pfert, C., Hammer, B., and Biehl, M. Supervised learning in the presence of concept drift: a modelling framework. Neural Computing and Applications, 34 0 (1): 0 101--118, 2022

    2022

  35. [43]

    Provable contrastive continual learning

    Wen, Y., Tan, Z., Zheng, K., Xie, C., and Huang, W. Provable contrastive continual learning. arXiv preprint arXiv:2405.18756, 2024

    2024 arXiv

  36. [44]

    Optimization and generalization of regularization-based continual learning: a loss approximation viewpoint

    Yin, D., Farajtabar, M., Li, A., Levine, N., and Mott, A. Optimization and generalization of regularization-based continual learning: a loss approximation viewpoint. arXiv preprint arXiv:2006.10974, 2020

    2006 arXiv

  37. [45]

    and Okada, M

    Yoshida, Y. and Okada, M. Data-dependence of plateau phenomenon in learning with neural network---statistical mechanical analysis. Advances in Neural Information Processing Systems, 32, 2019

    2019

  38. [46]

    Feature forgetting in continual representation learning

    Zhang, X., Dou, D., and Wu, J. Feature forgetting in continual representation learning. arXiv preprint arXiv:2205.13359, 2022

    2022 arXiv

  39. [47]

    Integrating present and past in unsupervised continual learning

    Zhang, Y., Charlin, L., Zemel, R., and Ren, M. Integrating present and past in unsupervised continual learning. arXiv preprint arXiv:2404.19132, 2024

    2024 arXiv

  40. [48]

    A statistical theory of regularization-based continual learning

    Zhao, X., Wang, H., Huang, W., and Lin, W. A statistical theory of regularization-based continual learning. arXiv preprint arXiv:2406.06213, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.