Pith. sign in

REVIEW 4 major objections 4 minor 76 references

C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that adding first-order flatness—small gradient norms in a neighborhood of the solution—to sharpness-aware training improves continual learning, and that a gated variant achieves most of the gain at about 30% of the computa

desk verdict Useful extension with a modest, mostly honest empirical story; the C-Flat++ gate is conceptually shaky and the theory has an invalid step. read the letter →

arxiv 2508.18860 v2 pith:FL3R7YJZ submitted 2025-08-26 cs.LG cs.CV

Wei Li , Hangjie Yuan , Zixiang Zhao , Yifan Zhu , Aojun Lu , Tao Feng , Yanan Sun This is my paper
classification cs.LGcs.CV
keywords continuallearningcatastrophicforgettingsharpness-awareminimizationflatminimafirst-orderflatnessclass-incrementalHessianeigenvaluesselectivesharpness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that continual-learning models should be trained not just to flat minima in the SAM sense, but to minima that are flat in both the zeroth-order sense (low loss in a neighborhood) and the first-order sense (small gradient magnitude in that neighborhood). It packages this as C-Flat, a plug-in change to the loss used by any continual-learning method, and reports consistent accuracy gains—usually 0.7 to 1.9 points—across memory-, regularization-, and expansion-based methods on five benchmarks. To make the idea affordable, C-Flat++ applies the heavy sharpness updates selectively, using a cheap squared-gradient-norm proxy to decide when the landscape is sharp, and claims to keep most of the gain while spending roughly 25–30% of C-Flat's sharpness computations. A reader should care because catastrophic forgetting is often framed as a data or architecture problem; this paper argues it is also a landscape-geometry problem that an optimizer can fix.

What carries the argument

The load-bearing object is the C-Flat objective of Eq. (4): L_C(theta_T) = L_R0_rho(theta_T) + lambda * R1_rho(theta_T), with R0 the maximal loss increase inside a radius-rho ball and R1 the maximal gradient norm inside the ball, computed with Hessian-vector products instead of a full Hessian. The first-order term is the upper bound of the zeroth-order term and is linked to the maximal Hessian eigenvalue, so it drives optimization toward smooth basins. For C-Flat++, the machinery is the gating rule in Algorithm 3: it tracks E_{B_t}||grad L||^2, compares it against a sigmoidal threshold A/(1+e^{-k(i-i0)}) with error feedback, and only then applies full C-Flat updates.

What would settle it

Take a continual-learning training run and compute both the squared gradient norm and an actual sharpness estimate (e.g., the top Hessian eigenvalue or trace) at each checkpoint. If the gate triggers on low squared-gradient norm while Hessian sharpness is high, or fails to trigger when sharpness is high, the proxy is not measuring curvature. A stronger test: hold the gate's update schedule fixed but replace the proxy with a random schedule matched in frequency; if accuracy stays comparable, the selective rule itself is not the source of the gain.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that zeroth-order sharpness alone is insufficient for continual learning: for a fixed neighborhood radius, a lower neighborhood-maximum loss does not guarantee a smaller maximal Hessian eigenvalue, so SAM-style updates can settle in sharper basins. C-Flat therefore minimizes L_C(theta_T) = L_R0_rho(theta_T) + lambda * R1_rho(theta_T), where R1_rho is the first-order flatness rho * max||grad L|| inside the ball. Because R1 upper-bounds R0 and equals rho^2 times the maximal Hessian eigenvalue at a minimum, the combined objective targets smooth, wide basins that tolerate the parameter shifts induced by later tasks. The paper further claims that

Load-bearing premise

The C-Flat++ speedup assumes the squared norm of the batch gradient is a faithful cheap proxy for loss-landscape sharpness; near any local minimum the gradient norm is small whether the basin is flat or sharp, so the gate may be measuring progress toward an optimum rather than curvature.

Editorial extensions

If this is right

  • Plugging C-Flat into any of the seven base methods (Replay, iCaRL, WA, PODNet, DER, FOSTER, MEMO) raises average accuracy in nearly every dataset and split, with average returns around +0.72 to +1.88 accuracy points.
  • C-Flat++ keeps most of that gain while applying C-Flat-style updates only about 25–30% of the time, moving training speed close to SGD.
  • The first-order flatness term gives a concrete generalization index: at a minimum, R1 equals rho^2 times the maximal Hessian eigenvalue, so lower C-Flat loss implies a flatter basin.
  • The method replaces the zeroth-order flatness term in the FS-DGPM/GPM family and improves those gradient-projection baselines, so flatness can be combined with gradient-memory methods.
  • The same recipe extends beyond class-incremental learning to domain-incremental learning with a pre-trained model, where C-Flat and C-Flat++ improve DUCT on DomainNet.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because E||grad||^2 = ||E grad||^2 + Var(grad), the C-Flat++ gate conflates two things: near any local optimum the norm is small whether the basin is flat or sharp, so the gate may be detecting proximity to a stationary point rather than curvature. A direct test is to compare gate decisions with Hessian-based sharpness along the same trajectory.
  • If flatness is the active ingredient, the same optimizer should transfer to online, task-free continual learning and to parameter-efficient fine-tuning of large models; the paper demonstrates CIL and DIL but leaves those settings implicit.
  • The paper's own tables contain cells where C-Flat slightly hurts (e.g., FOSTER under one split), so the 'flatter is better' principle likely saturates when the base method already imposes strong regularization; characterizing that boundary would make the plug-in more predictable.
  • The gating hyperparameters A, k, and i0 are fixed across experiments; an adaptive per-layer or per-task gate based on the same proxy could push the cost below the reported 25–30% without losing accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes C-Flat, a plug-and-play optimization loss for continual learning that combines zeroth-order sharpness (SAM-style) with a first-order flatness regularizer (Eq. 4), and C-Flat++, a gated variant that applies C-Flat updates only when a squared-gradient-norm proxy exceeds a sigmoidal threshold (Alg. 3). The authors evaluate the two methods by wrapping seven CL baselines spanning memory-, regularization-, and expansion-based approaches across five datasets and multiple class-incremental schedules. They report average accuracy gains of roughly 0.7 to 1.9 points with C-Flat, and comparable gains with C-Flat++ while using about 24-31% of the sharpness-related computations. Supporting analyses include Hessian eigenvalue/trace reduction, loss-landscape visualizations, a hybrid SGD/C-Flat study, ablations over lambda and rho, and a domain-incremental experiment.

Significance. If the central claim is accepted, the paper provides a practically attractive result: a simple optimizer-level wrapper that improves many CL methods and, via C-Flat++, reduces the training overhead substantially. The empirical coverage is broad (seven methods, five datasets, multiple incremental settings), the code is publicly available, and the Hessian and landscape analyses are a useful complement to the accuracy tables. The GPM-family experiments also show a plausible path for integrating flatness into gradient-projection methods. However, the strength of the evidence is currently lower than the abstract's 'consistently improves' wording suggests: the effect sizes in Table II are often within one standard deviation of the baseline, no significance tests are reported, and two cells show clear degradations. The efficiency claim for C-Flat++ additionally rests on a proxy whose relationship to curvature is asserted rather than demonstrated. These issues are fixable, but they need to be addressed before the paper's central claims can be considered established.

major comments (4)
  1. [Table II and Section IV.B] The central claim that C-Flat 'consistently improves performance across a wide range of settings' is stronger than the data support. Many of the reported gains (Average Return +0.72 to +1.88 points) are comparable to or smaller than the reported standard deviations, and no significance tests or confidence intervals are given. There are also clear negative cells: WA on CIFAR-100 B0 Inc10 drops from 75.07±1.16 to 74.03±0.75 with C-Flat and 74.74±0.43 with C-Flat++; FOSTER on Tiny-ImageNet B0 Inc20 drops from 31.84±1.44 to 29.28±0.83 with C-Flat and 29.79±0.82 with C-Flat++. Please report paired significance tests (or at least confidence intervals over the same seeds) and revise the 'consistently improves' claim accordingly.
  2. [Section III.C, Algorithm 3] The sharpness proxy justification is mathematically incomplete. The text equates a lower E_{B_t}||∇L(θ)||² with 'reduced gradient variance,' but E||g||² = ||E g||² + tr Var(g). Near a local minimum, ||E g||² tends to zero both for flat and sharp minima, so the gate may simply be detecting convergence rather than curvature. The empirical support in Figure 3 and Figure 5 shows the proxy trajectory and trigger rate, but not the relationship between triggered epochs and Hessian curvature. Please validate the proxy by correlating it with Hessian eigenvalue/trace estimates during training, or by showing that the triggered updates occur selectively at high-curvature regions rather than following a fixed early-training schedule.
  3. [Section III.C, Algorithm 3] The gating mechanism depends on four hand-set hyperparameters: A=5, k=0.01, i0=80, and eta0=5e-3. The paper provides no sensitivity analysis for these parameters, although the whole efficiency claim of C-Flat++ rests on them. An ablation varying these values, or a principled data-dependent setting, is needed to establish that the reported 25-30% cost and the comparable accuracy are not tied to a single ad-hoc configuration.
  4. [Section III.A.3, Lemma 1 and Theorem 1] The convergence analysis contains a derivation that is at least incomplete and likely dimensionally inconsistent. After stating E[||∇L_{R1}||²] ≤ β² E[||ε1||²], the text substitutes ≤ β²η²E[||ρ_i^T||²] and then concludes ≤ 16(2√n_T−1)/(β² n_T). With ρ_i = ρ/4√i and ρ ≤ 1/(4β), the intermediate bound should scale as ρ²/i, not as 1/(β²√n_T). Please provide a fully spelled-out derivation or correct the lemma; as written, Theorem 1's bound is not verifiable from the preceding steps.
minor comments (4)
  1. [Algorithm 3] The notation g0 and g1 is used in the update step but never defined in Algorithm 3. Please refer explicitly to the corresponding quantities from Algorithm 1 (lines 11 and 14).
  2. [Algorithm 1, line 12] The denominator ||∇L(θ)||₂ + ε mixes the perturbation radius ρ with the small constant ε in notation; please rename the small constant (e.g., δ) to avoid confusion with the perturbation epsilon.
  3. [Table I / Figure 5d] The 'Proportion' for C-Flat++ is reported as an average percentage (e.g., 25.1%), but the text does not clarify whether this is the fraction of optimizer steps that performed the full C-Flat computation, or the fraction of epochs, or the fraction of sharpness perturbations. Please state the exact counting procedure.
  4. [Section IV.H] The discussion of backward transfer acknowledges larger relative forgetting in four of seven scenarios but explains it via 'higher standards' without a quantitative decomposition. Consider reporting absolute performance after the final task (as in Table V) directly in Figure 10 or in the text, so the reader can separate relative forgetting from absolute accuracy.

Circularity Check

1 steps flagged · score 4.0 of 10

External accuracy benchmarks keep the central claim non-circular, but the Hessian-eigenvalue evidence in Sec. IV.C largely re-measures the C-Flat objective, and the C-Flat++ gate's 'sharpness proxy' is an assumed identification rather than a derived prediction.

  1. self definitional [Section III.A.3 (Eq. 12) and Section IV.C (Hessian Eigenvalues and Hessian Traces)]
    ""The first-order flatness is proven to be related to the maximal eigenvalue of the Hessian matrix as 𝑅1 𝜌(𝜃∗) = 𝜌2·𝜆𝑚𝑎𝑥(∇2L(𝜃∗))" ... "In contrast, C-Flat induces a substantial reduction in Hessian eigenvalues to 28.11/6.25 at epochs 50/150 ... leading to flatter minima and improved CL performance.""

    C-Flat's optimization objective (Eq. 4) is explicitly L_R0 + λ·R1, and Eq. 12 shows that, at a stationary point, the first-order flatness term R1 equals ρ²·λ_max(H). Consequently, any successful minimization of the C-Flat objective is mathematically the same as minimizing the maximal Hessian eigenvalue (up to constants). Section IV.C then reports that C-Flat and C-Flat++ reduce Hessian eigenvalues and traces as evidence of flatter minima. This is a restatement of the training objective, not an independent empirical confirmation of the 'flatter is better' mechanism. The external accuracy comparisons in Table II are not circular, but this particular supporting evidence is self-confirming.

full rationale

The central performance claim is benchmarked against external baselines (Replay, iCaRL, WA, PODNet, DER, FOSTER, MEMO) in Table II and Figure 5, so it does not reduce to a fit or to a self-citation chain. Self-citations to the authors' prior conference paper [42] are used for the original C-Flat formulation, evaluation metrics, and code settings; they are not load-bearing in a way that forces the new results. The one genuinely circular evidentiary loop is Section IV.C: the paper defines C-Flat via a loss term that Eq. 12 equates with Hessian maximal eigenvalue, and then presents C-Flat's reduction of Hessian eigenvalues/traces as empirical support. That is measuring the objective being optimized. The C-Flat++ gating rule's identification of squared gradient norm with sharpness is a substantive unvalidated assumption (since E||g||² = ||E g||² + Var(g)), and it weakens the principled-efficiency claim, but I do not classify it as circularity because the reported accuracy/efficiency numbers are external measurements, not derivations from the proxy. Overall, partial circularity in a supporting analysis while the main accuracy claim retains independent content, hence score 4.

Assumptions & free parameters 7 free parameters · 6 assumptions · 1 invented entities

The central claims rest on several hand-set hyperparameters (λ, ρ, A, k, i0, η0 and the ρ scheduler endpoints) and on the stated smoothness/boundedness assumptions in Assumptions 1, which are not verified for the ResNet models in the experiments. The main domain assumption is that squared gradient norm is a usable sharpness proxy, which underpins C-Flat++ and is only loosely justified. No new physical or architectural entities are introduced; the sigmoid sharpness proxy is a heuristic controller rather than an independently evidenced quantity.

free parameters (7)
  • λ (first-order flatness weight) = 0.2 (all CIL runs)
    Hand-set; Fig. 8a shows average accuracy varies with λ, so the choice affects the reported gains.
  • ρ (perturbation radius) = 0.2 (CIL); 0.05 (DIL/DomainNet)
    Hand-set and changed per scenario with no stated rule; Fig. 8b shows sensitivity to ρ.
  • A (initial sharpness bound) = 5
    C-Flat++ hyperparameter, fixed; updated online by error feedback.
  • k (sigmoid curvature) = 0.01
    C-Flat++ hyperparameter, fixed across all runs.
  • i0 (sigmoid inflection epoch) = 80
    C-Flat++ hyperparameter, fixed.
  • η0 (sharpness-bound update rate) = 5e-3
    C-Flat++ hyperparameter, fixed.
  • ρ scheduler endpoints (ρ-, ρ+) = not reported numerically
    ρ_i = ρ + (ρ+-ρ)/(η+-η)·(η_i - η); Fig. 8d shows performance varies with the scheduler, endpoints chosen by hand.
assumptions (6)
  • domain assumption Assumptions 1: loss twice differentiable, bounded by M, bounded variance σ², β-Lipschitz loss and second-order gradient, η ≤ 1/β, ρ ≤ 1/(4β), schedules η_i = η/√i and ρ_i = ρ/4√i
    Stated before Claim 1 in Section III.A.3 and used by Lemma 1 and Theorem 1; not verified for the ResNet-18/32 models in the experiments.
  • standard math Mean value theorem bound R0_ρ(θ) ≤ R1_ρ(θ) (Eq. 7) and SAM-type perturbation approximations ∇L^{R0} ≈ ∇L(θ + ρ∇L/||∇L||) (Eq. 5)
    Standard calculus and the usual SAM first-order approximation; the max-over-ball in R0/R1 is replaced by evaluation at one perturbed point, an approximation rather than an equality.
  • domain assumption At a local minimum, R1_ρ(θ*) = ρ²·λ_max(∇²L(θ*)) (Eq. 12)
    Holds to second order for a smooth loss; connects first-order flatness to Hessian curvature and is used to argue C-Flat regularizes curvature.
  • standard math Convergence of zeroth-order sharpness minimization from Andriushchenko and Flammarion [66] (Eq. 8)
    Imported external bound; the paper's Claim 1 rests on it for the R0 part of the C-Flat loss.
  • domain assumption E_{B_t}||∇L(θ)||² is a valid sharpness proxy for gating C-Flat updates
    Underpins Algorithm 3. The paper equates it with gradient variance, but E||g||² = ||Eg||² + Var(g); near a stationary point it is small at both flat and sharp minima.
  • domain assumption Flat minima generalize better across sequential CL tasks ('flatter is better')
    Motivating premise for the whole method; supported by the paper's Hessian and landscape evidence and by prior flat-minima CL work, but assumed rather than proven.
invented entities (1)
  • Sigmoid sharpness proxy with error feedback (A/(1+e^{-k(i-i0)}), A ← A - η0·E)
    purpose: Gates when C-Flat specialized updates are applied in C-Flat++ (Algorithm 3)
    Introduced as a heuristic controller; the hyperparameters A=5, k=0.01, i0=80, η0=5e-3 are hand-set, and the proxy has no falsifiable handle outside the paper's own benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learning." pith.science (2026). https://pith.science/paper/FL3R7YJZ

@misc{pith2026250818860,
  author       = {Pith},
  title        = {Pith review of: C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FL3R7YJZ}},
  note         = {Machine review of arXiv:2508.18860}
}
read the original abstract

Balancing sensitivity to new tasks and stability for retaining past knowledge is crucial in continual learning (CL). Recently, sharpness-aware minimization has proven effective in transfer learning and has also been adopted in continual learning (CL) to improve memory retention and learning efficiency. However, relying on zeroth-order sharpness alone may favor sharper minima over flatter ones in certain settings, leading to less robust and potentially suboptimal solutions. In this paper, we propose \textbf{C}ontinual \textbf{Flat}ness (\textbf{C-Flat}), a method that promotes flatter loss landscapes tailored for CL. C-Flat offers plug-and-play compatibility, enabling easy integration with minimal modifications to the code pipeline. Besides, we present a general framework that integrates C-Flat into all major CL paradigms and conduct comprehensive comparisons with loss-minima optimizers and flat-minima-based CL methods. Our results show that C-Flat consistently improves performance across a wide range of settings. In addition, we introduce C-Flat++, an efficient yet effective framework that leverages selective flatness-driven promotion, significantly reducing the update cost required by C-Flat. Extensive experiments across multiple CL methods, datasets, and scenarios demonstrate the effectiveness and efficiency of our proposed approaches. Code is available at https://github.com/WanNaa/C-Flat.

Figures

Figures reproduced from arXiv: 2508.18860 by the authors.

Figure 1
Figure 1. Considering a loss landscape with several sharp local [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of C-Flat overcoming catastrophe forgetting by fine-tuning the old model parameter to flat minima of new task. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The squared gradient norm E𝐵𝑡 ∥∇L (𝜃) ∥2 and sharpness proxy of SGD and C-Flat across tasks. after post-processing. L 𝐶 ˆ𝑆 𝑇 ( 𝑓 𝑇 ) = L 𝑅 0 𝜌 ˆ𝑆 𝑇 ( [𝜓 𝑜𝑙𝑑, 𝜓𝑛𝑒𝑤] ( [𝜙 𝑇−1 𝑓 (𝜙𝑔), 𝜙𝑛𝑒𝑤 𝑓 (𝜙𝑔)])) + 𝜆 · L𝑅 1 𝜌 ˆ𝑆 𝑇 ( [𝜓 𝑜𝑙𝑑, 𝜓𝑛𝑒𝑤] ( [𝜙 𝑇−1 𝑓 (𝜙𝑔), 𝜙𝑛𝑒𝑤 𝑓 (𝜙𝑔)])). (14) To conclude, C-Flat can be easily applied to any CL method with reconstructed loss function, and thus trained with the corresponding optimize as shown … view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The parametric loss landscapes of Replay ( [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Analysis of C-Flat++: C-Flat++ with adaptive sharpness [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: As shown, models trained with vanilla-SGD exhibit [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: C-Flat vs. Zero-order flatness. TABLE III: Revisiting FS-DGPM series using C-Flat and C￾Flat++. Bold indicates the best results and underline denotes the second-best. Method La-GPM FS-GPM DGPM La-DGPM FS-DGPM Oracle [41] 72.90 73.12 72.66 72.85 73.14 w/ C-Flat [42] 73.…
Figure 6
Figure 6. Figure 6: The Hessian eigenvalues and the traces at epochs 50, and [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: Ablation studies. (a) and (b) illustrate the effect of [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 10
Figure 10. Figure 10: Analysis of knowledge transfer. TABLE V: Analysis of old and new task accuracy when adapting last new task. RR refers to Relative Return compared to SGD. Method CIFAR-100 / B0 Inc10 RR (C-Flat / C-Flat++) SGD C-Flat C-Flat++ Replay [14] old 35.96 36.87 36.67 +2.53%/+1…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

76 extracted references · 60 canonical work pages

  1. [1]

    Embracing change: Continual learning in deep neural networks,

    R. Hadsell, D. Rao, A. A. Rusu, and R. Pascanu, “Embracing change: Continual learning in deep neural networks,”Trends in cognitive sciences, vol. 24, no. 12, pp. 1028–1040, 2020

    work page 2020

  2. [2]

    Identifying players in broad- cast videos using graph convolutional network,

    T. Feng, K. Ji, A. Bian, C. Liu, and J. Zhang, “Identifying players in broad- cast videos using graph convolutional network,”Pattern Recognition, vol. 124, p. 108503, 2022

    work page 2022

  3. [3]

    Class-incremental learning: survey and performance evalua- tion on image classification,

    M. Masana, X. Liu, B. Twardowski, M. Menta, A. D. Bagdanov, and J. Van De Weijer, “Class-incremental learning: survey and performance evalua- tion on image classification,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

    work page 2022

  4. [4]

    Deep class-incremental learning: A survey,

    D.-W. Zhou, Q.-W. Wang, Z.-H. Qi, H.-J. Ye, D.-C. Zhan, and Z. Liu, “Deep class-incremental learning: A survey,” arXiv preprint arXiv:2302.03648, 2023

    arXiv 2023

  5. [5]

    Overcoming catastrophic forgetting in incremental object detection via elastic response distillation,

    T. Feng, M. Wang, and H. Yuan, “Overcoming catastrophic forgetting in incremental object detection via elastic response distillation,” in CVPR, 2022

    work page 2022

  6. [6]

    Rlip: Relational language-image pre-training for human-object interaction detection,

    H. Yuan, J. Jiang, S. Albanie, T. Feng, Z. Huang, D. Ni, and M. Tang, “Rlip: Relational language-image pre-training for human-object interaction detection,” in NeurIPS, 2022

    work page 2022

  7. [7]

    Rlipv2: Fast scaling of relational language-image pre-training,

    H. Yuan, S. Zhang, X. Wang, S. Albanie, Y. Pan, T. Feng, J. Jiang, D. Ni, Y. Zhang, and D. Zhao, “Rlipv2: Fast scaling of relational language-image pre-training,” in ICCV, 2023

    work page 2023

  8. [8]

    Palm-e: An embodied multimodal language model,

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu et al. , “Palm-e: An embodied multimodal language model,” arXiv preprint arXiv:2303.03378, 2023

    arXiv 2023

Show all 76 references
  1. [9]

    A stronger mixture of low-rank experts for fine-tuning foundation models,

    M. Sun, Y. Wang, T. Feng, D. Zhang, Y. Zhu, and J. Tang, “A stronger mixture of low-rank experts for fine-tuning foundation models,” arXiv preprint arXiv:2502.15828, 2025

    2025 arXiv

  2. [10]

    Rethinking the stability-plasticity trade-off in continual learning from an architectural perspective,

    A. Lu, H. Yuan, T. Feng, and Y. Sun, “Rethinking the stability-plasticity trade-off in continual learning from an architectural perspective,” arXiv preprint arXiv:2506.03951, 2025

    2025 arXiv

  3. [11]

    Parameter-efficient fine-tuning for foundation models,

    D. Zhang, T. Feng, L. Xue, Y. Wang, Y. Dong, and J. Tang, “Parameter-efficient fine-tuning for foundation models,” arXiv preprint arXiv:2501.13787, 2025

    2025 arXiv

  4. [12]

    A continual learning survey: Defying forgetting in classification tasks,

    M. Delange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars, “A continual learning survey: Defying forgetting in classification tasks,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021

    2021

  5. [13]

    icarl: Incremental classifier and representation learning,

    S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” in CVPR, 2017

    2017

  6. [14]

    Experience replay for continual learning,

    D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne, “Experience replay for continual learning,” in NeurIPS, vol. 32, 2019

    2019

  7. [15]

    Pcr: Proxy-based contrastive replay for online class-incremental continual learning,

    H. Lin, B. Zhang, S. Feng, X. Li, and Y. Ye, “Pcr: Proxy-based contrastive replay for online class-incremental continual learning,” in CVPR, 2023

    2023

  8. [16]

    Birt: Bio-inspired replay in vision transformers for continual learning,

    K. Jeeveswaran, P. Bhat, B. Zonooz, and E. Arani, “Birt: Bio-inspired replay in vision transformers for continual learning,” ICML, 2023

    2023

  9. [17]

    A model or 603 exemplars: Towards memory-efficient class-incremental learning,

    D.-W. Zhou, Q.-W. Wang, H.-J. Ye, and D.-C. Zhan, “A model or 603 exemplars: Towards memory-efficient class-incremental learning,”ICLR, 2023

    2023

  10. [18]

    Dense network expansion for class incremental learning,

    Z. Hu, Y. Li, J. Lyu, D. Gao, and N. Vasconcelos, “Dense network expansion for class incremental learning,” in CVPR, 2023

    2023

  11. [19]

    Self-sustaining representation expansion for non-exemplar class-incremental learning,

    K. Zhu, W. Zhai, Y. Cao, J. Luo, and Z.-J. Zha, “Self-sustaining representation expansion for non-exemplar class-incremental learning,” in CVPR, 2022

    2022

  12. [20]

    DER: dynamically expandable representation for class incremental learning,

    S. Yan, J. Xie, and X. He, “DER: dynamically expandable representation for class incremental learning,” in CVPR, 2021, pp. 3014–3023

    2021

  13. [21]

    Learning without forgetting,

    Z. Li and D. Hoiem, “Learning without forgetting,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 40, no. 12, pp. 2935–2947, 2018

    2018

  14. [22]

    Progressive learning without forgetting,

    T. Feng, H. Yuan, M. Wang, Z. Huang, A. Bian, and J. Zhang, “Progressive learning without forgetting,” arXiv preprint arXiv:2211.15215, 2022. 12

    2022 arXiv

  15. [23]

    Parameter-level soft-masking for continual learning,

    T. Konishi, M. Kurokawa, C. Ono, Z. Ke, G. Kim, and B. Liu, “Parameter-level soft-masking for continual learning,” arXiv preprint arXiv:2306.14775, 2023

    2023 arXiv

  16. [24]

    Continual learn- ing via sequential function-space variational inference,

    T. G. Rudner, F. B. Smith, Q. Feng, Y. W. Teh, and Y. Gal, “Continual learn- ing via sequential function-space variational inference,” in International Conference on Machine Learning . PMLR, 2022, pp. 18 871–18 887

    2022

  17. [25]

    Warping the space: Weight space rotation for class-incremental few-shot learning,

    D.-Y. Kim, D.-J. Han, J. Seo, and J. Moon, “Warping the space: Weight space rotation for class-incremental few-shot learning,” inICLR, 2022

    2022

  18. [26]

    Subspace regularizers for few-shot class incremental learning,

    A. F. Aky¨ urek, E. Aky¨ urek, D. T. Wijaya, and J. Andreas, “Subspace regularizers for few-shot class incremental learning,”ICLR, 2022

    2022

  19. [27]

    Continual learning with recursive gradient optimiza- tion,

    H. Liu and H. Liu, “Continual learning with recursive gradient optimiza- tion,” International Conference on Learning Representations , 2022

    2022

  20. [28]

    Gradient-based editing of memory examples for online task-free continual learning,

    X. Jin, A. Sadhu, J. Du, and X. Ren, “Gradient-based editing of memory examples for online task-free continual learning,”NeurIPS, 2021

    2021

  21. [29]

    Branch, or layer? zeroth-order optimization for continual learning of vision-language models,

    Z. Liu, B. Kang, W. Li, H. Yuan, Y. Yang, W. Li, J. Luo, Y. Zhu, and T. Feng, “Branch, or layer? zeroth-order optimization for continual learning of vision-language models,” arXiv preprint arXiv:2506.12409, 2025

    2025

  22. [30]

    Asymmetric valleys: Beyond sharp and flat local minima,

    H. He, G. Huang, and Y. Yuan, “Asymmetric valleys: Beyond sharp and flat local minima,” NeurIPS, vol. 32, 2019

    2019

  23. [31]

    Sharpness-aware minimization for efficiently improving generalization,

    P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, “Sharpness-aware minimization for efficiently improving generalization,” arXiv preprint arXiv:2010.01412, 2020

    2010 arXiv

  24. [32]

    Improving sharpness-aware minimization with fisher mask for better generalization on language models,

    Q. Zhong, L. Ding, L. Shen, P. Mi, J. Liu, B. Du, and D. Tao, “Improving sharpness-aware minimization with fisher mask for better generalization on language models,” arXiv preprint arXiv:2210.05497, 2022

    2022 arXiv

  25. [33]

    Surrogate gap minimization improves sharpness- aware training,

    J. Zhuang, B. Gong, L. Yuan, Y. Cui, H. Adam, N. Dvornek, S. Tatikonda, J. Duncan, and T. Liu, “Surrogate gap minimization improves sharpness- aware training,” arXiv preprint arXiv:2203.08065, 2022

    2022 arXiv

  26. [34]

    On large-batch training for deep learning: Generalization gap and sharp minima,

    N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, “On large-batch training for deep learning: Generalization gap and sharp minima,” arXiv preprint arXiv:1609.04836, 2016

    2016 arXiv

  27. [35]

    Shaping the learning landscape in neural networks around wide flat minima,

    C. Baldassi, F. Pittorino, and R. Zecchina, “Shaping the learning landscape in neural networks around wide flat minima,”Proceedings of the National Academy of Sciences , vol. 117, no. 1, pp. 161–170, 2020

    2020

  28. [36]

    Towards efficient and scalable sharpness-aware minimization,

    Y. Liu, S. Mai, X. Chen, C.-J. Hsieh, and Y. You, “Towards efficient and scalable sharpness-aware minimization,” in CVPR, 2022

    2022

  29. [37]

    Sharpness-aware training for free,

    J. Du, D. Zhou, J. Feng, V. Tan, and J. T. Zhou, “Sharpness-aware training for free,” NeurIPS

  30. [38]

    Sharpness-aware minimization for efficiently improving generalization,

    P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, “Sharpness-aware minimization for efficiently improving generalization,” in ICLR, 2021. [Online]. Available: https://openreview.net/forum?id=6Tm1mposlrM

    2021

  31. [39]

    Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima,

    G. Shi, J. Chen, W. Zhang, L.-M. Zhan, and X.-M. Wu, “Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima,” NeurIPS, 2021

    2021

  32. [40]

    Overcoming catastrophic forgetting in continual learning by exploring eigenvalues of hessian matrix,

    Y. Kong, L. Liu, H. Chen, J. Kacprzyk, and D. Tao, “Overcoming catastrophic forgetting in continual learning by exploring eigenvalues of hessian matrix,” IEEE Transactions on Neural Networks and Learning Systems, 2023

    2023

  33. [41]

    Flattening sharpness for dynamic gradient projection memory benefits continual learning,

    D. Deng, G. Chen, J. Hao, Q. Wang, and P.-A. Heng, “Flattening sharpness for dynamic gradient projection memory benefits continual learning,” NeurIPS, vol. 34, 2021

    2021

  34. [42]

    Make continual learning stronger via c-flat,

    A. Bian, W. Li, H. Yuan, M. Wang, Z. Zhao, A. Lu, P. Ji, T. Feng et al., “Make continual learning stronger via c-flat,” Advances in Neural Information Processing Systems, vol. 37, pp. 7608–7630, 2025

    2025

  35. [43]

    Continual learning with pre-trained models: A survey,

    D.-W. Zhou, H.-L. Sun, J. Ning, H.-J. Ye, and D.-C. Zhan, “Continual learning with pre-trained models: A survey,” in IJCAI, 2024, pp. 8363– 8371

    2024

  36. [44]

    Regularizing second-order influences for continual learning,

    Z. Sun, Y. Mu, and G. Hua, “Regularizing second-order influences for continual learning,” in CVPR, 2023

    2023

  37. [45]

    Trgp: Trust region gradient projection for continual learning,

    S. Lin, L. Yang, D. Fan, and J. Zhang, “Trgp: Trust region gradient projection for continual learning,” arXiv preprint arXiv:2202.02931 , 2022

    2022 arXiv

  38. [46]

    Gradient projection memory for continual learning,

    G. Saha, I. Garg, and K. Roy, “Gradient projection memory for continual learning,” in International Conference on Learning Representations , 2020

    2020

  39. [47]

    Decoupling learning and remembering: A bilevel memory framework with knowledge projection for task-incremental learning,

    W. Sun, Q. Li, J. Zhang, W. Wang, and Y.-a. Geng, “Decoupling learning and remembering: A bilevel memory framework with knowledge projection for task-incremental learning,” inCVPR, 2023

    2023

  40. [48]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al. , “Overcoming catastrophic forgetting in neural networks,” Proceedings of the national academy of sciences , vol. 114, no. 13, pp. 3521–3...

    2017

  41. [49]

    Cpr: classifier- projection regularization for continual learning,

    S. Cha, H. Hsu, T. Hwang, F. P. Calmon, and T. Moon, “Cpr: classifier- projection regularization for continual learning,” ICLR, 2021

    2021

  42. [50]

    Task-aware information routing from common representation space in lifelong learning,

    P. Bhat, B. Zonooz, and E. Arani, “Task-aware information routing from common representation space in lifelong learning,” ICLR, 2023

    2023

  43. [51]

    R-dfcil: Relation-guided representation learning for data-free class incremental learning,

    Q. Gao, C. Zhao, B. Ghanem, and J. Zhang, “R-dfcil: Relation-guided representation learning for data-free class incremental learning,” inECCV, 2022

    2022

  44. [52]

    Alife: Adaptive logit regularizer and feature replay for incremental semantic segmentation,

    Y. Oh, D. Baek, and B. Ham, “Alife: Adaptive logit regularizer and feature replay for incremental semantic segmentation,”NeurIPS, 2022

    2022

  45. [53]

    Revisiting neural networks for continual learning: An architectural perspective,

    A. Lu, T. Feng, H. Yuan, X. Song, and Y. Sun, “Revisiting neural networks for continual learning: An architectural perspective,” IJCAI, 2024

    2024

  46. [54]

    Overcoming catas- trophic forgetting with hard attention to the task,

    J. Serr `a, D. Suris, M. Miron, and A. Karatzoglou, “Overcoming catas- trophic forgetting with hard attention to the task,” in ICML, 2018, pp. 4555–4564

    2018

  47. [55]

    Adaptive aggregation networks for class- incremental learning,

    Y. Liu, B. Schiele, and Q. Sun, “Adaptive aggregation networks for class- incremental learning,” in CVPR, 2021

    2021

  48. [56]

    Unigrad-fs: Unified gradient projection with flatter sharpness for continual learning,

    W. Li, T. Feng, H. Yuan, A. Bian, G. Du, S. Liang, J. Gan, and Z. Liu, “Unigrad-fs: Unified gradient projection with flatter sharpness for continual learning,” IEEE Transactions on Industrial Informatics , 2024

    2024

  49. [57]

    An empirical investigation of the role of pre-training in lifelong learning,

    S. V. Mehta, D. Patil, S. Chandar, and E. Strubell, “An empirical investigation of the role of pre-training in lifelong learning,” J. Mach. Learn. Res. , vol. 24, pp. 214:1–214:50, 2023. [Online]. Available: https://jmlr.org/papers/v24/22-0496.html

    2023

  50. [58]

    Zeroflow: Overcoming catastrophic forgetting is easier than you think,

    T. Feng, W. Li, D. Zhu, H. Yuan, W. Zheng, D. Zhang, and J. Tang, “Zeroflow: Overcoming catastrophic forgetting is easier than you think,” arXiv preprint arXiv:2501.01045, 2025

    2025 arXiv

  51. [59]

    Efficient lifelong learning with a-gem,

    A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny, “Efficient lifelong learning with a-gem,” arXiv preprint arXiv:1812.00420, 2018

    2018 arXiv

  52. [60]

    Gradient episodic memory for continual learning,

    D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,” NeurIPS, 2017

    2017

  53. [61]

    Orthogonal gradient descent for continual learning,

    M. Farajtabar, N. Azizan, A. Mott, and A. Li, “Orthogonal gradient descent for continual learning,” in International Conference on Artificial Intelligence and Statistics. PMLR, 2020, pp. 3762–3773

    2020

  54. [62]

    Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models,

    Z. Wang and Y. Tsvetkov, “Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models,” inProceedings of the International Conference on Learning Representations (ICLR) , 2021

    2021

  55. [63]

    Sharpness-aware gradient guidance for few-shot class-incremental learning,

    R. Chen, X. Jing, F. Wu, and H. Chen, “Sharpness-aware gradient guidance for few-shot class-incremental learning,” Knowl. Based Syst. , vol. 299, p. 112030, 2024. [Online]. Available: https://doi.org/10.1016/j.knosys.2024.112030

    2024

  56. [64]

    Data augmented flatness-aware gradient projection for continual learning,

    E. Yang, L. Shen, Z. Wang, S. Liu, G. Guo, and X. Wang, “Data augmented flatness-aware gradient projection for continual learning,” in IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023 . IEEE, 2023, pp. 5607–5616. [Online]. Avail...

    2023

  57. [65]

    Sharpness and gradient aware minimization for memory-based continual learning,

    L. T. Tung, V. N. Van, P. N. Hoang, and K. Than, “Sharpness and gradient aware minimization for memory-based continual learning,” in Proceedings of the 12th International Symposium on Information and Communication Technology, SOICT 2023, Ho Chi Minh, Vietnam, December 7-8, 202...

    2023

  58. [66]

    Towards understanding sharpness-aware minimization,

    M. Andriushchenko and N. Flammarion, “Towards understanding sharpness-aware minimization,” in ICML, 2022

    2022

  59. [67]

    Memory aware synapses: Learning what (not) to forget,

    R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, “Memory aware synapses: Learning what (not) to forget,” inProceedings of the European Conference on Computer Vision (ECCV) , 2018, pp. 139–154

    2018

  60. [68]

    Pycil: A python toolbox for class-incremental learning,

    D.-W. Zhou, F.-Y. Wang, H.-J. Ye, and D.-C. Zhan, “Pycil: A python toolbox for class-incremental learning,” 2023

    2023

  61. [69]

    Podnet: Pooled outputs distillation for small-tasks incremental learning,

    A. Douillard, M. Cord, C. Ollion, T. Robert, and E. Valle, “Podnet: Pooled outputs distillation for small-tasks incremental learning,” inECCV, 2020

    2020

  62. [70]

    Maintaining discrimination and fairness in class incremental learning,

    B. Zhao, X. Xiao, G. Gan, B. Zhang, and S.-T. Xia, “Maintaining discrimination and fairness in class incremental learning,” inCVPR, 2020

    2020

  63. [71]

    Foster: Feature boosting and compression for class-incremental learning,

    F.-Y. Wang, D.-W. Zhou, H.-J. Ye, and D.-C. Zhan, “Foster: Feature boosting and compression for class-incremental learning,” in European conference on computer vision , 2022, pp. 398–414

    2022

  64. [72]

    Three types of incremental learning,

    G. M. van de Ven, T. Tuytelaars, and A. S. Tolias, “Three types of incremental learning,” Nature Machine Intelligence , pp. 1185–1197, 2022

    2022

  65. [73]

    Pyhessian: Neural networks through the lens of the hessian,

    Z. Yao, A. Gholami, K. Keutzer, and M. W. Mahoney, “Pyhessian: Neural networks through the lens of the hessian,” 2019. [Online]. Available: http://arxiv.org/abs/1912.07145

    2019 arXiv

  66. [74]

    A comprehensive survey of continual learning: Theory, method and application,

    L. Wang, X. Zhang, H. Su, and J. Zhu, “A comprehensive survey of continual learning: Theory, method and application,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024

    2024

  67. [75]

    Dual consolidation for pre-trained model-based domain-incremental learning,

    D.-W. Zhou, Z.-W. Cai, H.-J. Ye, L. Zhang, and D.-C. Zhan, “Dual consolidation for pre-trained model-based domain-incremental learning,” arXiv preprint arXiv:2410.00911, 2024. 13

    2024 arXiv

  68. [76]

    Moment matching for multi-source domain adaptation,

    X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang, “Moment matching for multi-source domain adaptation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1406– 1415

    2019

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.