Pith. sign in

REVIEW 5 major objections 4 minor 78 references

Enhancing Pre-Trained Model-Based Class-Incremental Learning through Neural Collapse

T0 review · 5 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Catastrophic forgetting in pre-trained-model class-incremental learning is a loss of inter-class separability, and maintaining a growing equiangular feature frame keeps old and new classes separable.

desk verdict Plausible NC-based CIL method with strong reported gains, but the inference equation omits the alignment layer the whole mechanism depends on—needs a careful rewrite before the results can be trusted. read the letter →

arxiv 2504.18437 v1 pith:J4J32G4E submitted 2025-04-25 cs.LG cs.AI

Kun He , Zijian Song , Shuoxi Zhang , John E. Hopcroft This is my paper · ORCID
classification cs.LGcs.AI
keywords class-incrementallearningneuralcollapsepre-trainedmodelssimplexequiangulartightframecatastrophicforgettingETFalignmentcontinual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that catastrophic forgetting in class-incremental learning with pre-trained models is, at root, a geometric failure: as new classes arrive, the features of old classes lose their linear separability relative to one another. To test that diagnosis, the authors monitor the neural-collapse metric NC2 during incremental training and find it rises as accuracy falls, across several existing methods. They then design NCPTM-CIL, which keeps features close to a growing simplex equiangular tight frame by combining a dynamic ETF classifier, an alignment layer fitted to stored class means, and a pull-and-push loss that attracts each class mean to its own prototype and repels it from others. On CIFAR-100, CUB-200, VTAB, and OmniBenchmark, they report state-of-the-art average accuracy, and on CIFAR-100 and OmniBenchmark they come within 1.5% and 0.52% of joint-training accuracy respectively. If the diagnosis is right, neural-collapse geometry is a practical design principle for continual learning rather than just a terminal-phase curiosity.

What carries the argument

The load-bearing object is the simplex equiangular tight frame (ETF), a set of class weight vectors in which every pair has the same angle and the vectors are balanced around zero; in neural collapse, class means converge to such a frame and to the classifier weights, a property called self-duality. NCPTM-CIL makes this geometry dynamic: the dynamic ETF classifier builds the frame for the current number of classes and expands it when new classes arrive; the alignment layer, an MLP over stored class-mean features, maps drifting class means onto the frame; and the pull-and-push loss enforces unit cosine similarity to the correct prototype and negative cosine similarity to all others, whose global optimum is exactly the ETF condition. Together these components translate the claim that forgetting is loss of separability into a procedure that re-establishes the separability after every task.

What would settle it

Run a trained NCPTM-CIL model and measure classification accuracy and NC2 score on held-out test features in two inference variants: raw features compared with ETF weights, and raw features first passed through the learned alignment layer then compared. If applying the alignment layer at test time changes accuracy substantially, or if raw test features have high NC2 while class means used in training are well aligned, then the learned geometry is not actually carried by the raw representation and the reported procedure does not implement its own claim.

Watch

Extended reading notes

Core claim

Central claim: catastrophic forgetting in PTM-based CIL essentially stems from the attenuation of inter-class linear separability capacity. The paper supports this with NC2 trajectories showing separability degrades across tasks, and with a correlation between methods' accuracy and NC2. Its proposed NCPTM-CIL maintains a dynamic ETF classifier: as classes grow, the classifier adds a new vertex to a simplex equiangular tight frame, an alignment layer maps stored class-mean features to that frame, and a pull-push loss aligns each class mean with its own weight vector while pushing it away from the others. With ViT-B/16-IN1K initialization, it reports gains over the runner-up of 6.73% on VTAB, 1.25% on CIFAR-100, and 2.5% on OmniBenchmark, and near-joint-learning final accuracy on CIFAR-100, CUB-200, and OmniBenchmark. The intended upshot: keeping the feature space in equiangular form throughout incremental training is what prevents old classes from being forgotten.

Load-bearing premise

The method assumes that aligning stored class-mean features to the ETF geometry also brings raw test features into that geometry, although the reported inference compares raw features directly with the ETF weight vectors rather than passing them through the alignment layer.

Editorial extensions

If this is right

  • If forgetting is loss of inter-class separability, then NC metrics, especially NC2, can serve as a monitoring signal for when a continual learner is about to forget, and as a cheap model-selection criterion using only the base-stage NC1 value.
  • Maintaining an ETF geometry that grows with the class count removes the need to know the final label space in advance, which fixes a practical limitation of fixed simplex-ETF classifiers.
  • NCPTM-CIL's near-joint accuracy on CIFAR-100, CUB-200, and OmniBenchmark suggests that rehearsal-free incremental learning can approach the joint-training upper bound when the feature geometry is stabilized.
  • The reported gains on out-of-distribution benchmarks such as VTAB and OmniBenchmark imply the benefit is largest when the pre-trained features are far from the target domain, since that is where drift is most severe.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the alignment layer were applied at inference time, the method would likely become more consistent with its own mechanism; the paper is ambiguous about whether raw features are already in ETF space. This is a testable change, not a claim the paper makes.
  • Beyond the paper: the class-mean-only memory footprint suggests NCPTM-CIL could be adapted to privacy-sensitive or memory-constrained continual learning, where raw exemplars are not stored; the paper does not discuss this.
  • Beyond the paper: the observed monotone relation between base-stage NC1 and final accuracy could be turned into a predictive probe for picking among fine-tuning strategies across other backbones and datasets, independent of the NCPTM-CIL pipeline.
  • Beyond the paper: dynamic ETF expansion may also be applicable to open-set or few-shot incremental settings where the number of classes is unknown and class prototypes are sparse; the paper does not test these.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes NCPTM-CIL, a class-incremental learning method for pre-trained models that uses neural collapse (NC) geometry as a guiding principle. The method fine-tunes a ViT on the base task, then freezes it and introduces a dynamic ETF classifier whose weight matrix is expanded as new classes arrive. An alignment layer maps stored class-mean features to the ETF, and a pull-and-push (PAP) loss is designed to enforce the ETF geometry. The paper claims state-of-the-art average accuracy on CIFAR-100, CUB-200, VTAB, and OmniBenchmark, and reports that NCPTM-CIL approaches joint-training upper bounds. It also argues, based on NC2 metric trajectories and correlations, that catastrophic forgetting in PTM-CIL essentially stems from attenuation of inter-class linear separability.

Significance. If the results hold, the paper makes a useful contribution: it brings the neural-collapse perspective to PTM-based CIL, proposes a conceptually simple dynamic ETF classifier with an alignment layer and PAP loss, and reports consistent gains over existing methods, including a large 6.73% improvement on VTAB. The ablation study and NC-metric analysis are a good start toward explaining why the method works. However, the central mechanism as described is not reproducible from the manuscript: the inference equation omits the alignment layer, the per-task re-initialization of that layer is not reconciled with the dynamic ETF, and the causality claims rest on correlations that are partly by construction. These issues currently prevent acceptance.

major comments (5)
  1. [Sec. 4.1, Eq. (5) vs Sec. 4.3] The inference rule in Eq. (5) computes cosine similarity between the raw test feature f(x_i) and the ETF weight vectors w_hat_k, with no application of the alignment layer A_t. However, Sec. 4.3 trains A_t only on class-mean features h_k to produce aligned class means c_hat_k = A_t(h_k) that match the ETF weights. The paper nowhere states that raw features and ETF weights reside in the same space, nor does it specify which alignment layer (if any) is applied at test time. Consequently, as written, the alignment layer cannot affect the predictions, and the reported accuracy gains plus the NC-alignment mechanism in Secs. 3.2 and 4.2 are unexplained. Please provide the corrected inference equation (e.g., applying A_t to f(x_i) or mapping both to a common space) and specify how alignment layers from different tasks are combined or selected at test time.
  2. [Sec. 4.3] The text states that a new set of alignment layer parameters is initialized for each task. With this design, the paper does not explain how previously learned classes remain aligned with the dynamic ETF after their alignment layer is discarded. The class-mean pool is updated, but no procedure is given for recomputing old class prototypes under the new alignment mapping. Without this detail, the method's ability to "preserve the orthogonality and separability of previously learned classes" is not demonstrated, and the experimental results are not reproducible.
  3. [Sec. 4.2, Eqs. (7)-(8)] The push term in Eq. (7) uses 1/(k-1) where k is the class index, but the simplex ETF target for the cosine between distinct class vectors is -1/(K_t - 1), with K_t the current total number of classes. Since k ranges over classes, the target varies per class, which is inconsistent with the stated global optimum. Additionally, Eq. (8) is not the gradient of Eq. (7) as written: for unnormalized vectors, the derivative of (w^T c - 1)^2 with respect to c is 2(w^T c - 1)w, not (1 - cos angle)w. If the loss or feature normalization differs from what is written, the derivation must be provided. This affects the theoretical claim that PAP loss drives the class-mean matrix to a simplex ETF.
  4. [Sec. 3.2 and Sec. 5.4] The conclusion that catastrophic forgetting "essentially stems from the attenuation of inter-class linear separability capacity" is supported only by correlational evidence (Fig. 1b, Fig. 5, Tab. 5). In Tab. 5, the NC2 values come from a method whose PAP loss explicitly optimizes the NC2 objective, so observing lower NC2 for NCPTM-CIL is partly by construction and is not independent evidence that NC causes the accuracy gain. A control experiment, such as aligning to a random orthonormal frame instead of the ETF, or reporting NC2 on the actual inference-time features after correcting Eq. (5), would provide a more direct test of the causal mechanism.
  5. [Sec. 5] All results in Tables 1-5 are reported as single numbers without standard deviations or number of seeds, and no code is provided. Given the method's free choices (Phase II epochs, PAP loss weight, alignment-layer architecture, fine-tuning ranks), the smaller reported gains (e.g., the 1.25% on CIFAR-100 in Fig. 4) could be within run-to-run variation. Reporting mean±std over at least three seeds and releasing code would substantiate the state-of-the-art claim.
minor comments (4)
  1. [Sec. 5] The text says "five benchmark datasets" but only four are listed and evaluated (CIFAR-100, CUB-200, VTAB, OmniBenchmark).
  2. [Abstract and Sec. 5.2] The 6.73%, 1.25%, and 2.5% improvements are quoted in the abstract and Sec. 5.2, but they correspond to Fig. 4 (ViT-B/16-IN1K, final-stage accuracy), whereas Table 1 reports average accuracy with ViT-B/16-IN21K. Please clarify which backbone and which metric each number refers to.
  3. [Sec. 5.1] The training details state "we use the SGD optimizer to train ALIGNMENT LAYER" and later mention reducing epochs when overfitting is observed; specifying the alignment-layer architecture, learning rate, and the exact stopping rule would improve reproducibility.
  4. [Table 4] The column header "N C1 CIFAR-100" is misformatted and should read "NC1" with the dataset name separated clearly.

Circularity Check

1 steps flagged · score 4.0 of 10

External benchmark claims are not circular, but the paper's NC2-based validation of the neural-collapse explanation is circular because the PAP loss is defined to optimize the NC2 metric it later cites as evidence.

  1. self definitional [Sec. 4.2 Eq. (7)-(8) and Sec. 5.4 Table 5]
    "It can be concluded that global optimality occurs when cos ∠( ˆwk, ˆck) = 1 and cos ∠( ˆwj, ˆck) = − 1/(k−1), which theoretically demonstrates that the class mean matrix ultimately forms a simplex ETF, satisfying the NC 2 property. ... As shown in Tab. 5, performance declines as key components are removed, and the corresponding NC 2 and NC 3 values increase. ... This aligns with the objective of our PAP loss, further validating the effectiveness of our approach."

    The PAP loss in Eq. (7) is exactly the squared-error objective whose global optimum is the simplex-ETF condition measured by NC2 in Eq. (3): the pull term drives (w_k^T c_k - 1) to zero and the push term drives (w_j^T c_k + 1/(k-1)) to zero for every j. Thus the class-mean matrix is, by construction, being optimized to make NC2 small. Observing that removing PAP loss raises NC2 and adding it lowers NC2 is therefore a mathematical consequence of the loss definition, not independent empirical evidence that neural collapse causes the accuracy gain.

full rationale

The main performance claim of NCPTM-CIL is benchmarked against external methods (L2P, DualPrompt, CODA-Prompt, LAE, Aper, EASE) and against a joint-learning upper bound, so it is not circular: the reported accuracies are not fitted from the quantities being explained. There is no load-bearing self-citation chain; the dynamic ETF construction and the alignment-layer architecture are drawn from external prior work ([75], [17]). The notable circular element is the NC2-based validation: the PAP loss is literally constructed to realize the simplex-ETF/NC2 geometry, so the measured NC2 improvement with PAP loss (and worsening without it) is by construction rather than an independent test of the hypothesis that neural collapse causes the gain. A separate correctness gap is worth flagging but is not circularity: Eq. (5) scores test samples using the raw backbone feature f(x_i) against ETF weights, while Sec. 4.3 trains the alignment layer only on stored class means and even re-initializes it each task; the paper never states that raw test features and ETF weights are in the same aligned space. If Eq. (5) is literal, the alignment layers cannot affect inference, and if it is a typo, the paper is not reproducible as written. Either way this does not make the external accuracy claim circular, but it weakens support for the proposed mechanism. Overall, the central benchmark claim retains independent content, while the explanatory NC claim is partially circular, supporting a score of 4.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method rests on standard ETF existence and several domain assumptions about frozen pre-trained features. No new physical or mathematical entities are introduced. The dynamic ETF classifier is a construction, not a postulated entity. The least supported pieces are the orthogonal extension of U_t and the transfer from aligned class means to raw test features.

free parameters (4)
  • Phase II training epochs = 40, reduced per dataset when overfitting is observed
    Training length is tuned per dataset with no explicit protocol, making it a free experimental choice.
  • PAP loss weighting = implicitly 1.0
    The total loss L_CE + L_PAP uses equal weighting with no sensitivity study; the balance is chosen without justification.
  • Fine-tuning ranks = LoRA rank 8, AdaptFormer rank 64
    Ranks are inherited from prior work and not varied or justified in this paper.
  • Alignment layer architecture = unknown
    The paper only says MLP following Hersche [17]; hidden sizes, depth, and normalization are not specified.
assumptions (4)
  • standard math A simplex ETF exists whenever d >= K_t - 1 and U_t can be extended orthogonally.
    Invoked in Eq. (6) for the dynamic ETF classifier, but the construction of the expanding orthogonal matrix U_t is not specified.
  • domain assumption Catastrophic forgetting is primarily a loss of linear separability, so enforcing NC geometry is a sufficient corrective.
    Section 3.2 states this based on correlational evidence in Fig. 1b; it is the main motivation for the method.
  • domain assumption Class means are sufficient summary statistics for classifying all future samples after alignment.
    Section 4.3 stores class means and trains the alignment layer on them, but inference acts on individual test features. The transfer from class means to test features is assumed.
  • domain assumption The pre-trained backbone features are stable enough that freezing after base learning preserves representational quality.
    Section 4.1 freezes the model in phase II; any feature drift after base learning would invalidate the stored class means.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Pre-Trained Model-Based Class-Incremental Learning through Neural Collapse." pith.science (2026). https://pith.science/paper/J4J32G4E

@misc{pith2026250418437,
  author       = {Pith},
  title        = {Pith review of: Enhancing Pre-Trained Model-Based Class-Incremental Learning through Neural Collapse},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J4J32G4E}},
  note         = {Machine review of arXiv:2504.18437}
}
read the original abstract

Class-Incremental Learning (CIL) is a critical capability for real-world applications, enabling learning systems to adapt to new tasks while retaining knowledge from previous ones. Recent advancements in pre-trained models (PTMs) have significantly advanced the field of CIL, demonstrating superior performance over traditional methods. However, understanding how features evolve and are distributed across incremental tasks remains an open challenge. In this paper, we propose a novel approach to modeling feature evolution in PTM-based CIL through the lens of neural collapse (NC), a striking phenomenon observed in the final phase of training, which leads to a well-separated, equiangular feature space. We explore the connection between NC and CIL effectiveness, showing that aligning feature distributions with the NC geometry enhances the ability to capture the dynamic behavior of continual learning. Based on this insight, we introduce Neural Collapse-inspired Pre-Trained Model-based CIL (NCPTM-CIL), a method that dynamically adjusts the feature space to conform to the elegant NC structure, thereby enhancing the continual learning process. Extensive experiments demonstrate that NCPTM-CIL outperforms state-of-the-art methods across four benchmark datasets. Notably, when initialized with ViT-B/16-IN1K, NCPTM-CIL surpasses the runner-up method by 6.73% on VTAB, 1.25% on CIFAR-100, and 2.5% on OmniBenchmark.

Figures

Figures reproduced from arXiv: 2504.18437 by the authors.

Figure 1
Figure 1. (a): The N C2 metric with respect to the number of epochs during fine-tuning for ViT-B/16-IN1K and ViT-B/16-IN21K. (b): The N C2 values of different PTM-CIL methods on the VTAB dataset during the incremental learning phase, where VTAB is divided into five incremental learning tasks, each containing 10 classes. dition: M = r K K − 1 U  IK − 1 K 1K1 T K  , (1) where IK ∈ R K×K is the identity matrix, 1K ∈ R K is a v… view at source ↗
Figure 2
Figure 2. Illustration of NCPTM-CIL. Phase I: Base Learning. We utilize the initial task dataset [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Description of the gradient of PAP Loss with respect [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: All methods are initialized with ViT-B/16-IN1K. We annotate the relative improvement of NCPTM-CIL over the runner-up method with numerical values at the final incremental stage. 1.055 0.975 0.864 0.842 0.821 0.801 0.562 0 0.2 0.4 0.6 0.8 1 1.2 40 50 60 70 80 90 100 L2P…
Figure 5
Figure 5. Figure 5: The relationship between the performance of existing [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Top: t-SNE [47] visualization of the features of the Aper algorithm on the CIFAR-100 test dataset with 30 and 50 classes. Bottom: t-SNE visualization of the features of NCPTM-CIL on the CIFAR-100 test dataset with 30 and 50 classes. an input x ∈ R k×d , the AdaptFormer…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 44 canonical work pages

  1. [1]

    On the implicit geometry of cross- entropy parameterizations for label-imbalanced data

    Tina Behnia, Ganesh Ramachandra Kini, Vala Vakilian, and Christos Thrampoulidis. On the implicit geometry of cross- entropy parameterizations for label-imbalanced data. In In- ternational Conference on Artificial Intelligence and Statis- tics, pages 10815–10838. PMLR, 2023. 2

    work page 2023

  2. [2]

    Dark experience for gen- eral continual learning: a strong, simple baseline

    Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for gen- eral continual learning: a strong, simple baseline. Advances in neural information processing systems, 33:15920–15930,

  3. [3]

    Riemannian walk for incremen- tal learning: Understanding forgetting and intransigence

    Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajan- than, and Philip HS Torr. Riemannian walk for incremen- tal learning: Understanding forgetting and intransigence. In Proceedings of the European conference on computer vision (ECCV), pages 532–547, 2018. 2

    2018

  4. [4]

    Mmdetection: Open mmlab detection tool- box and benchmark

    Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection tool- box and benchmark. arXiv preprint arXiv:1906.07155, 2019. 1

    arXiv 1906

  5. [5]

    Adaptformer: Adapting vision transformers for scalable visual recogni- tion

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recogni- tion. Advances in Neural Information Processing Systems , 35:16664–16678, 2022. 4, 5, 1

    work page 2022

  6. [6]

    MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark

    MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https : / / github . com / open - mmlab/mmsegmentation, 2020. 1

    work page 2020

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 1

    arXiv 2010

  8. [8]

    Podnet: Pooled outputs dis- tillation for small-tasks incremental learning

    Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs dis- tillation for small-tasks incremental learning. In Computer vision–ECCV 2020: 16th European conference, Glasgow, UK, August 23–28, 2020, proceedings, part XX 16 , pages 86–102. Springer, 2020. 2

    work page 2020

Show all 78 references
  1. [9]

    Explor- ing deep neural networks via layer-peeled model: Minority collapse in imbalanced training

    Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Explor- ing deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43):e2103091118, 2021. 2

    2021

  2. [10]

    On the role of neural collapse in transfer learning

    Tomer Galanti, Andr ´as Gy ¨orgy, and Marcus Hutter. On the role of neural collapse in transfer learning. arXiv preprint arXiv:2112.15121, 2021. 2

    2021 arXiv

  3. [11]

    A unified continual learn- ing framework with general parameter-efficient tuning

    Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learn- ing framework with general parameter-efficient tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11483–11493, 2023. 6, 7

    2023

  4. [12]

    Fast r-cnn

    Ross Girshick. Fast r-cnn. In Proceedings of the IEEE Inter- national Conference on Computer Vision, pages 1440–1448,

  5. [13]

    Dissecting supervised contrastive learning

    Florian Graf, Christoph Hofer, Marc Niethammer, and Roland Kwitt. Dissecting supervised contrastive learning. In International Conference on Machine Learning , pages 3821–3830. PMLR, 2021. 2

    2021

  6. [14]

    Neural col- lapse under mse loss: Proximity to and dynamics on the cen- tral path

    XY Han, Vardan Papyan, and David L Donoho. Neural col- lapse under mse loss: Proximity to and dynamics on the cen- tral path. arXiv preprint arXiv:2106.02073, 2021. 2

    2021 arXiv

  7. [15]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 770–778, 2016. 1

    2016

  8. [16]

    Identity mappings in deep residual networks

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision–ECCV 2016: 14th European Conference, 2016. 1

    2016

  9. [17]

    Con- strained few-shot class-incremental learning

    Michael Hersche, Geethan Karunaratne, Giovanni Cheru- bini, Luca Benini, Abu Sebastian, and Abbas Rahimi. Con- strained few-shot class-incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9057–9067, 2022. 6

    2022

  10. [18]

    Distilling the knowledge in a neural net- work

    Geoffrey Hinton. Distilling the knowledge in a neural net- work. arXiv preprint arXiv:1503.02531, 2015. 2

    2015 arXiv

  11. [19]

    Learning a unified classifier incrementally via rebalancing

    Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 831–839,

  12. [20]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 1

    2021 arXiv

  13. [21]

    Squeeze-and-excitation net- works

    Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation net- works. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7132–7141, 2018. 1

    2018

  14. [22]

    Neural collapse inspired federated learning with non-iid data.arXiv preprint arXiv:2303.16066,

    Chenxi Huang, Liang Xie, Yibo Yang, Wenxiao Wang, Bin- bin Lin, and Deng Cai. Neural collapse inspired federated learning with non-iid data.arXiv preprint arXiv:2303.16066,

    arXiv

  15. [23]

    An unconstrained layer-peeled perspective on neural collapse

    Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Wei- jie J Su. An unconstrained layer-peeled perspective on neural collapse. arXiv preprint arXiv:2110.02796, 2021. 2

    2021 arXiv

  16. [24]

    Vi- sual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. In European Conference on Computer Vision, pages 709–727. Springer, 2022. 4, 5, 1

    2022

  17. [25]

    Class- incremental learning by knowledge distillation with adaptive feature consolidation

    Minsoo Kang, Jaeyoo Park, and Bohyung Han. Class- incremental learning by knowledge distillation with adaptive feature consolidation. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 16071–16080, 2022. 2

    2022

  18. [26]

    Overcoming catastrophic forgetting in neu- ral networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neu- ral networks. Proceedings of the national academy of sc...

    2017

  19. [27]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 6

    2009

  20. [28]

    Overcoming catastrophic forget- ting by incremental moment matching

    Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak Zhang. Overcoming catastrophic forget- ting by incremental moment matching. Advances in neural information processing systems, 30, 2017. 2

    2017

  21. [29]

    The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691, 2021

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691, 2021. 1

    2021 arXiv

  22. [30]

    Principled and efficient transfer learning of deep models via neural col- lapse

    Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Granda, Zhihui Zhu, and Qing Qu. Principled and efficient transfer learning of deep models via neural col- lapse. arXiv preprint arXiv:2212.12206, 2022. 2

    2022 arXiv

  23. [31]

    Learning without forgetting

    Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelli- gence, 40(12):2935–2947, 2017. 2

    2017

  24. [32]

    Feature pyra- mid networks for object detection

    Tsung-Yi Lin, Piotr Doll ´ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyra- mid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 2117–2125, 2017. 1

    2017

  25. [33]

    Model behavior preserving for class-incremental learning

    Yu Liu, Xiaopeng Hong, Xiaoyu Tao, Songlin Dong, Jin- gang Shi, and Yihong Gong. Model behavior preserving for class-incremental learning. IEEE Transactions on Neural Networks and Learning Systems , 34(10):7529–7540, 2022. 2

    2022

  26. [34]

    Neural collapse under cross-entropy loss

    Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis, 59:224–241, 2022. 2

    2022

  27. [35]

    Premonition: Using generative mod- els to preempt future data changes in continual learning

    Mark D McDonnell, Dong Gong, Ehsan Abbasnejad, and Anton van den Hengel. Premonition: Using generative mod- els to preempt future data changes in continual learning. arXiv preprint arXiv:2403.07356, 2024. 1, 2

    2024 arXiv

  28. [36]

    Neural col- lapse with unconstrained features

    Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural col- lapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20(2):11, 2022. 2

    2022

  29. [37]

    Traces of class/cross-class structure per- vade deep learning spectra

    Vardan Papyan. Traces of class/cross-class structure per- vade deep learning spectra. Journal of Machine Learning Research, 21(252):1–64, 2020. 4

    2020

  30. [38]

    Prevalence of neural collapse during the terminal phase of deep learning training

    Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40):24652–24663, 2020. 2, 3, 4

    2020

  31. [39]

    Explicit regularization and implicit bias in deep network classifiers trained with the square loss

    Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv preprint arXiv:2101.00072, 2020. 2

    2020 arXiv

  32. [40]

    Gdumb: A simple approach that questions our progress in continual learning

    Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 524–540. Springer, 2020. 2

    2020

  33. [41]

    icarl: Incremental classifier and representation learning

    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE con- ference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017. 2, 6

    2001

  34. [42]

    Progressive neural networks

    Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Raz- van Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016. 2

    2016 arXiv

  35. [43]

    Overcoming catastrophic forgetting with hard attention to the task

    Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International conference on machine learning, pages 4548–4557. PMLR, 2018. 2

    2018

  36. [44]

    Coda-prompt: Contin- ual decomposed attention-based prompting for rehearsal-free continual learning

    James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Contin- ual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the I...

    2023

  37. [45]

    Imbalance trouble: Revisiting neural-collapse geometry

    Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. Advances in Neural Information Processing Systems, 35:27225–27238, 2022. 2

    2022

  38. [46]

    Extended unconstrained fea- tures model for exploring deep neural collapse

    Tom Tirer and Joan Bruna. Extended unconstrained fea- tures model for exploring deep neural collapse. In Inter- national Conference on Machine Learning , pages 21478– 21505. PMLR, 2022. 2

    2022

  39. [47]

    Visualizing data using t-sne

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11), 2008. 1

    2008

  40. [48]

    The caltech-ucsd birds-200-2011 dataset

    Catherine Wah, Steve Branson, Peter Welinder, Pietro Per- ona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011. 6

    2011

  41. [49]

    Triple-memory networks: A brain-inspired method for continual learning

    Liyuan Wang, Bo Lei, Qian Li, Hang Su, Jun Zhu, and Yi Zhong. Triple-memory networks: A brain-inspired method for continual learning. IEEE Transactions on Neural Net- works and Learning Systems, 33(5):1925–1934, 2021. 2

    1925

  42. [50]

    Ordisco: Effective and efficient usage of incremental unlabeled data for semi-supervised continual learning

    Liyuan Wang, Kuo Yang, Chongxuan Li, Lanqing Hong, Zhenguo Li, and Jun Zhu. Ordisco: Effective and efficient usage of incremental unlabeled data for semi-supervised continual learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, page...

    2021

  43. [51]

    Coscl: Cooperation of small continual learners is stronger than a big one

    Liyuan Wang, Xingxing Zhang, Qian Li, Jun Zhu, and Yi Zhong. Coscl: Cooperation of small continual learners is stronger than a big one. In European Conference on Com- puter Vision, pages 254–271. Springer, 2022. 2

    2022

  44. [52]

    Memory replay with data compression for continual learning

    Liyuan Wang, Xingxing Zhang, Kuo Yang, Longhui Yu, Chongxuan Li, Lanqing Hong, Shifeng Zhang, Zhen- guo Li, Yi Zhong, and Jun Zhu. Memory replay with data compression for continual learning. arXiv preprint arXiv:2202.06592, 2022. 2

    2022 arXiv

  45. [53]

    Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub- optimality

    Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub- optimality. Advances in Neural Information Processing Sys- tems, 36, 2024. 2

    2024

  46. [54]

    Towards under- standing neural collapse in supervised contrastive learning with the information bottleneck method

    Siwei Wang and Stephanie E Palmer. Towards under- standing neural collapse in supervised contrastive learning with the information bottleneck method. arXiv preprint arXiv:2305.11957, 2023. 2

    2023 arXiv

  47. [55]

    Dualprompt: Complementary prompting for rehearsal-free continual learning

    Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. InEuropean Conference on Computer Vision , pages 631–648. Springer,

  48. [56]

    Learning to prompt for con- tinual learning

    Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jen- nifer Dy, and Tomas Pfister. Learning to prompt for con- tinual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 139–149,

  49. [57]

    On the emergence of simplex symmetry in the final and penultimate layers of neural net- work classifiers

    Stephan Wojtowytsch et al. On the emergence of simplex symmetry in the final and penultimate layers of neural net- work classifiers. arXiv preprint arXiv:2012.05420, 2020. 2

    2012 arXiv

  50. [58]

    Large scale incre- mental learning

    Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incre- mental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 374–382,

  51. [59]

    Neural collapse inspired attraction–repulsion-balanced loss for im- balanced learning

    Liang Xie, Yibo Yang, Deng Cai, and Xiaofei He. Neural collapse inspired attraction–repulsion-balanced loss for im- balanced learning. Neurocomputing, 527:60–70, 2023. 2

    2023

  52. [60]

    Continual object detection via prototypical task correlation guided gating mechanism

    Binbin Yang, Xinchi Deng, Han Shi, Changlin Li, Geng- wei Zhang, Hang Xu, Shen Zhao, Liang Lin, and Xiao- dan Liang. Continual object detection via prototypical task correlation guided gating mechanism. In Proceedings of the IEEE/CVF conference on computer vision and pattern r...

    2022

  53. [61]

    Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing neural collapse in imbalanced learning: Do we really need a learnable clas- sifier at the end of deep neural network? Advances in neural information processing systems, 35:37991–38002, 2022. 2

    2022

  54. [62]

    Sepa- ration and concentration in deep networks

    John Zarka, Florentin Guth, and St ´ephane Mallat. Sepa- ration and concentration in deep networks. arXiv preprint arXiv:2012.10424, 2020. 4

    2012 arXiv

  55. [63]

    Contin- ual learning through synaptic intelligence

    Friedemann Zenke, Ben Poole, and Surya Ganguli. Contin- ual learning through synaptic intelligence. In International conference on machine learning, pages 3987–3995. PMLR,

  56. [64]

    A large-scale study of representation learning with the visual task adaptation benchmark

    Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djo- longa, Andre Susano Pinto, Maxim Neumann, Alexey Doso- vitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv ...

    1910 arXiv

  57. [65]

    Slca: Slow learner with classifier align- ment for continual learning on a pre-trained model

    Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slca: Slow learner with classifier align- ment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19148–19158, 2023. 1

    2023

  58. [66]

    Class-incremental learning via deep model consolidation

    Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafet- tin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Class-incremental learning via deep model consolidation. In Proceedings of the IEEE/CVF winter conference on applica- tions of computer vision, pages 1131–1140, 2020. 2

    2020

  59. [67]

    Benchmarking omni-vision representation through the lens of visual realms

    Yuanhan Zhang, Zhenfei Yin, Jing Shao, and Ziwei Liu. Benchmarking omni-vision representation through the lens of visual realms. In European Conference on Computer Vi- sion, pages 594–611. Springer, 2022. 6

    2022

  60. [68]

    Pyramid scene parsing network

    Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. InPro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2881–2890, 2017. 1

    2017

  61. [69]

    Understanding imbalanced semantic segmentation through neural collapse

    Zhisheng Zhong, Jiequan Cui, Yibo Yang, Xiaoyang Wu, Xi- aojuan Qi, Xiangyu Zhang, and Jiaya Jia. Understanding imbalanced semantic segmentation through neural collapse. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 19550–19560, ...

    2023

  62. [70]

    Revisiting class-incremental learning with pre- trained models: Generalizability and adaptivity are all you need

    Da-Wei Zhou, Zi-Wen Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremental learning with pre- trained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision , pages 1– 21, 2024. 1, 2, 6, 7

    2024

  63. [71]

    Continual learning with pre-trained mod- els: A survey

    Da-Wei Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learning with pre-trained mod- els: A survey. arXiv preprint arXiv:2401.16386, 2024. 2

    2024 arXiv

  64. [72]

    Expandable subspace ensemble for pre-trained model- based class-incremental learning

    Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model- based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23554–23564, 2024. 1, 2, 6, 7

    2024

  65. [73]

    On the optimization landscape of neural col- lapse under mse loss: Global optimality with unconstrained features

    Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural col- lapse under mse loss: Global optimality with unconstrained features. In International Conference on Machine Learning, pages 27179–27202. PMLR, 2022. 2

    2022

  66. [74]

    Melo: Low-rank adaptation is better than fine-tuning for medical image diagnosis

    Yitao Zhu, Zhenrong Shen, Zihao Zhao, Sheng Wang, Xin Wang, Xiangyu Zhao, Dinggang Shen, and Qian Wang. Melo: Low-rank adaptation is better than fine-tuning for medical image diagnosis. In 2024 IEEE International Sym- posium on Biomedical Imaging (ISBI) , pages 1–5. IEEE,

    2024

  67. [75]

    A geometric analysis of neu- ral collapse with unconstrained features

    Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neu- ral collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820–29834, 2021. 2, 4, 5 Enhancing Pre-Trained Model-Based Clas...

    2021

  68. [76]

    Each class includes 100 images

    Experimental Settings In this section, we provide an overview of the datasets em- ployed in our study: CIFAR-100: This dataset comprises 60,000 images span- ning 100 classes, with 50,000 images designated for train- ing and 10,000 for testing. Each class includes 100 images. C...

  69. [77]

    Fine-tuning methods We experimented with four fine-tuning methods for the ViT- B/16 pretrained model: LoRA [74], AdaptFormer [5], VPT Deep, and VPT Shallow [24]. LoRA: The core idea of LoRA [20] is to freeze the weights of the pretrained model and decompose its weight matrices...

  70. [78]

    Visualizations In Fig. 6, we present a t-SNE comparison of feature distri- butions between the Aper and NCPTM-CIL on the CIFAR- 100 test dataset under different incremental tasks, specifi- cally for 30 and 50 classes. Fig. 6a and Fig. 6b illustrate the feature distributions of...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.