Pith. sign in

REVIEW 3 major objections 6 minor 89 references

Continual self-supervised learning still needs large-scale pre-training paradigms; SSL's relative robustness to forgetting is not enough by itself.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 15:44 UTC pith:37DG3PRV

load-bearing objection Solid first dedicated CSSL survey: clean taxonomy, honest protocol critique, and a scaling argument that matches the literature; the SSL-robustness story is synthesis, not new causal proof. the 3 major comments →

arxiv 2607.09785 v1 pith:37DG3PRV submitted 2026-07-08 cs.CV cs.AI

Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Models

Sergi Masip , Alicja Dobrzeniecka , Jonathan Swinnen , Joachim Collin , Bart{\l}omiej Twardowski , Szymon {\L}ukasik , Tinne Tuytelaars This is my paper
classification cs.CV cs.AI
keywords continual self-supervised learningcatastrophic forgettingrepresentation learningvision modelsvision-language modelscontinual pre-trainingstability-plasticityevaluation protocols
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Real systems that must keep learning from unlabeled visual streams cannot wait for labels or full retraining from scratch. This survey argues that continual self-supervised learning (CSSL) is the natural setting for that problem, yet the field lacks consistent evaluation and still treats itself largely as supervised continual learning without labels. The authors show why self-supervised objectives forget less readily than supervised ones: they learn task-agnostic features and sit in smoother, flatter loss landscapes. They then organize existing methods into a single taxonomy of forgetting-mitigation strategies and conclude that progress now requires leaving small benchmarks behind for continual pre-training of large models. A reader who cares about lifelong robots or foundation models that keep adapting without labels will find both a map of the current toolkit and a clear statement of what is still missing.

Core claim

Self-supervised objectives produce more stable, task-agnostic representations and flatter loss landscapes than supervised objectives, so they are intrinsically less prone to catastrophic forgetting; yet this advantage does not solve continual adaptation. Existing CSSL methods still suffer gradual forgetting and plasticity loss, evaluation protocols are inconsistent, and advancing the field requires moving from small-scale benchmarks to continual pre-training paradigms for large-scale systems.

What carries the argument

A unified taxonomy of forgetting-mitigation strategies (distillation, especially projected distillation; replay; weight regularization; architectural isolation; model merging; objective-level adaptation) that organizes the literature and exposes the stability-plasticity trade-off under self-supervised losses.

Load-bearing premise

The claim that SSL's reduced forgetting is mainly caused by task-agnostic features and flatter loss landscapes rests on a limited set of cited empirical studies rather than a controlled analysis that isolates those factors from training length, data volume, or other confounds.

What would settle it

A controlled multi-objective experiment that holds architecture, data volume, and training length fixed while swapping only the loss (supervised versus several SSL families) and measures both representation rank/CKA stability and forgetting; if the robustness gap disappears or is explained by other factors, the paper's explanatory claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This survey reviews continual self-supervised learning (CSSL) for vision, with extensions to vision–language settings. It (i) catalogs training protocols, metrics, and benchmarks and argues that inconsistent evaluation practices impede fair comparison; (ii) synthesizes prior evidence that SSL objectives are relatively robust to catastrophic forgetting, attributing this mainly to task-agnostic representations and smoother/flatter loss landscapes; (iii) organizes existing methods into a six-family taxonomy (distillation, weight regularization, replay, architectural methods, model merging, objective-level adaptation), with additional discussion of SSL as a component in hybrid supervised CL pipelines; and (iv) identifies open challenges—evaluation standardization, the stability–plasticity trade-off, cross-objective generalization, and scaling—and argues that the field should move from small-scale class-incremental benchmarks toward continual pre-training of large-scale systems.

Significance. The paper fills a clear gap: CSSL for vision (and its multimodal extensions) has lacked a dedicated, systematic survey, while related reviews treat it only peripherally. The taxonomy is coherent and usable, the multimodal and hybrid sections are timely, and the scaling argument is well motivated by documented gaps in evaluation, compute, and long-horizon plasticity. Strengths include explicit takeaways per method family, a challenges map (Fig. 2), and a programmatic agenda that is falsifiable in the sense that it points to concrete next experiments (unified protocols, ViT-native objectives under streams, continual foundation-model pre-training). As a literature synthesis rather than a new causal study, its value is organizational and directional; if the field adopts the recommended evaluation and scaling practices, the survey would serve as a useful reference point.

major comments (3)
  1. [Section II / Table I] §II and Table I: The abstract and introduction list “analysis of existing evaluation protocols and highlight[ing of] inconsistencies that hinder fair comparison” as a primary contribution, yet §II-D and Table I only name a handful of datasets with coarse protocol/modality/scale labels. There is no systematic documentation of the concrete variations that actually break comparability (e.g., number of tasks/splits of CIFAR-100 or ImageNet-100, epochs vs. one-pass, linear vs. k-NN vs. fine-tuning probes, presence/absence of task boundaries, reporting of AA vs. F vs. BWT vs. CKA). Without such a comparison, the inconsistency claim is asserted more than demonstrated. A compact protocol-variation table (or expanded Table I) is needed to make this contribution load-bearing.
  2. [Section IV] §IV: The six-family taxonomy is clear in narrative form and Fig. 3, but the survey lacks a consolidated methods table (method, year, family, SSL objective(s), offline vs. online/task-free, replay/distillation/etc. components, main benchmarks). For a taxonomy survey this is a standard reference artifact; its absence reduces the paper’s utility as a lookup resource and makes it harder to verify coverage and family boundaries. Adding such a table would substantially strengthen the central organizational claim without changing the taxonomy itself.
  3. [Section III] §III: The robustness discussion is framed as SSL “stems from two core properties” (task-agnostic features; smoother/flatter landscapes), citing a relatively small set of CKA and landscape studies. As a survey synthesis this is acceptable, but the causal language should be tempered: confounds such as training length, data volume, and batch size are only lightly acknowledged, and no multi-objective controlled isolation is claimed in the cited works. Softening to “working hypotheses supported by prior empirical studies” and explicitly listing confounds would align the epistemic status with the evidence and avoid over-reading §III as a primary mechanistic result. This does not undermine the taxonomy or scaling agenda, which do not depend on that mechanism being uniquely true.
minor comments (6)
  1. [Section II-C] §II-C: Multimodal metrics (AKA, AZS) are introduced briefly; a one-line formal definition or pointer to the exact formulas in [22], [23] would help readers who do not have those papers open.
  2. [Figure 2] Fig. 2 is useful but dense; ensuring that every acronym (PFR, CaSSLe, CLA, POCON, LUMP, etc.) is expanded at first mention in the caption or adjacent text would improve standalone readability.
  3. [Section IV-A] §IV-A takeaway notes the extra forward pass cost of projected distillation; a short quantitative remark (e.g., relative wall-clock or FLOPs where reported in the cited works) would make the efficiency caveat more concrete.
  4. [Section VI-C] §VI-C: The observation that findings may not transfer across contrastive / non-contrastive / MIM families is important; citing the specific CaSSLe/Branch-Tuning/CLA results in a small summary bullet or table would make the “family-specific behaviors” claim easier to reuse.
  5. Minor polish: ensure consistent hyphenation of “self-supervised” / “task-agnostic” / “vision-language” and consistent capitalization of method names (e.g., SwAV vs. SwA V) throughout.
  6. [References] References: a few arXiv-only entries that now have venue versions (if any) could be updated for archival stability; not blocking.

Circularity Check

0 steps flagged

No significant circularity: literature survey with external grounding, no self-referential derivations or fitted-as-prediction steps.

full rationale

This paper is a systematic survey of continual self-supervised learning. Its contributions are organizational (evaluation-protocol critique, method taxonomy by forgetting-mitigation strategy, open-challenge framing) and programmatic (call for continual pre-training at foundation-model scale). It does not claim a first-principles derivation, uniqueness theorem, or quantitative prediction obtained by fitting parameters to data. Section III’s account of SSL robustness (task-agnostic geometry, flatter loss landscapes) is explicitly a synthesis of prior empirical CKA and landscape studies, not a closed-form result forced by the authors’ own definitions. Occasional citations of co-authored related work (e.g., PFR, POCON) appear as ordinary pointers within a broad external literature and do not load-bear any uniqueness claim or redefine the target quantities. There is therefore no self-definitional loop, no fitted input renamed as prediction, and no ansatz smuggled in via self-citation. Circularity score is zero.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

As a survey the paper inherits the empirical and theoretical claims of the cited CSSL literature rather than introducing free parameters or new physical entities. The main load-bearing background assumptions are standard continual-learning and SSL premises (catastrophic forgetting exists, SSL pretext tasks yield useful representations, CKA measures representational similarity). No numerical free parameters are fitted; the taxonomy categories are organizational rather than postulated mechanisms.

axioms (3)
  • domain assumption Self-supervised objectives produce more task-agnostic, higher-rank representations and flatter loss landscapes than supervised cross-entropy, thereby reducing catastrophic forgetting.
    Invoked throughout Section III as the explanatory core; rests on cited empirical studies (CKA analyses, landscape visualizations) rather than a derivation internal to the survey.
  • domain assumption Existing CSSL evaluation protocols differ substantially in task construction, offline/online setting, and probe type, rendering direct numerical comparison unreliable.
    Stated in Section II and used to motivate the call for unified benchmarks; treated as an observed fact of the literature.
  • standard math Joint training on the full data stream remains a valid upper-bound reference for measuring forgetting and plasticity gaps.
    Standard continual-learning convention adopted without re-derivation.

pith-pipeline@v1.1.0-grok45 · 20888 in / 2268 out tokens · 27015 ms · 2026-07-14T15:44:50.026657+00:00 · methodology

0 comments
read the original abstract

Traditionally, continual learning has assumed access to labeled data, yet many real-world applications -- such as lifelong robotics -- require models to adapt continuously from unlabeled streams. This has led to the development of continual self-supervised learning (CSSL), a rapidly growing area that lacks a dedicated, systematic review. In this work, we present a comprehensive survey of CSSL for vision, with connections to emerging vision-language settings. First, we analyze existing evaluation protocols and highlight inconsistencies that hinder fair comparison. We then examine why self-supervised objectives exhibit improved robustness to catastrophic forgetting, relating this to task-agnostic representations and smoother loss landscapes. Next, we organize existing methods into a unified taxonomy based on their forgetting-mitigation strategies, including distillation, replay, regularization, architectural approaches, model merging, and objective-level adaptation. Finally, we identify open challenges such as scalability and the need for fast adaptability. We argue that advancing CSSL requires moving beyond small-scale benchmarks towards continual pre-training paradigms for large-scale systems.

Figures

Figures reproduced from arXiv: 2607.09785 by Alicja Dobrzeniecka, Bart{\l}omiej Twardowski, Joachim Collin, Jonathan Swinnen, Sergi Masip, Szymon {\L}ukasik, Tinne Tuytelaars.

Figure 1
Figure 1. Figure 1: A conceptual visualization of CSSL. The bottom part [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Challenges in CSSL and representative solutions. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

89 extracted references · 11 linked inside Pith

  1. [1]

    Geode: a geographically diverse evaluation dataset for object recognition,

    V . V . Ramaswamyet al., “Geode: a geographically diverse evaluation dataset for object recognition,”Advances in Neural Information Process- ing Systems, vol. 36, pp. 66 127–66 137, 2023

    2023

  2. [2]

    Catastrophic interference in connec- tionist networks: The sequential learning problem,

    M. McCloskey and N. J. Cohen, “Catastrophic interference in connec- tionist networks: The sequential learning problem,” ser. Psychology of Learning and Motivation, G. H. Bower, Ed. Academic Press, 1989, vol. 24, pp. 109–165

    1989

  3. [3]

    Three continual learning scenarios,

    G. M. Van de Ven and A. S. Tolias, “Three continual learning scenarios,” inNeurIPS Continual Learning Workshop, vol. 1, no. 9, 2018, p. 4

    2018

  4. [4]

    Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey ,

    L. Jing and Y . Tian, “ Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey ,”IEEE TPAMI, vol. 43, no. 11, pp. 4037–4058, Nov. 2021

    2021

  5. [5]

    The future of continual learning in the era of foundation models: Three key directions,

    J. Bellet al., “The future of continual learning in the era of foundation models: Three key directions,” 2025. [Online]. Available: https://arxiv.org/abs/2506.03320

    Pith/arXiv arXiv 2025

  6. [6]

    Continual learning of large language models: A compre- hensive survey,

    H. Shiet al., “Continual learning of large language models: A compre- hensive survey,”ACM Comput. Surv., vol. 58, no. 5, Nov. 2025

    2025

  7. [7]

    Continual learning for vlms: A survey and taxonomy beyond forgetting,

    Y . Liuet al., “Continual learning for vlms: A survey and taxonomy beyond forgetting,” 2025. [Online]. Available: https: //arxiv.org/abs/2508.04227

    Pith/arXiv arXiv 2025

  8. [8]

    Beyond supervised continual learning: a review,

    B. Bagus, A. Gepperth, and T. Lesort, “Beyond supervised continual learning: a review,” 2022. [Online]. Available: https: //arxiv.org/abs/2208.14307

    Pith/arXiv arXiv 2022

  9. [9]

    Towards label-efficient incremental learning: A survey,

    M. Kilickaya, J. van de Weijer, and Y . M. Asano, “Towards label-efficient incremental learning: A survey,” 2023. [Online]. Available: https://arxiv.org/abs/2302.00353

    Pith/arXiv arXiv 2023

  10. [10]

    A survey of the self supervised learning mechanisms for vision transformers,

    A. Khanet al., “A survey of the self supervised learning mechanisms for vision transformers,” 2025

    2025

  11. [11]

    A simple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” inInt. Conf. on Machine Learning, 2020, pp. 1597–1607

    2020

  12. [12]

    Momentum contrast for unsupervised visual representation learning,

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2020, pp. 9729–9738

    2020

  13. [13]

    Improved baselines with momentum contrastive learning,

    X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” 2020. [Online]. Available: https://arxiv.org/abs/2003.04297

    Pith/arXiv arXiv 2020

  14. [14]

    Representation learning with contrastive predictive coding,

    A. van den Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” 2019. [Online]. Available: https://arxiv.org/abs/1807.03748

    Pith/arXiv arXiv 2019

  15. [15]

    Bootstrap your own latent-a new approach to self- supervised learning,

    J.-B. Grillet al., “Bootstrap your own latent-a new approach to self- supervised learning,” inAdvances in Neural Information Processing Systems, vol. 33, 2020, pp. 21 271–21 284

    2020

  16. [16]

    Exploring simple siamese representation learning,

    X. Chen and K. He, “Exploring simple siamese representation learning,” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2021, pp. 15 750–15 758

    2021

  17. [17]

    Barlow twins: Self-supervised learning via redundancy reduction,

    J. Zbontar, L. Jing, I. Misra, Y . LeCun, and S. Deny, “Barlow twins: Self-supervised learning via redundancy reduction,” inInt. Conf. on Machine Learning. PMLR, 2021, pp. 12 310–12 320

    2021

  18. [18]

    Vicreg: Variance-invariance- covariance regularization for self-supervised learning,

    A. Bardes, J. Ponce, and Y . LeCun, “Vicreg: Variance-invariance- covariance regularization for self-supervised learning,” inInt. Conf. on Learning Representations, 2022

    2022

  19. [19]

    Unsupervised learning of visual features by contrasting cluster assignments,

    M. Caronet al., “Unsupervised learning of visual features by contrasting cluster assignments,” vol. 33, 2020, pp. 9912–9924

    2020

  20. [20]

    Masked autoencoders are scalable vision learners,

    K. Heet al., “Masked autoencoders are scalable vision learners,” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2022, pp. 16 000–16 009

    2022

  21. [21]

    Cla: Latent alignment for online continual self-supervised learning,

    G. Cignoni, A. Cossu, A. Gomez-Villa, J. van de Weijer, and A. Carta, “Cla: Latent alignment for online continual self-supervised learning,”

  22. [22]

    Available: https://arxiv.org/abs/2507.10434

    [Online]. Available: https://arxiv.org/abs/2507.10434

    Pith/arXiv arXiv

  23. [23]

    A practitioner’s guide to continual multimodal pretraining,

    V . Udandaraoet al., “A practitioner’s guide to continual multimodal pretraining,” inAdvances in Neural Information Processing Systems, 2024

    2024

  24. [24]

    Tic-clip: Continual training of clip models,

    S. Garget al., “Tic-clip: Continual training of clip models,” inInt. Conf. on Learning Representations, 2024

    2024

  25. [25]

    Branch-tuning: balancing stability and plasticity for continual self-supervised learning,

    W. Liu, F. Zhu, and C.-L. Liu, “Branch-tuning: balancing stability and plasticity for continual self-supervised learning,”IEEE Transactions on Neural Networks and Learning Systems, 2025

    2025

  26. [26]

    Similarity of neural network representations revisited,

    S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, “Similarity of neural network representations revisited,” inInt. Conf. on Machine Learning. PMlR, 2019, pp. 3519–3529

    2019

  27. [27]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, “Learning multiple layers of features from tiny images,” pp. 32–33, 2009. [Online]. Available: https://www.cs.toronto.edu/~kriz/ learning-features-2009-TR.pdf

    2009

  28. [28]

    Imagenet: A large-scale hierarchical image database,

    J. Denget al., “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE Conf. on Computer Vision and Pattern Recognition. Ieee, 2009, pp. 248–255

    2009

  29. [29]

    Representational continuity for unsupervised continual learning,

    D. Madaan, J. Yoon, Y . Li, Y . Liu, and S. J. Hwang, “Representational continuity for unsupervised continual learning,” inInt. Conf. on Learning Representations, 2022

    2022

  30. [30]

    Self-supervised class incremental learning,

    Z. Ni, S. Tang, and Y . Zhuang, “Self-supervised class incremental learning,” 2021. [Online]. Available: https://arxiv.org/abs/2111.11208

    Pith/arXiv arXiv 2021

  31. [31]

    Self-supervised models are continual learners,

    E. Finiet al., “Self-supervised models are continual learners,” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2022, pp. 9621–9630

    2022

  32. [32]

    Prevalence of neural collapse during the terminal phase of deep learning training,

    V . Papyan, X. Han, and D. L. Donoho, “Prevalence of neural collapse during the terminal phase of deep learning training,”Proc. of the National Academy of Sciences, vol. 117, no. 40, pp. 24 652–24 663, 2020

    2020

  33. [33]

    Perturbation analysis of neural collapse,

    T. Tirer, H. Huang, and J. Niles-Weed, “Perturbation analysis of neural collapse,” inInt. Conf. on Machine Learning. PMLR, 2023, pp. 34 301– 34 329

    2023

  34. [34]

    Why do better loss functions lead to less transferable features?

    S. Kornblith, T. Chen, H. Lee, and M. Norouzi, “Why do better loss functions lead to less transferable features?”Advances in Neural Information Processing Systems, vol. 34, pp. 28 648–28 662, 2021

    2021

  35. [35]

    The tunnel effect: Building data representations in deep neural networks,

    W. Masarczyket al., “The tunnel effect: Building data representations in deep neural networks,”Advances in Neural Information Processing Systems, vol. 36, pp. 76 772–76 805, 2023

    2023

  36. [36]

    What variables affect out-of-distribution generalization in pretrained models?

    Y . Harun, K. Lee, J. Gallardo, G. Krishnan, and C. Kanan, “What variables affect out-of-distribution generalization in pretrained models?” Advances in Neural Information Processing Systems, vol. 37, pp. 56 479– 56 525, 2024

    2024

  37. [37]

    Know your self-supervised learning: A survey on image-based generative and discriminative training,

    U. Ozbulak,et al., “Know your self-supervised learning: A survey on image-based generative and discriminative training,”Transactions on Machine Learning Research, 2024

    2024

  38. [38]

    Reverse engineering self-supervised learning,

    I. Ben-Shaul, R. Shwartz-Ziv, T. Galanti, S. Dekel, and Y . LeCun, “Reverse engineering self-supervised learning,”Advances in Neural Information Processing Systems, vol. 36, pp. 58 324–58 345, 2023

    2023

  39. [39]

    On the discriminability of self-supervised representation learning,

    Z. Song, W. Qiang, C. Zheng, F. Sun, and H. Xiong, “On the discriminability of self-supervised representation learning,”Information Sciences, p. 122556, 2025

    2025

  40. [40]

    Emerging properties in self-supervised vision trans- formers,

    M. Caronet al., “Emerging properties in self-supervised vision trans- formers,” inProc. of the IEEE/CVF Int. Conf. on Computer Vision, 2021, pp. 9650–9660

    2021

  41. [41]

    Self-supervised learning from images with a joint- embedding predictive architecture,

    M. Assranet al., “Self-supervised learning from images with a joint- embedding predictive architecture,” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2023, pp. 15 619–15 629

    2023

  42. [42]

    Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank,

    Q. Garrido, R. Balestriero, L. Najman, and Y . Lecun, “Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank,” inInt. Conf. on Machine Learning. PMLR, 2023, pp. 10 929–10 974

    2023

  43. [43]

    How well does self-supervised pre-training perform with streaming data?

    D. Huet al., “How well does self-supervised pre-training perform with streaming data?” inInt. Conf. on Learning Representations, 2022

    2022

  44. [44]

    Task agnostic representation consolidation: a self-supervised based continual learning approach,

    P. S. Bhat, B. Zonooz, and E. Arani, “Task agnostic representation consolidation: a self-supervised based continual learning approach,” in Conf. on Lifelong Learning Agents. PMLR, 2022, pp. 390–405

    2022

  45. [45]

    Towards efficient and effective self-supervised learning of visual representations,

    S. Addepalli, K. Bhogale, P. Dey, and R. V . Babu, “Towards efficient and effective self-supervised learning of visual representations,” inEuropean Conf. on Computer Vision. Springer, 2022, pp. 523–538

    2022

  46. [46]

    Accelerating self- supervised learning via efficient training strategies,

    M. T. Koçyi ˘git, T. M. Hospedales, and H. Bilen, “Accelerating self- supervised learning via efficient training strategies,” inProc. of the IEEE/CVF Winter Conf. on Applications of Computer Vision, 2023, pp. 5654–5664

    2023

  47. [47]

    Faster convergence and uncorrelated gradients in self-supervised online continual learning,

    K. Imai, N. Hayashi, T. Hirakawa, T. Yamashita, and H. Fujiyoshi, “Faster convergence and uncorrelated gradients in self-supervised online continual learning,” inProc. of the Asian Conf. on Computer Vision, 2024, pp. 436–453

    2024

  48. [48]

    Continually learning self-supervised representations with pro- jected functional regularization,

    A. Gomez-Villa, B. Twardowski, L. Yu, A. D. Bagdanov, and J. Van de Weijer, “Continually learning self-supervised representations with pro- jected functional regularization,” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2022, pp. 3867–3877

    2022

  49. [49]

    Plasticity-optimized complementary networks for unsupervised continual learning,

    A. Gomez-Villa, B. Twardowski, K. Wang, and J. Van de Weijer, “Plasticity-optimized complementary networks for unsupervised continual learning,” inProc. of the IEEE/CVF Winter Conf. on Applications of Computer Vision, 2024, pp. 1690–1700

    2024

  50. [50]

    Continual pre-training mitigates forgetting in language and vision,

    A. Cossuet al., “Continual pre-training mitigates forgetting in language and vision,”Neural Networks, vol. 179, p. 106492, 2024

    2024

  51. [51]

    Adaptive self-supervised continual learning,

    L. Wu, Z. Wang, and J. Liu, “Adaptive self-supervised continual learning,” inECAI 2023. IOS Press, 2023, pp. 2680–2687

    2023

  52. [52]

    How to merge your multimodal models over time?

    S. Dziadzioet al., “How to merge your multimodal models over time?” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2025, pp. 20 479–20 491

    2025

  53. [53]

    Cromo-mixup: Augmenting cross-model represen- tations for continual self-supervised learning,

    E. Mushtaqet al., “Cromo-mixup: Augmenting cross-model represen- tations for continual self-supervised learning,” inEuropean Conf. on Computer Vision. Springer, 2024, pp. 311–328

    2024

  54. [54]

    Continual self-supervised learning: Towards universal multi- modal medical data representation learning,

    Y . Yeet al., “Continual self-supervised learning: Towards universal multi- modal medical data representation learning,” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2024, pp. 11 114– 11 124

    2024

  55. [55]

    Continual self-supervised learning with masked autoencoders in remote sensing,

    L. Möllenbrok, B. Rasti, and B. Demir, “Continual self-supervised learning with masked autoencoders in remote sensing,”IEEE Geoscience and Remote Sensing Letters, 2025

    2025

  56. [56]

    Generative negative text replay for continual vision- language pretraining,

    S. Yanet al., “Generative negative text replay for continual vision- language pretraining,” inEuropean Conf. on Computer Vision. Springer, 2022, pp. 22–38

    2022

  57. [57]

    Ctp: Towards vision- language continual pretraining via compatible momentum contrast and topology preservation,

    H. Zhu, Y . Wei, X. Liang, C. Zhang, and Y . Zhao, “Ctp: Towards vision- language continual pretraining via compatible momentum contrast and topology preservation,” inProc. of the IEEE/CVF Int. Conf. on Computer Vision, 2023, pp. 22 257–22 267

    2023

  58. [58]

    Continual vision- language representation learning with off-diagonal information,

    Z. Ni, L. Wei, S. Tang, Y . Zhuang, and Q. Tian, “Continual vision- language representation learning with off-diagonal information,” inProc. of the 40th Int. Conf. on Machine Learning. PMLR, 2023, pp. 26 129– 26 149

    2023

  59. [59]

    Continual contrastive learning for image classification,

    Z. Lin, Y . Wang, and H. Lin, “Continual contrastive learning for image classification,” in2022 IEEE International conference on multimedia and expo (ICME). IEEE, 2022, pp. 1–6

    2022

  60. [60]

    Revisiting supervision for continual representation learning,

    D. Marczak, S. Cygert, T. Trzci ´nski, and B. Twardowski, “Revisiting supervision for continual representation learning,” inEuropean Conf. on Computer Vision. Springer, 2024, pp. 181–197

    2024

  61. [61]

    Regularizing with pseudo-negatives for continual self-supervised learning,

    S. Cha, K. Cho, and T. Moon, “Regularizing with pseudo-negatives for continual self-supervised learning,” inInt. Conf. on Machine Learning. PMLR, 2024, pp. 6048–6065

    2024

  62. [62]

    C-clip: Multimodal continual learn- ing for vision-language model,

    W. Liu, F. Zhu, L. Wei, and Q. Tian, “C-clip: Multimodal continual learn- ing for vision-language model,” inInt. Conf. on Learning Representations, 2025

    2025

  63. [63]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatricket al., “Overcoming catastrophic forgetting in neural networks,”Proc. of the national academy of sciences, vol. 114, no. 13, pp. 3521–3526, 2017

    2017

  64. [64]

    Continual learning through synaptic intelligence,

    F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,” inInt. Conf. on Machine Learning. Pmlr, 2017, pp. 3987–3995

    2017

  65. [65]

    Memory aware synapses: Learning what (not) to forget,

    R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, “Memory aware synapses: Learning what (not) to forget,” inProc. of the European Conf on Computer Vision, 2018, pp. 139–154

    2018

  66. [66]

    Are labels needed for incremental instance learning?

    M. Kilickaya and J. Vanschoren, “Are labels needed for incremental instance learning?” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2023, pp. 2401–2409

    2023

  67. [67]

    Continual barlow twins: continual self- supervised learning for remote sensing semantic segmentation,

    V . Marsocci and S. Scardapane, “Continual barlow twins: continual self- supervised learning for remote sensing semantic segmentation,”IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 16, pp. 5049–5060, 2023

    2023

  68. [68]

    Continual retinal vision-language pre-training upon incremental imaging modalities,

    Y . Yao, R. Wu, Y . Zhou, and T. Zhou, “Continual retinal vision-language pre-training upon incremental imaging modalities,” inMedical Image Computing and Computer Assisted Intervention. Springer, 2025, pp. 111–121

    2025

  69. [69]

    Beyond cosine decay: On the effectiveness of infinite learning rate schedule for continual pre-training,

    V . Singhet al., “Beyond cosine decay: On the effectiveness of infinite learning rate schedule for continual pre-training,”Fourth Conf. on Lifelong Learning Agents, 2025

    2025

  70. [70]

    Experience replay for continual learning,

    D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne, “Experience replay for continual learning,”Advances in Neural Information Processing Systems, vol. 32, 2019

    2019

  71. [71]

    mixup: Beyond empirical risk minimization,

    H. Zhang, M. Cisse, Y . N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,”Int. Conf. on Learning Representations, 2018

    2018

  72. [72]

    Contrastive continuity on augmentation stability re- hearsal for continual self-supervised learning,

    H. Chenget al., “Contrastive continuity on augmentation stability re- hearsal for continual self-supervised learning,” inProc. of the IEEE/CVF Int. Conf. on Computer Vision, 2023, pp. 5707–5717

    2023

  73. [73]

    Cucl: Codebook for unsupervised continual learning,

    C. Chenget al., “Cucl: Codebook for unsupervised continual learning,” inProc. of the 31st ACM Int. Conf. on Multimedia, 2023, pp. 1729–1737

    2023

  74. [74]

    Effective data selection and replay for unsupervised continual learning,

    L. Hanmoet al., “Effective data selection and replay for unsupervised continual learning,” inIEEE 40th Int. Conf. on Data Engineering. IEEE, 2024, pp. 1449–1463

    2024

  75. [75]

    Memory storyboard: Leveraging temporal segmentation for streaming self-supervised learning from egocentric videos,

    Y . Yang and M. Ren, “Memory storyboard: Leveraging temporal segmentation for streaming self-supervised learning from egocentric videos,”Fourth Conference on Lifelong Learning Agents, 2025

    2025

  76. [76]

    Unsupervised continual learning via self-adaptive deep clustering approach,

    M. Pratama, A. Ashfahani, and E. Lughofer, “Unsupervised continual learning via self-adaptive deep clustering approach,” inInt. Workshop on Continual Semi-Supervised Learning. Springer, 2021, pp. 48–61

    2021

  77. [77]

    Efficient self-supervised continual learning with progressive task-correlated layer freezing,

    L. Yang, S. Lin, F. Zhang, J. Zhang, and D. Fan, “Efficient self-supervised continual learning with progressive task-correlated layer freezing,” in 26th Int. Symposium on Quality Electronic Design. IEEE, 2025, pp. 1–8

    2025

  78. [78]

    Lora: Low-rank adaptation of large language models,

    E. Huet al., “Lora: Low-rank adaptation of large language models,” in Int. Conf. on Learning Representations, 2022

    2022

  79. [79]

    Learning transferable visual models from natural language supervision,

    A. Radfordet al., “Learning transferable visual models from natural language supervision,”CoRR, vol. 139, pp. 8748–8763, 2021. [Online]. Available: https://arxiv.org/abs/2103.00020

    Pith/arXiv arXiv 2021

  80. [80]

    Replay-free online continual learning with self-supervised multipatches,

    G. Cignoni, A. Cossu, A. Gomez-Villa, J. van de Weijer, and A. Carta, “Replay-free online continual learning with self-supervised multipatches,”

Showing first 80 references.