REVIEW 3 major objections 3 minor 2 cited by
Replay-free Online Continual Learning with Self-Supervised MultiPatches
T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Online continual learning typically relies on replaying stored examples, but this paper presents a multi-patch self-supervised objective that matches or beats replay without storing past data.
desk verdict BYOL-CMP is a strong empirical result, but the paper's 'replay is unnecessary' claim conflates multi-patch augmentation with the absence of replay. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the CMP loss $L_{\text{CMP}} = \beta L_{\text{TCR}}([z_1,\dots,z_N]) + \alpha \sum_{i=1}^N L_{\text{SSL}}(z_i, z_{\text{avg}})$, where $z_i = \theta(x_i)$ are the encoder representations of the $N$ patches from one example and $z_{\text{avg}}$ is their mean. The second term is the base instance-discrimination loss (SimSiam or BYOL) evaluated between each patch and the average; the first term is the total coding rate $L_{\text{TCR}}(Z)=\frac12 \log\det\left(I+\frac{d}{b\epsilon^2}ZZ^\top\right)$, a collapse-prevention regularizer. The mechanism that carries the argument is batch expansion without memory: by replacing the standard two views with $N=20$ patches, CMP raises the effective per-step minibatch from 10 to 200 using only current data, and the stop-gradient or EMA designs inherited from SimSiam/BYOL keep training stable. This is what lets CMP match replay's computational benefit while being replay-free.
What would settle it
Run the same OCSSL streams but report test accuracy separately for each of the 20 class splits after training ends, for BYOL-CMP and for reservoir replay with a 2,000-sample buffer. If early-split accuracy for CMP is substantially lower while final total accuracy is higher, the no-forgetting claim is falsified; the win would come from recent-task dominance.
Extended reading notes
Core claim
The paper's claim is that in a class-incremental online stream, a model trained with CMP—pulling N patch representations of the same image together while preventing collapse—learns representations at least as well as models that replay stored examples. Concretely, CMP is a plug-in loss for instance-discrimination SSL methods: SimSiam-CMP matches each patch's predictor output to the stop-gradient average of all patches, and BYOL-CMP does the same using the EMA-updated encoder's normalized average. With a stream batch of 10 and 20 patches per example, CMP trains on an effective batch of 200, the same minibudget as the replay baselines, yet stores no past samples. In the reported results BYOL-CMP achieves the highest linear-probing accuracy on both benchmarks; SimSiam-CMP beats all ER-based methods except reservoir replay with a 2,000-sample buffer on ImageNet100. The authors conclude that this undermines the assumption that replay is the go-to solution for self-supervised OCL.
Load-bearing premise
The headline comparison rests on final linear-probing accuracy averaged over the whole test set; if early-task performance is quietly overwritten while recent tasks improve, the total score could flatter CMP.
Editorial extensions
If this is right
- Replay becomes optional: under the same per-step compute, a replay-free SSL method can match or beat experience replay, which removes the storage and privacy constraints that motivated the paper.
- CMP is a plug-in: any instance-discrimination SSL method—SimSiam and BYOL in the paper—can be upgraded to OCSSL by adding the multi-patch loss, improving over its two-view fine-tuning baseline.
- Effective batch size is the driver: the authors' hypothesis is that expanding the minibatch with current-data patches is what enables fast adaptation; if so, the benefit should appear for any buffer-free batch-expansion scheme.
- BYOL-CMP's combination of an EMA-updated teacher with multi-patch training produces the largest gains, pointing to the encoder update rule as an important design choice for CMP-style methods.
Reading between the lines
- Editorial inference: since the paper reports only final linear-probing accuracy, the 'mitigates forgetting without replay' conclusion is inferred from end-of-stream totals; computing per-split accuracies or backward transfer would test whether CMP actually preserves earlier classes or simply converges faster on recent ones.
- Editorial inference: CMP's gain is attributed to a larger effective minibatch from current data; if that is the mechanism, other buffer-free batch-expansion tricks—more views per example, higher stream batch sizes, or mixed augmentation strategies—should reproduce similar gains, and that prediction is testable without new storage.
- Editorial inference: because CMP requires only an encoder and an instance-discrimination loss, the recipe should transfer to privacy-sensitive non-visual streams (audio, sensor, or clinical) where keeping past samples is forbidden; a small empirical check would be to run BYOL-CMP on a non-vision stream with the same patch-by-augmentation strategy.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Continual MultiPatches (CMP), a replay-free self-supervised learning method for online continual learning (OCL). For each stream example, CMP extracts N patches, computes an average patch representation, and combines an instance-discrimination SSL loss between each patch and the average with the Total Coding Rate loss to avoid collapse. CMP is applied on top of SimSiam and BYOL, and is evaluated on Split CIFAR-100 and Split ImageNet100 with 10-example streaming minibatches and 20 patches per example, yielding a 200-view batch. The experiments compare against fine-tuning, ER variants (Reservoir/FIFO with buffer sizes 500 and 2000), and EMP-SSL, reporting final linear probing accuracy after the full stream. The authors report that BYOL-CMP is the best-performing method on both benchmarks and claim that CMP surpasses replay-based strategies, challenging the role of replay in self-supervised OCL.
Significance. If the central claim holds, this is a significant result: a replay-free method that matches or exceeds experience replay under the same per-iteration view budget would be directly relevant to privacy-constrained and memory-limited OCL applications. The paper is clearly written, releases code, and evaluates on two standard benchmarks with realistic online mini-batches. However, the significance is conditional on isolating the multi-patch mechanism from the absence of replay, which the current experiments do not do, and on demonstrating that the method actually mitigates forgetting rather than simply optimizing final probing accuracy.
major comments (3)
- [Section 4, Table 1, Baselines paragraph] The headline comparison confounds replay with multi-patch augmentation. CMP fills a 200-view minibatch with 10 stream images x 20 patches each, while the ER baselines fill the same 200-view budget with 10 stream images plus 90 buffer images using the standard two-view augmentation. The comparison therefore changes two variables at once: whether a memory buffer is used and how many views are generated per image. Since multi-patch objectives (EMP-SSL, BagSSL) are known to improve representations independently of replay, the reported advantage of BYOL-CMP (46.3 on ImageNet100 vs. 40.3 for Reservoir ER 2000) may be due to the multi-patch construction rather than to being replay-free. The authors should run ablations that control the number of views per image, for example by giving ER baselines the same 20-patch extraction on their stream and buffer examples, or by adding a buffer to CMP. Without such a control, the conclusion that replay is unnecessary is not supported by the reported results.
- [Abstract, Section 4 Results, Section 5 Conclusion] The abstract and conclusion state that 'CMP surpasses replay' without qualification, but Table 1 shows that SimSiam-CMP on ImageNet100 attains 33.3 ± 0.7, which is well below Reservoir ER 2000 at 39.5 ± 0.5. The Results section acknowledges this exception ('CMP surpasses all ER-based methods, except SimSiam with M size = 2000'), but the abstract and conclusion repeat the unqualified claim. Since the central claim of the paper is precisely this comparison, the exception must be stated prominently rather than buried in the results; otherwise the paper is internally inconsistent.
- [Section 4 evaluation, Section 5 Conclusion] The paper only reports final linear probing accuracy on the full stream and does not report per-split accuracies or any forgetting metric. The conclusion that CMP 'mitigate[s] forgetting without the need for revisiting previous samples' cannot be established from final accuracy alone: a model with high final accuracy can still have forgotten earlier classes if the last classes dominate the linear probe. To support the forgetting claim, the authors should report per-split accuracy curves or an explicit forgetting measure, such as the average per-task accuracy measured at the end of the stream.
minor comments (3)
- [Section 4, first Results paragraph] The sentence 'constraints on buffer availability hinder performance, confirming that CMP enhances fast adaptation in OCSSL scenarios' appears to be a fragment or a leftover from an earlier draft; please rewrite it as a complete sentence that follows from the preceding discussion.
- [Section 3, Eq. (2)] The notation 'LTCR([z1, ..., zN ] = Z)' is awkward; it would be clearer to write LTCR(Z) with Z = [z1, ..., zN ] defined just before the expression.
- [Section 4, hyperparameter selection] Please specify whether the baseline methods (SimSiam, BYOL, EMP-SSL, and the ER variants) were tuned under the same validation protocol and search budget as CMP, since the learning rate and α/β values are selected on a held-out validation set and this could affect the fairness of the comparison.
Circularity Check
No significant circularity: CMP is an empirically evaluated SSL objective and its central claim is an external comparison, not a derivation from its own inputs.
full rationale
The paper proposes a training loss, CMP, defined in Eq. (1) as a combination of a total coding rate term and an instance-discrimination SSL loss. This is a construction, not a derived prediction: the loss is not claimed to follow from first principles or to predict a quantity that was already used in its definition. The multi-patch idea is explicitly credited to BagSSL and EMP-SSL, and the specific applications to SimSiam and BYOL are written out with their respective losses. The central empirical claim, that replay-free CMP can match or surpass replay-based OCSSL, is supported by Table 1, which compares against external baselines (ER, FIFO ER, EMP-SSL, fine-tuning) using held-out linear probing. Hyperparameters (learning rate, alpha, beta) are selected on a 10% validation set, which is standard model selection rather than a fitted parameter being relabeled as a prediction. The paper's self-citations, such as [7], [9], and [15], are used for motivation or background and are not load-bearing for the CMP result. The skeptical concern that CMP uses 20 patches per image while ER baselines use two views per image is a potential confound in the empirical comparison, but it is a question of experimental validity, not circular reasoning. No step in the paper asserts a conclusion that is equivalent to its premises by construction, so no circularity is identified.
Assumptions & free parameters
free parameters (4)
- alpha =
not reported
- beta =
not reported
- number of patches N =
20
- learning rate =
not reported
assumptions (3)
- domain assumption Patches extracted from the same image define a valid equivalence class for representation learning.
- domain assumption Linear probing on the final features is a valid measure of continual learning quality.
- domain assumption ResNet-18 with SGD and the described augmentation protocol is a representative setup for OCSSL.
Cite this review
Pith. "Pith review of Replay-free Online Continual Learning with Self-Supervised MultiPatches." pith.science (2026). https://pith.science/paper/2CXEBZWB
@misc{pith2026250209140,
author = {Pith},
title = {Pith review of: Replay-free Online Continual Learning with Self-Supervised MultiPatches},
year = {2026},
howpublished = {\url{https://pith.science/paper/2CXEBZWB}},
note = {Machine review of arXiv:2502.09140}
}
read the original abstract
Online Continual Learning (OCL) methods train a model on a non-stationary data stream where only a few examples are available at a time, often leveraging replay strategies. However, usage of replay is sometimes forbidden, especially in applications with strict privacy regulations. Therefore, we propose Continual MultiPatches (CMP), an effective plug-in for existing OCL self-supervised learning strategies that avoids the use of replay samples. CMP generates multiple patches from a single example and projects them into a shared feature space, where patches coming from the same example are pushed together without collapsing into a single point. CMP surpasses replay and other SSL-based strategies on OCL streams, challenging the role of replay as a go-to solution for self-supervised OCL.
Figures
Forward citations
Cited by 2 Pith papers
-
CLA: Latent Alignment for Online Continual Self-Supervised Learning
CLA aligns current with past latent representations using an EMA teacher or stored features, and under a fixed backward-pass budget it beats prior online continual SSL methods and can improve early pretraining.
-
Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Models
CSSL for vision is more robust to forgetting than supervised CL due to task-agnostic features and flatter losses, yet still needs better protocols and scaling beyond small benchmarks to foundation-model continual pretraining.
Reference graph
Works this paper leans on
- [1]
- [2]
-
[3]
A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K . Dokania, P. H. S. Torr, and M. Ranzato. On tiny episodic memories in continual learn ing, 2019
work page 2019
-
[4]
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations, 2020
work page 2020
-
[5]
X. Chen and K. He. Exploring simple siamese representati on learning, 2020
work page 2020
-
[6]
Y. Chen, A. Bardes, Z. Li, and Y. LeCun. Bag of image patch e mbedding behind the success of self-supervised learning. arXiv preprint arXiv:2206.08954 , 2022
arXiv 2022
- [7]
-
[8]
E. Fini, V. G. T. da Costa, X. Alameda-Pineda, E. Ricci, K. Alahari, and J. Mairal. Self-supervised models are continual learners, 2022
work page 2022
Show all 17 references
-
[9]
Gomez-Villa, B
A. Gomez-Villa, B. Twardowski, L. Yu, A. D. Bagdanov, and J. van de W eijer. Continually learning self-supervised representations with projected functional regularization, 2022
2022
-
[10]
Grill, F
J.-B. Grill, F. Strub, F. Altch´ e, C. Tallec, P. H. Riche mond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. M unos, and M. Valko. Bootstrap your own latent: A new approach to self-supervise d learning, 2020
2020
-
[11]
J. Gui, T. Chen, J. Zhang, Q. Cao, Z. Sun, H. Luo, and D. Tao . A survey on self-supervised learning: Algorithms, applications, and future trends, 20 23
-
[12]
Z. Mai, R. Li, J. Jeong, D. Quispe, H. Kim, and S. Sanner. O nline continual learning in image classification: An empirical survey, 2021
2021
-
[13]
Purushwalkam, P
S. Purushwalkam, P. Morgado, and A. Gupta. The challeng es of continuous self-supervised learning. In S. Avidan, G. Brostow, M. Ciss´ e, G. M. Farinell a, and T. Hassner, editors, Computer Vision – ECCV 2022 , pages 702–721, Cham, 2022. Springer Nature Switzer- land
2022
-
[14]
Rebuffi, A
S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 2001–2010, 2017
2001
-
[15]
Soutif-Cormerais, A
A. Soutif-Cormerais, A. Carta, A. Cossu, J. Hurtado, H. Hemati, V. Lomonaco, and J. V. de W eijer. A comprehensive empirical evaluation on online c ontinual learning, 2023
2023
-
[16]
S. Tong, Y. Chen, Y. Ma, and Y. Lecun. Emp-ssl: Towards se lf-supervised learning in one training epoch. arXiv preprint arXiv:2304.03977 , 2023
2023 arXiv
-
[17]
X. Yu, Y. Guo, S. Gao, and T. Rosing. Scale: Online self-s upervised lifelong learning without prior knowledge, 2023
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.