{"id":"cb6f8f8b-2b2b-45e1-be18-476cebccd533","arxiv_id":"2507.10434","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CLA aligns current with past latent representations using an EMA teacher or stored features, and under a fixed backward-pass budget it beats prior online continual SSL methods and can improve early pretraining.","lead":"This paper introduces CLA, a training method for self-supervised learning on streaming data that aligns new representations with a moving average of past ones, either from an EMA network or from stored features. The authors report that CLA outperforms existing continual learning strategies and sometimes beats standard offline training when compute is equalized with a new metric, CBP.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own SimCLR results contradict the unqualified 'CLA surpasses i.i.d.' claim; the conclusion needs scoping to SimSiam and to low-CBP settings, with the CBP forward-pass issue as a secondary concern.","rationale":"The reader correctly identified CBP as a potential fairness issue, and the paper does acknowledge that CBP ignores forward-pass overhead. However, the per-sample-update part of the reader's reasoning is not accurate: in the reported high-CBP setup, i.i.d. gives each sample roughly 42 gradient updates (42 epochs at b=138), whereas a CLA replay sample receives about 3 stream passes plus an expected ~12.8 buffer draws, so CLA does not perform more updates per sample than i.i.d. The more decisive and directly observable problem is the paper's own SimCLR appendix: for a standard contrastive SSL method, CLA does not surpass i.i.d. at high CBP. This is an internal inconsistency between the unqualified conclusion and the reported tables, not a matter of external consensus. The conditional verdict remains appropriate because the method still shows meaningful gains for SimSiam and on low-CBP budgets, but the conclusion must be scoped and the CBP metric should be supplemented with wall-clock or FLOPs measurements. I credit the authors for releasing code, using multiple seeds, and reporting standard deviations, which makes the contradiction checkable rather than speculative.","tokens_in":22581,"tokens_out":10067,"duration_ms":116047,"concrete_test":"Run the authors' released code to reproduce Table 5 (SimCLR, High CBP) and compute the paired per-seed difference (CLA minus i.i.d.) for final accuracy. If the difference is negative on ImageNet100, as reported (-3.4 for CLA-E), then the conclusion 'CLA surpasses i.i.d.' must be revised to name SimSiam and the low-CBP setting, or the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim as stated in the conclusion ('CLA surpasses existing methods, and even i.i.d. training, under a constrained computational budget') is not supported by the paper's full empirical record. In the SimCLR High-CBP experiments (Table 5), i.i.d. remains ahead of both CLA variants: on ImageNet100, i.i.d. reaches 48.5±0.7 versus 45.1±0.2 for the best CLA variant (CLA-E), and on CIFAR-100, i.i.d. reaches 42.9±0.3 versus 42.6±0.3 for CLA-R. The appendix itself notes that SimCLR i.i.d. baselines are 'unbeaten by methods trained on the OCSSL stream.' The conclusion does not restrict the claim to SimSiam or to low-CBP regimes, so the headline is overgeneralized. The CBP metric's exclusion of forward passes is a real secondary issue, especially for CLA-E's extra EMA forward pass, but it is not the decisive problem: CLA-R, which has no extra forward pass, also beats i.i.d. under SimSiam, so the CBP critique alone would not invalidate that part of the result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Continual Latent Alignment (CLA), a replay-based strategy for online continual self-supervised learning (OCSSL) that aligns current representations with representations from the recent past, supplied either by an exponential moving average (EMA) network (CLA-E) or by stored buffer features (CLA-R). A new budget metric, Cumulative Backward Passes (CBP), is proposed to compare methods under a fixed computational budget, and experiments on Split CIFAR-100 and Split ImageNet100 with SimSiam (and SimCLR in the appendix) claim that CLA surpasses existing OCSSL methods and, in some settings, even i.i.d. training at the same CBP. The paper also reports that CLA pretraining can serve as a more efficient early-stage initialization than i.i.d. pretraining. The manuscript includes ablations on buffer type, EMA parameter tau, regularization strength omega, and promises code release.","tokens_in":22812,"tokens_out":9422,"duration_ms":99833,"significance":"If the central claims are made precise and the comparison is properly controlled, the paper makes a useful contribution: it demonstrates that a replay-based alignment loss can accelerate early SSL training and provides a strong baseline for the understudied OCSSL scenario. The CBP metric, despite its limitations, is a constructive step toward standardizing compute-aware comparisons in online continual learning. The empirical protocol is largely careful: two datasets, two SSL backbones, matched CBP across methods, ablations, and multiple seeds. However, the headline claims in the abstract and conclusion are substantially stronger than what the full empirical record supports, and the CBP metric has a load-bearing fairness issue that needs to be addressed. With appropriate scoping and additional control experiments, the paper's core phenomenon is plausible and worth publishing.","major_comments":[{"comment":"The claims \"CLA surpasses existing methods, and even i.i.d. training\" (Section 8) and \"using CLA as a pretraining protocol in the early stages of pretraining leads to a better final performance\" (Abstract) are not supported by the full empirical record. In the SimCLR experiments (Table 5, Appendix B.1), i.i.d. reaches 48.5 ± 0.7 on ImageNet100 versus 45.1 ± 0.2 for CLA-E and 44.2 ± 0.4 for CLA-R, and 42.9 ± 0.3 on CIFAR-100 versus 42.6 ± 0.3 for CLA-R; the appendix itself states that SimCLR i.i.d. baselines are \"unbeaten by methods trained on the OCSSL stream.\" Likewise, Figure 8 shows that on CIFAR-100 with SimCLR, CLA pretraining only reaches, not surpasses, full i.i.d. training. The abstract and conclusion should be revised to restrict these claims to the SimSiam backbone and the specific CBP budgets/datasets where they hold, and to acknowledge the SimCLR counterexamples explicitly.","section":"Abstract and Section 8 (Conclusion)"},{"comment":"The sentence \"In Low CBP, again, CLA-E surpasses other strategies in both datasets\" is contradicted by Table 2: on Split CIFAR-100 with b = 30, np = 1, CaSSLe-R reaches 26.6 ± 0.5 final accuracy, while CLA-E reaches 24.4 ± 1.3 and CLA-R 20.8 ± 0.8. The text also states that \"CLA-R being comparable to CLA-E\" in this setting, which is not the case on CIFAR-100. Please correct this summary and qualify the state-of-the-art claim to the configurations in which it actually holds.","section":"Section 7 (Results) and Table 2"},{"comment":"The fairness of the CBP budget is the load-bearing premise for the comparison with i.i.d. training. CBP counts only backward passes: it does not count forward passes (so CLA-E's additional EMA forward pass is free), and it does not account for the number of gradient updates each unique sample receives. Because CLA's replay buffer samples are repeatedly drawn into the minibatch, they receive far more optimizer updates than any sample in the i.i.d. baseline (which sees each sample exactly nepochs times, Eq. (6)). The paper itself identifies \"the presence of replay, as more training iterations are performed on buffer samples\" as a cause of the i.i.d. improvement in Section 7. This means the headline \"CLA surpasses i.i.d. under the same computational budget\" may be an artifact of the budget definition rather than a property of the alignment loss. Please either provide a control experiment that gives the i.i.d. baseline a comparable number of gradient updates per sample (for example, by allowing it to revisit the full stream more often), or reformulate the claim to a \"backward-pass budget\" without implying equal sample-level training.","section":"Section 5, Eq. (4), and Section 7"},{"comment":"The comparison to prior methods is not fully controlled: the regularization weight ω of CLA is grid-searched on the validation set in every setting (Appendix C), while the baselines inherit all hyperparameters except learning rate from their original implementations. This asymmetry can inflate the reported margin of CLA over methods such as SCALE and Osiris-R. I ask for either a matched hyperparameter search for the baselines in this OCSSL scenario, or a sensitivity analysis showing that the relative ordering is stable when baseline hyperparameters are re-tuned.","section":"Section 6 and Appendix C"}],"minor_comments":[{"comment":"The caption reads \"CBP = for ImageNet100\" with the value missing; the corresponding budget should be filled in (presumably 690k).","section":"Table 6 caption"},{"comment":"The ceiling function in Eq. (6) is typeset as \"l ... m\"; please use proper ceiling brackets and define the notation.","section":"Equation (6)"},{"comment":"The caption states \"CLA outperforms i.i.d. on the FINAL ACCURACY\" without noting that this holds for the SimSiam backbone and specific CBP settings; please add the qualifier to match the rest of the paper.","section":"Figure 1 caption"},{"comment":"In the pseudocode, the variable names for the replay targets are introduced as z* in the text and as z1, z2 in the algorithm description; please unify the notation (e.g., use zr* consistently).","section":"Section 4, Algorithm 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is accepted at CoLLAs 2025 and the authors appear to have run a careful set of experiments. The main issues are scope overclaiming and the fairness of the CBP metric; both are fixable with textual revision and additional control experiments, but the latter may require non-trivial new runs. The hyperparameter-tuning asymmetry is also worth raising, but it is less central to the i.i.d. comparison. I would not reject, but the current version overstates the generality of the results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a real, careful empirical study of online continual self-supervised learning, a genuinely understudied setting. The specific strategy—alignment losses applied only to replayed features, using an EMA target or stored features, with a FIFO buffer—is new in this combination. The paper also introduces CBP, a backward-pass budget measure, which is a useful conceptual contribution even though it is imperfect. The experimental work is solid: two datasets, two SSL backbones, ablations on buffer, tau, omega, and the code is available. The authors deserve credit for running the right control experiments, including comparing to i.i.d. under the same budget.\n\nThe soft spots are real but not fatal. The abstract and conclusion claim CLA 'surpasses i.i.d. training', but the paper's own SimCLR experiments contradict that: in High-CBP, i.i.d. is ahead of both CLA variants on both datasets (48.5 vs 45.1 on ImageNet100; 42.9 vs 42.6 on CIFAR-100). The conclusion needs to be scoped to SimSiam, or at least acknowledge the backbone dependence. Second, CBP counts backward passes but ignores forward passes (CLA-E has an extra EMA forward pass) and, more importantly, replay-based methods give many more gradient updates to buffer samples than i.i.d. gives to any sample. The paper itself cites this as an explanation for the i.i.d. advantage, so the headline comparison is partly manufactured by the metric. Third, CLA's omega is grid-searched on the validation set while baseline hyperparameters are fixed from the original papers. That asymmetry is an advantage to CLA, though the effect is likely modest.\n\nThese are addressable. The central mechanism—alignment on replay improves convergence in a single-pass stream—holds up under SimSiam and partially under SimCLR. The CBP metric, while imperfect, is a step forward for comparing methods in this setting. This paper deserves a serious referee. I'd conditionally accept, with the authors required to soften the i.i.d. claim and do a fairer hyperparameter comparison. I'd bring it to the reading group; the discussion about what counts as a fair budget in online SSL is genuinely interesting.","headline":"Solid empirical OCSSL paper with a genuinely new combination, but the 'beats i.i.d.' claim holds only for SimSiam and is partly an artifact of the budget metric.","tokens_in":23373,"tokens_out":2646,"would_cite":true,"duration_ms":27190,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Online self-supervised learning beats i.i.d. training at equal compute, via latent alignment","keywords":["online continual learning","self-supervised learning","latent alignment","replay buffer","exponential moving average","computational budget","catastrophic forgetting","representation learning"],"falsifier":"A controlled experiment that measures wall-clock time or FLOPs for each method at the same CBP on identical hardware, then re-plots accuracy versus actual training time; if the gap between CLA and i.i.d. shrinks or reverses when forward passes and per-sample update counts are accounted for, the headline claim is an artifact of the CBP metric.","tokens_in":22356,"feed_emoji":"🧠","tokens_out":3311,"duration_ms":35002,"temperature":0.7,"pith_summary":"This paper tackles online continual self-supervised learning, where unlabeled data arrives as a one-pass minibatch stream with no task boundaries and a strict compute budget. The authors propose Continual Latent Alignment (CLA), a regularizer that pulls the current encoder's representations toward representations from the recent past, obtained either from an exponential-moving-average encoder (CLA-E) or from a replay buffer (CLA-R). The central claim is that, under a budget measured by cumulative backward passes (CBP), CLA surpasses existing continual-SSL methods and even beats standard i.i.d. training in final accuracy. A second, more surprising claim is that using CLA for the early phase of pretraining, then continuing with i.i.d. data, yields a better final model than full i.i.d. pretraining at the same total budget.","feed_headline":"Online self-supervised learning beats i.i.d. at equal compute","feed_subtitle":"A replay-based alignment regularizer makes online SSL converge fast enough to beat offline training under the same backward-pass budget.","key_machinery":"The central object is the alignment regularizer L_reg = -SC(a_phi(z_r), z_hat)/2, a negative cosine similarity that pushes current representations of replayed samples, after a projection head a_phi, toward past target features. Two target sources are proposed: an exponential moving average network theta' = tau theta' + (1-tau) theta for CLA-E, and stored past features z* in a FIFO buffer for CLA-R. The alignment loss is added to the base SSL loss, so the model keeps learning on fresh stream data while being stabilized by past representations, which the paper argues yields fast convergence and mitigates forgetting without needing task boundaries.","core_discovery":"The central discovery is that a lightweight alignment loss makes an online SSL learner converge much faster than plain SSL training, to the point where it overtakes the i.i.d. upper bound at the same computational budget. The paper formalizes the budget as the total number of backward passes, CBP = nv x nsteps x b, and reports results on Split CIFAR-100 and Split ImageNet100 with both SimSiam and SimCLR backbones. At the high budget, CLA-E reaches 43.1% final probing accuracy on CIFAR-100 versus 39.9% for i.i.d., and CLA variants also dominate average accuracy, keeping performance high across the whole stream. The paper further shows that a CLA-pretrained checkpoint, followed by ordinary i.i.d. training, matches or exceeds full i.i.d. training, which it interprets as evidence that CLA accelerates the early, convergence-dominated phase of self-supervised pretraining.","pith_inferences":["The i.i.d. comparison is exact only if backward passes are the dominant cost; the authors note CLA-E adds an EMA forward pass and SCALE copies the encoder. If forward-pass cost is material, the 'same budget' claim is approximate, though their wall-clock plots show modest differences.","The paper's own stated explanation for the i.i.d. gap is that replay methods perform many more gradient updates on buffer samples than i.i.d. does on any given sample, and that alignment adds fast adaptation. That suggests the headline result may owe part of its size to the non-i.i.d. update distribution rather than to alignment alone.","A testable extension is whether the early-pretraining benefit persists at larger scale, for example with full ImageNet or transformer backbones, where i.i.d. convergence dynamics differ from ResNet-18.","The finding hints at a broader principle: a small replay buffer plus feature alignment acts as a variance-reduction mechanism for optimization on non-stationary streams, potentially connecting to momentum or preconditioning ideas beyond continual learning."],"forward_implications":["An online continual strategy can match or beat the i.i.d. upper bound when the budget is measured in backward passes, so the online stream is not inherently a handicap under small budgets.","CLA checkpoints are reusable warm starts: continuing with i.i.d. data from a CLA-pretrained model reaches the same accuracy as a full i.i.d. run, which is a concrete way to shorten SSL pretraining.","The FIFO buffer beating reservoir and MinRed buffers indicates that uniform iteration counts per sample matter more than unbiased stream sampling in the online setting.","CLA supports larger learning rates than plain replay baselines, suggesting the alignment term stabilizes optimization, which could transfer to other budget-limited training regimes."],"supporting_citations":[{"why":"Provides the SimCLR contrastive SSL method used in the repeat experiments and the observation that SSL benefits from larger minibatch sizes.","marker":"Chen et al. 2020a"},{"why":"Defines SimSiam, the base SSL method used for the main CLA experiments.","marker":"Chen & He 2021"},{"why":"CaSSLe, a distillation-based CSSL method whose loss formulation is generalized to derive CLA's alignment loss.","marker":"Fini et al. 2022"},{"why":"PFR, the other task-boundary distillation approach whose negative-cosine alignment CLA adapts for the online setting.","marker":"Gomez-Villa et al. 2022"},{"why":"Experience replay baseline with a reservoir buffer, a key comparison and the origin of the memory-replay component.","marker":"Chaudhry et al. 2019"},{"why":"MinRed, a replay-based CSSL method used both as a baseline and as one of the buffer choices in the ablation.","marker":"Purushwalkam et al. 2022"},{"why":"Provides the theoretical grounding that EMA-based distillation reduces feature drift and improves stability in online continual learning.","marker":"Michel et al. 2024"}],"fun_headline_variants":["CLA beats i.i.d. in self-supervised learning at equal compute","Online SSL with alignment loss outlearns i.i.d. under same budget","Latent alignment pushes online SSL past offline training","CLA speeds SSL convergence, beating i.i.d. at fixed compute","CLA pretraining early beats full i.i.d. pretraining later"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that counting backward passes is a fair way to measure computational budget: if two methods have the same CBP, they are assumed to be compared equally, even though CLA-E has an extra EMA forward pass and replay methods perform many more gradient updates on buffer samples than i.i.d. training does on any one sample.","fun_headline_variants_meta":{"raw":{"variants":["CLA beats i.i.d. in self-supervised learning at equal compute","Online SSL with alignment loss outlearns i.i.d. under same budget","Latent alignment pushes online SSL past offline training","CLA speeds SSL convergence, beating i.i.d. at fixed compute","CLA pretraining early beats full i.i.d. pretraining later"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001024,"raw_usage":{"total_tokens":4282,"prompt_tokens":872,"completion_tokens":3410,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":3320}},"tokens_in":488,"tokens_out":3410,"duration_ms":26396,"temperature":1.0,"reasoning_tokens":3320,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:31:43.672017+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled experiment that measures wall-clock time or FLOPs for each method at the same CBP on identical hardware, then re-plots accuracy versus actual training time; if the gap between CLA and i.i.d. shrinks or reverses when forward passes and per-sample update counts are accounted for, the headline claim is an artifact of the CBP metric.","supporting_citations":[{"cited_title":"The challenges of continuous self-supervised learning","cited_arxiv_id":null,"evidence_quote":"MinRed, a replay-based CSSL method used both as a baseline and as one of the buffer choices in the ablation."},{"cited_title":"Rethinking momentum knowledge distillation in online continual learning","cited_arxiv_id":null,"evidence_quote":"Provides the theoretical grounding that EMA-based distillation reduces feature drift and improves stability in online continual learning."}],"review_version":1}