Pith. sign in

REVIEW 4 major objections 5 minor 65 references

AlphaWiSE claims that two frozen continual-learning checkpoints can be composed post-hoc through per-tensor weight interpolation, yielding a single retrieval model that outperforms either checkpoint alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 00:09 UTC pith:SLTHSWQB

load-bearing objection Plausible post-hoc fusion method with a clearly specified mechanism, but the paper's central claim of consistent gains over standard weight interpolation is contradicted by its own Table 1. the 4 major comments →

arxiv 2607.15094 v1 pith:SLTHSWQB submitted 2026-07-16 cs.CV cs.LG

AlphaWiSE: Adaptive Weight Interpolation for Continual Multimodal Representation Learning

Sarthak Jain , Qiran Hu , Zhen Zhu , Yaoyao Liu This is my paper
classification cs.CV cs.LG
keywords continual learningweight interpolationmultimodal retrievalCLIPAudioCLIPcheckpoint mergingexemplar memorycatastrophic forgetting
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that continual multimodal retrieval does not have to commit to one stability–plasticity trade-off. Instead, after training, two frozen checkpoints of the same AudioCLIP model—one from sequential fine-tuning and one from a stability-preserving method such as EWC, LwF, or iCaRL—can be merged by learning one interpolation coefficient per aligned parameter tensor on a small exemplar memory. The fused checkpoint keeps the same architecture and inference cost while improving audio–image–text retrieval over both individual baselines and global weight interpolation. If correct, this reframes continual-learning checkpoints as reusable building blocks rather than competing final answers, and gives a post-hoc way to recover cross-modal alignment without retraining.

Core claim

For each named parameter tensor p shared by two compatible checkpoints θ_un and θ_reg, AlphaWiSE defines θ̃_p = σ(β_p) θ_un_p + (1−σ(β_p)) θ_reg_p, initializing all coefficients at 0.5 and fitting the logits β on an exemplar memory with a directed InfoNCE loss summed over retrieval directions. Because α_p is a per-tensor scalar, different layers and modality towers can draw different proportions from the plastic and stable endpoints. The paper reports that the materialized interpolated checkpoint consistently outperforms both individual continual-learning baselines and standard global-coefficient WiSE-FT on AudioSet with 840 exemplars, with gains up to +0.053 R@1 for image-to-text retrieval

What carries the argument

Tensor-wise sigmoid interpolation. Each aligned parameter tensor gets its own coefficient α_p = σ(β_p) ∈ (0,1), applied entry-wise as θ̃_p = α_p θ_un_p + (1−α_p) θ_reg_p. The coefficient logits are fitted by minimizing the average directed InfoNCE retrieval loss over an exemplar memory while both endpoints stay frozen; the gradient with respect to β_p is α_p(1−α_p) times the inner product between the retrieval-loss gradient and the tensor difference, so each coefficient moves along a local descent direction and encodes an effective per-tensor retention preference for the stable endpoint.

Load-bearing premise

The central premise is that two continual-learning checkpoints of the same multimodal model are linearly mode-connected enough that a convex per-tensor average can be fitted on a small exemplar memory and still transfer to the full candidate pool; if the endpoints sit in different basins, or if the memory is unrepresentative, the learned coefficients need not help.

What would settle it

Train the two endpoints on a continual audio-image-text benchmark, fit AlphaWiSE coefficients on one split of the exemplar memory, and evaluate on a disjoint held-out candidate pool; if the fused checkpoint does not beat the better endpoint on that held-out pool—or if coefficients refit on a different memory draw change retrieval results by a large margin—the linear-mode-connectivity premise is falsified. A simpler check: compute the retrieval loss along the line segment between θ_un and θ_reg; if the loss is not approximately convex along that line, per-tensor convex interpolation cannot reli

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A single fused checkpoint delivers the same architecture and inference latency as either source, so the gains come at no additional deployment cost.
  • Composing a plastic sequential checkpoint with a stability-preserving EWC, LwF, or iCaRL endpoint recovers cross-modal alignment on retrieval directions that individual methods sacrifice.
  • The best endpoint pairing depends on the retrieval direction and metric, so practitioners can choose which checkpoints to fuse based on the target modality pair.
  • Jointly optimizing coefficients over A→T, I→A, and I→T gives the strongest audio-related results, while optimizing A→T alone best helps image-text retrieval in this setting.
  • Learned per-tensor coefficients reveal a division of labor: weight matrices stay near the stable model while normalization and bias parameters absorb the fine-tuned update.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • AlphaWiSE's formulation extends naturally to more than two checkpoints: if multiple continual-learning trajectories are available, per-tensor coefficients could be learned over a simplex of endpoints, potentially covering more modality trade-offs.
  • Because the coefficients are fit on an exemplar memory that also serves the candidate pool, the method implicitly assumes that memory is representative; a stress test with disjoint or out-of-distribution exemplars would reveal how brittle the transfer is.
  • The observation that norm and bias parameters carry most of the adaptation suggests a cheaper variant: fix weight-matrix coefficients at conservative values and only fit normalization and bias coefficients, which may need far fewer exemplars.
  • The cross-objective asymmetry (objectives involving the image modality help other retrieval directions) hints that AlphaWiSE could be used as a probe to measure which modality anchors the shared embedding space in a continual learner.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AlphaWiSE, a post-hoc weight-space interpolation method for continual multimodal retrieval with AudioCLIP. Given two frozen checkpoints—one from sequential fine-tuning and one from a stability-oriented continual-learning method (EWC, LwF, or iCaRL)—AlphaWiSE fits one scalar interpolation coefficient per parameter tensor, with all coefficients optimized on a small exemplar memory using a directed InfoNCE retrieval loss. The fused checkpoint has the same architecture and inference cost as either endpoint. Experiments on audio-image-text retrieval over AudioSet phases report average and last-phase R@1/mAP for three retrieval directions, plus a cross-objective ablation and qualitative analyses, and the paper claims consistent improvements over both individual continual-learning baselines and standard weight-interpolation methods such as WiSE-FT.

Significance. The post-hoc, frozen-checkpoint fusion framework is appealing: it adds no inference overhead, keeps the coefficient search low-dimensional (499 scalar logits for AudioCLIP ViT-B/32), and directly addresses the observation that different continual-learning trajectories preserve different modality-pair alignments. If the empirical claims are substantiated, the method would be a practical model-composition tool for continual multimodal systems. However, the current evidence is not strong enough to support the headline claims: the paper reports single-run point estimates, omits the coefficient-fitting protocol, does not clarify whether evaluation is disjoint from the exemplar memory used for fitting, and its own Table 1 contains counterexamples to the claimed consistent superiority over WiSE-FT.

major comments (4)
  1. [Abstract; §1; Table 1] The central claim that AlphaWiSE 'consistently outperforms ... standard weight-interpolation methods' is contradicted by the paper's own results. In Table 1, AlphaWiSE (N+EWC) has I→T average R@1 0.2304 versus 0.2330 for WiSE-FT (N+EWC), and last-phase I→T R@1 0.1842 versus 0.2298 — a 4.6-point drop. AlphaWiSE (N+iCaRL) also underperforms WiSE-FT (N+iCaRL) on average I→T R@1 (0.2725 vs 0.2737). The text in §4.2 later concedes that the best pairing depends on metric, stage, and direction, but no selection rule is provided, so the 'consistent improvement' claim is not actionable. Please revise the claims to describe the actual pattern, or provide a protocol for choosing among pairings and verify that AlphaWiSE improves over the corresponding WiSE-FT for that protocol.
  2. [§3, Algorithm 1; §4.1] The coefficient-fitting procedure is not reproducible. Algorithm 1 depends on a batch size B, an optimizer Opt, a number of steps T, and a learning rate, but Section 4.1 reports none of these for the β optimization. It also does not specify the initialization/optimizer for the three 'logit-scale parameters' mentioned in §1, nor whether the sigmoid coefficients are fit with early stopping, weight decay, or gradient clipping. These values are load-bearing because the entire method is the fit of β on the exemplar memory; without them, readers cannot reconstruct Table 1.
  3. [§3, Eq. (3); §4.1] The paper does not state whether the retrieval evaluation is disjoint from the 840-exemplar memory used to fit the interpolation coefficients. If the test queries/candidates or retrieval pool include exemplars from M, then Eq. (3) is minimized on the evaluation set and the reported improvements are circular. Please specify the exact data split: which samples form the retrieval gallery/query set, whether exemplars appear in evaluation, and how the 79-class candidate pool relates to the eight training phases and to M.
  4. [Table 1; Table 2] All results are single point estimates with no error bars, multiple seeds, or significance tests. Several reported differences are very small (e.g., AlphaWiSE (N+iCaRL) vs WiSE-FT (N+iCaRL) in average I→T R@1 differs by 0.0012; Table 2 joint vs A→T-only I→T mAP differs by 0.0088). Without variance estimates, the reader cannot determine whether the claimed improvements are systematic or noise. Please report at least 3 seeds (or confidence intervals) for the main tables, or justify why a single run is sufficient in this setting.
minor comments (5)
  1. [§1] Typo: 'cweight-space interpolation' should be 'weight-space interpolation.'
  2. [§4.2] Inconsistent capitalization/spelling: 'WiseFT(N + iCARL)' should be 'WiSE-FT (N+iCaRL)' to match Table 1 and other rows.
  3. [Appendix A.4, Eq. (4)] The inner-product notation in Eq. (4) is garbled in the PDF (appears as '⣨ ... ⣩'). Please typeset the Frobenius inner product cleanly, e.g., '⟨A, B⟩_F = Σᵢⱼ Aᵢⱼ Bᵢⱼ.'
  4. [§4.5, Figure 5] The caption is long and contains a line break in 'α'; also the figure's legend is duplicated in the caption text. Please simplify and ensure all labels are readable.
  5. [§4.4, Figures 3–4] The qualitative t-SNE comparison is presented as supporting evidence, but only selected classes and samples are shown. It would be helpful to state explicitly that these are illustrative and to quantify the claimed clustering improvement if used as evidence.

Circularity Check

0 steps flagged

No significant circularity: fitted coefficients are evaluated on a separate retrieval candidate pool, and self-citations are not load-bearing.

full rationale

The paper's derivation chain is self-contained with respect to circularity. AlphaWiSE fits per-tensor interpolation coefficients β by minimizing the exemplar-memory retrieval loss of Eq. 3 (Eq. 1 defines the interpolation; Eq. 2 defines the directed InfoNCE loss). The reported results are retrieval metrics (R@1, mAP) on the 79-class candidate pool (Figure 2, Table 1), not on the exemplar memory used for fitting. No equation or passage states or implies that the evaluation set is the same as the coefficient-fitting set, so the reported gains do not reduce to the training loss by construction. The only mathematically explicit relationship between the fitted objective and the evaluation is the gradient identity in Eq. 4, which is a local interpretation and not a claim that the evaluation is the fitting objective. The paper also contains an explicit limitation in Appendix A.3 that the cross-objective analysis is 'a local first-order interpretation' and does not 'establish a causal transfer mechanism'; this is a caveat, not a circular step. Self-citations appear in the related work and references (e.g., [63], [64], [65], and earlier CL papers by the same authors), but none is load-bearing for the central interpolation claim: the method explicitly builds on WiSE-FT [53], an external result, and the evaluation uses the external AudioSet benchmark. No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via a self-citation. The skeptical observation that Table 1 contains counterexamples to the phrase 'consistently outperforms' is an internal consistency/correctness issue, not a circularity issue, because the claim is not derived from its own inputs. Overall, the derivation is an empirical fitting-plus-evaluation procedure with independent external evaluation, and no circular step is identifiable from the paper's own equations or citations.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claim rests on per-tensor interpolation coefficients and three logit scales fitted on the exemplar memory, hand-set source-checkpoint regularization strengths, linear interpolability of CL checkpoints, representativeness of the 840-exemplar memory, and validity of the BN→GroupNorm replacement. No new physical entities are introduced; the method is empirical and self-contained against external AudioSet retrieval benchmarks.

free parameters (3)
  • Per-tensor interpolation logits β_p (P≈499) = learned, not reported
    Eq. 1/3: one β_p per named tensor, sigmoid-transformed to α_p; optimized on exemplar memory; these are the method's main fitted quantities.
  • Modality-pair logit-scale parameters (3) = learned, not reported
    Section 3/4.1: temperature/logit-scale parameters for audio-image, audio-text, image-text similarities, optimized together with β.
  • Regularization strengths for source checkpoints (λ_EWC=0.8, λ_LwF=0.1, λ_iCaRL=1.0; LwF T=2) = 0.8, 0.1, 1.0, 2
    Section 4.1: chosen by hand/tuned to create θ_reg endpoints; the fused model inherits their behavior, so the central results depend on these choices.
axioms (5)
  • domain assumption Two continual-learning checkpoints with identical architecture can be meaningfully combined by per-tensor convex combination (weight-space linearity).
    Section 3, Eq. 1; borrowed from WiSE-FT/model-soup work; not proven for continual-learning checkpoints or multimodal encoders.
  • domain assumption The 840-exemplar memory is representative enough for fitting interpolation coefficients that transfer to the evaluated retrieval candidate pool.
    Section 3 Eq. 3 and Section 4.1; no sensitivity analysis or overlap check between exemplar classes and test candidates.
  • domain assumption Replacing BatchNorm with GroupNorm while preserving affine parameters preserves the cross-modal retrieval geometry.
    Section 4.1; the authors replace BN to avoid running-stat inconsistencies but do not validate equivalence empirically.
  • domain assumption Per-tensor scalar interpolation is sufficiently expressive; per-entry or finer-grained coefficients are unnecessary.
    Section 3; defended only by analysis in Section 4.5, not by an ablation against finer granularity.
  • domain assumption InfoNCE retrieval loss on a batch from the exemplar memory is a valid surrogate for the reported R@1/mAP.
    Eq. 2/3; standard, but a gap exists between the fitted objective and the reported retrieval metrics.

pith-pipeline@v1.3.0-alltime-deepseek · 14181 in / 12617 out tokens · 125241 ms · 2026-08-02T00:09:38.548682+00:00 · methodology

0 comments
read the original abstract

Multimodal models such as CLIP learn a shared embedding space for cross-modal retrieval, but continual adaptation to sequentially arriving data can disrupt the cross-modal alignment acquired from earlier phases. Conventional continual-learning methods return a single checkpoint, which commits every retrieval direction to the same stability-plasticity trade-off. We propose AlphaWiSE, a post-hoc weight-space interpolation method that composes two frozen source checkpoints. For each aligned parameter tensor identified by its checkpoint key, AlphaWiSE fits one scalar interpolation coefficient shared by all tensor entries. The coefficients are fitted on a smaller exemplar memory and used to materialize one interpolated checkpoint. The deployed model has the same architecture and parameter count as either source checkpoint, which does not require additional inference time. Extensive experiments on audio-image-text retrieval show consistent improvements over strong continual-learning baselines across multiple retrieval directions and evaluation metrics.

Figures

Figures reproduced from arXiv: 2607.15094 by Qiran Hu, Sarthak Jain, Yaoyao Liu, Zhen Zhu.

Figure 1
Figure 1. Figure 1: AlphaWiSE performs post-hoc, per-tensor fusion of two compatible continual-learning checkpoints. Both source checkpoints remain frozen. The exemplar memory is used only to optimize the coefficients β, with αp = σ(βp) and ˜θp = αpθ un p + (1 − αp)θ reg p . After optimization, the fused tensors are materialized into one checkpoint. The final model has the same architecture and inference-time computation as e… view at source ↗
Figure 2
Figure 2. Figure 2: Phase-wise R@1 for the listed continual-learning and AlphaWiSE configurations. Results are shown over phases 1–7 for audio-to-text (A→T), image-to-audio (I→A), and image-to-text (I→T) retrieval on the 79-class candidate pool. Thin solid lines denote the listed non-fusion baselines, and thick dashed lines denote the three AlphaWiSE pairings. At phase 7, the best AlphaWiSE pairing is numerically higher than … view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative image-to-text (I→T) t-SNE projections. Rows compare the continual-learning checkpoint and the corresponding AlphaWiSE fusion, while columns correspond to EWC, iCaRL, and LwF. Colors denote five selected classes (25 image samples per class), circles denote image embeddings, crosses denote text embeddings, and gray crosses denote the text embeddings of all remaining background classes not highlig… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative image-to-audio (I→A) t-SNE projections. Rows compare the continual-learning checkpoint and the corresponding AlphaWiSE fusion, while columns correspond to EWC, iCaRL, and LwF. Colors denote five selected classes (25 image samples per class), circles denote image embeddings, triangles denote audio embeddings, and gray markers denote the embeddings of all remaining background classes not highligh… view at source ↗
Figure 5
Figure 5. Figure 5: Learned interpolation coefficients by depth and parameter type. Per-block mean of the learned coefficient α (α=1: audio–visual fine-tuned model; α=0: stable continual model), averaged over incremental phases 1–7; bands and error bars denote ±1 std across phases. Rows correspond to encoder branch (audio, image, text) and columns to the stable continual learner (LwF, EWC, iCaRL). Parameter type, rather than … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 34 linked inside Pith

  1. [1]

    Learning to route for dynamic adapter composition in continual learning with language models.arXiv preprint arXiv:2408.09053, 2024

    Vladimir Araujo, Marie-Francine Moens, and Tinne Tuytelaars. Learning to route for dynamic adapter composition in continual learning with language models.arXiv preprint arXiv:2408.09053, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization.arXiv preprint arXiv:1607.06450, 2016

    Pith/arXiv arXiv 2016

  3. [3]

    Rainbow memory: Continual learning with a memory of diverse samples

    Jihwan Bang, Heesu Kim, Youngjoon Yoo, Jung-Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. InCVPR, 2021

    2021

  4. [4]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.arXiv preprint arXiv:2106.10199, 2022

    Elad Ben-Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.arXiv preprint arXiv:2106.10199, 2022

    arXiv 2022

  5. [5]

    Rebalancing batch normalization for exemplar-based class-incremental learning.arXiv preprint arXiv:2201.12559, 2023

    Sungmin Cha, Sungjun Cho, Dasol Hwang, Sunwon Hong, Moontae Lee, and Taesup Moon. Rebalancing batch normalization for exemplar-based class-incremental learning.arXiv preprint arXiv:2201.12559, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Doka- nia, Philip H

    Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Doka- nia, Philip H. S. Torr, and Marc’Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486, 2019

    Pith/arXiv arXiv 1902

  7. [7]

    Dual-teacher class-incremental learning with data- free generative replay

    Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Dual-teacher class-incremental learning with data- free generative replay. InCVPR, 2021

    2021

  8. [8]

    A continual learning survey: Defying forgetting in classification tasks

    Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. arXiv preprint arXiv:1909.08383, 2019

    Pith/arXiv arXiv 1909

  9. [9]

    Beyond classification: Dynamic adapter routing for continual multimodal retrieval.arXiv preprint arXiv:2605.31229, 2026

    Alicja Dobrzeniecka, Filip Szatkowski, Sebastian Cygert, Szymon Lukasik, and Bartlomiej Twardowski. Beyond classification: Dynamic adapter routing for continual multimodal retrieval.arXiv preprint arXiv:2605.31229, 2026

    Pith/arXiv arXiv 2026

  10. [10]

    Podnet: Pooled outputs distillation for small-tasks incremental learning

    Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. InECCV, 2020

    2020

  11. [11]

    Prompt-based exemplar super-compression and regeneration for class-incremental learning

    Ruxiao Duan, Jieneng Chen, Adam Kortylewski, Alan Yuille, and Yaoyao Liu. Prompt-based exemplar super-compression and regeneration for class-incremental learning. InBMVC, 2025

    2025

  12. [12]

    inemo: Incremental neural mesh models for robust class-incremental learning

    Tom Fischer, Yaoyao Liu, Artur Jesslen, Noor Ahmed, Prakhar Kaushik, Angtian Wang, Alan L Yuille, Adam Kortylewski, and Eddy Ilg. inemo: Incremental neural mesh models for robust class-incremental learning. InECCV, 2024. 12

    2024

  13. [13]

    Schwab, and Ari S

    Jonathan Frankle, David J. Schwab, and Ari S. Morcos. Training batchnorm and only batchnorm: On the expressive power of random features in cnns.arXiv preprint arXiv:2003.00152, 2021

    Pith/arXiv arXiv 2003

  14. [14]

    Gemmeke, Daniel P

    Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. InICASSP, 2017

    2017

  15. [15]

    Imagebind: One embedding space to bind them all.arXiv preprint arXiv:2305.05665, 2023

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all.arXiv preprint arXiv:2305.05665, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio

    Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical inves- tigation of catastrophic forgetting in gradient-based neural networks.arXiv preprint arXiv:1312.6211, 2015

    Pith/arXiv arXiv 2015

  17. [17]

    Audioclip: Extending clip to image, text and audio.arXiv preprint arXiv:2106.13043, 2021

    Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio.arXiv preprint arXiv:2106.13043, 2021

    Pith/arXiv arXiv 2021

  18. [18]

    Esresne(x)t-fbsp: Learning robust time-frequency transformation of audio.arXiv preprint arXiv:2104.11587, 2021

    Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Esresne(x)t-fbsp: Learning robust time-frequency transformation of audio.arXiv preprint arXiv:2104.11587, 2021

    Pith/arXiv arXiv 2021

  19. [19]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Pith/arXiv arXiv 2021

  20. [20]

    Editing models with task arithmetic.arXiv preprint arXiv:2212.04089, 2023

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic.arXiv preprint arXiv:2212.04089, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    Batchnormalization: Acceleratingdeepnetworktrainingbyreducing internal covariate shift.arXiv preprint arXiv:1502.03167, 2015

    SergeyIoffeandChristianSzegedy. Batchnormalization: Acceleratingdeepnetworktrainingbyreducing internal covariate shift.arXiv preprint arXiv:1502.03167, 2015

    Pith/arXiv arXiv 2015

  22. [22]

    Averaging weights leads to wider optima and better generalization.arXiv preprint arXiv:1803.05407, 2019

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization.arXiv preprint arXiv:1803.05407, 2019

    Pith/arXiv arXiv 2019

  23. [23]

    Energy-based latent aligner for incremental learning

    KJ Joseph, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Vineeth N Balasubra- manian. Energy-based latent aligner for incremental learning. InCVPR, 2022

    2022

  24. [24]

    Overcoming catastrophic forgetting in neural networks.PNAS, 2017

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks.PNAS, 2017

    2017

  25. [25]

    Continual learning with weight interpolation.arXiv preprint arXiv:2404.04002, 2024

    Jędrzej Kozal, Jan Wasilewski, Bartosz Krawczyk, and Michał Woźniak. Continual learning with weight interpolation.arXiv preprint arXiv:2404.04002, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Online optimal control with linear dynamics and predictions: Algorithms and regret analysis.NeurIPS, 2019

    Yingying Li, Xin Chen, and Na Li. Online optimal control with linear dynamics and predictions: Algorithms and regret analysis.NeurIPS, 2019

    2019

  27. [27]

    Online optimization with predictions and switching costs: Fast algorithms and the fundamental limit.IEEE TAC, 2020

    Yingying Li, Guannan Qu, and Na Li. Online optimization with predictions and switching costs: Fast algorithms and the fundamental limit.IEEE TAC, 2020

    2020

  28. [28]

    Online optimal control with affine constraints

    Yingying Li, Subhro Das, and Na Li. Online optimal control with affine constraints. InAAAI, 2021

    2021

  29. [29]

    Learning without forgetting.arXiv preprint arXiv:1606.09282, 2017

    Zhizhong Li and Derek Hoiem. Learning without forgetting.arXiv preprint arXiv:1606.09282, 2017

    Pith/arXiv arXiv 2017

  30. [30]

    Mnemonics training: Multi-class incremental learning without forgetting

    Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi-class incremental learning without forgetting. InCVPR, 2020

    2020

  31. [31]

    Adaptive aggregation networks for class-incremental learn- ing

    Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class-incremental learn- ing. InCVPR, 2021. 13

    2021

  32. [32]

    Rmm: Reinforced memory management for class- incremental learning.NeurIPS, 34:3478–3490, 2021

    Yaoyao Liu, Bernt Schiele, and Qianru Sun. Rmm: Reinforced memory management for class- incremental learning.NeurIPS, 34:3478–3490, 2021

    2021

  33. [33]

    Online hyperparameter optimization for class-incremental learning

    Yaoyao Liu, Yingying Li, Bernt Schiele, and Qianru Sun. Online hyperparameter optimization for class-incremental learning. InAAAI, 2023

    2023

  34. [34]

    Continual detection transformer for incremental object detection

    Yaoyao Liu, Bernt Schiele, Andrea Vedaldi, and Christian Rupprecht. Continual detection transformer for incremental object detection. InCVPR, pp. 23799–23808, 2023

    2023

  35. [35]

    Wakening past concepts without past data: Class-incremental learning from online placebos

    Yaoyao Liu, Yingying Li, Bernt Schiele, and Qianru Sun. Wakening past concepts without past data: Class-incremental learning from online placebos. InW ACV, 2024

    2024

  36. [36]

    Class-incremental exemplar compression for class-incremental learning

    Zilin Luo, Yaoyao Liu, Bernt Schiele, and Qianru Sun. Class-incremental exemplar compression for class-incremental learning. InCVPR, pp. 11371–11380, 2023

    2023

  37. [37]

    Merging models with fisher-weighted averaging.arXiv preprint arXiv:2111.09832, 2022

    Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging.arXiv preprint arXiv:2111.09832, 2022

    Pith/arXiv arXiv 2022

  38. [38]

    Continual vision-language repre- sentation learning with off-diagonal information.arXiv preprint arXiv:2305.07437, 2023

    Zixuan Ni, Longhui Wei, Siliang Tang, Yueting Zhuang, and Qi Tian. Continual vision-language repre- sentation learning with off-diagonal information.arXiv preprint arXiv:2305.07437, 2023

    Pith/arXiv arXiv 2023

  39. [39]

    Continual normalization: Rethinking batch normalization for online continual learning.arXiv preprint arXiv:2203.16102, 2022

    Quang Pham, Chenghao Liu, and Steven Hoi. Continual normalization: Rethinking batch normalization for online continual learning.arXiv preprint arXiv:2203.16102, 2022

    Pith/arXiv arXiv 2022

  40. [40]

    Adapters: A unified library for parameter- efficient and modular transfer learning.arXiv preprint arXiv:2311.11077, 2023

    Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Engländer, Timo Imhof, Ivan Vulić, Sebastian Ruder, Iryna Gurevych, and Jonas Pfeiffer. Adapters: A unified library for parameter- efficient and modular transfer learning.arXiv preprint arXiv:2311.11077, 2023

    Pith/arXiv arXiv 2023

  41. [41]

    Gdumb: A simple approach that questions our progress in continual learning

    Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. InECCV, 2020

    2020

  42. [42]

    Learning transferable visual models from natural language supervision.arXiv preprint arXiv:2103.00020, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision.arXiv preprint arXiv:2103.00020, 2021

    Pith/arXiv arXiv 2021

  43. [43]

    iCaRL: Incre- mental classifier and representation learning

    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: Incre- mental classifier and representation learning. InCVPR, 2017

    2017

  44. [44]

    Continual learning with deep generative replay

    Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. InNeurIPS, 2017

    2017

  45. [45]

    On learning the geodesic path for incremental learning

    Christian Simon, Piotr Koniusz, and Mehrtash Harandi. On learning the geodesic path for incremental learning. InCVPR, 2021

    2021

  46. [46]

    Momentum-based weight interpolation of strong zero-shot models for continual learning.arXiv preprint arXiv:2211.03186, 2022

    Zafir Stojanovski, Karsten Roth, and Zeynep Akata. Momentum-based weight interpolation of strong zero-shot models for continual learning.arXiv preprint arXiv:2211.03186, 2022

    Pith/arXiv arXiv 2022

  47. [47]

    Topology-preserving class- incremental learning

    Xiaoyu Tao, Xinyuan Chang, Xiaopeng Hong, Xing Wei, and Yihong Gong. Topology-preserving class- incremental learning. InECCV, 2020

    2020

  48. [48]

    C-clip: Contrastive image-text encoders to close the descriptive- commentative gap.arXiv preprint arXiv:2309.03921, 2023

    William Theisen and Walter Scheirer. C-clip: Contrastive image-text encoders to close the descriptive- commentative gap.arXiv preprint arXiv:2309.03921, 2023

    Pith/arXiv arXiv 2023

  49. [49]

    Foster: Feature boosting and compres- sion for class-incremental learning

    Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compres- sion for class-incremental learning. InECCV, 2022

    2022

  50. [50]

    Continual learning in cross-modal retrieval.arXiv preprint arXiv:2104.06806, 2021

    Kai Wang, Luis Herranz, and Joost van de Weijer. Continual learning in cross-modal retrieval.arXiv preprint arXiv:2104.06806, 2021. 14

    Pith/arXiv arXiv 2021

  51. [51]

    Dualprompt: Complementary prompting for rehearsal-free continual learning.arXiv preprint arXiv:2204.04799, 2022

    Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guo- long Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Dualprompt: Complementary prompting for rehearsal-free continual learning.arXiv preprint arXiv:2204.04799, 2022

    Pith/arXiv arXiv 2022

  52. [52]

    Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt

    Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.arXiv preprint arXiv:2203.05482, 2022

    Pith/arXiv arXiv 2022

  53. [53]

    Robust fine-tuning of zero-shot models.arXiv preprint arXiv:2109.01903, 2022

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models.arXiv preprint arXiv:2109.01903, 2022

    Pith/arXiv arXiv 2022

  54. [54]

    Memory replay gans: Learning to generate new categories without forgetting.NeurIPS, 2018

    Chenshen Wu, Luis Herranz, Xialei Liu, Joost Van De Weijer, Bogdan Raducanu, et al. Memory replay gans: Learning to generate new categories without forgetting.NeurIPS, 2018

    2018

  55. [55]

    Wav2clip: Learning robust audio representations from clip.arXiv preprint arXiv:2110.11499, 2022

    Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip.arXiv preprint arXiv:2110.11499, 2022

    Pith/arXiv arXiv 2022

  56. [56]

    Group normalization.arXiv preprint arXiv:1803.08494, 2018

    Yuxin Wu and Kaiming He. Group normalization.arXiv preprint arXiv:1803.08494, 2018

    Pith/arXiv arXiv 2018

  57. [57]

    Generative negative text replay for continual vision-language pretraining

    Shipeng Yan, Lanqing Hong, Hang Xu, Jianhua Han, Tinne Tuytelaars, Zhenguo Li, and Xuming He. Generative negative text replay for continual vision-language pretraining. InECCV, 2022

    2022

  58. [58]

    Low-rank prompt interaction for continual vision-language retrieval.arXiv preprint arXiv:2501.14369, 2025

    Weicai Yan, Ye Wang, Wang Lin, Zirun Guo, Zhou Zhao, and Tao Jin. Low-rank prompt interaction for continual vision-language retrieval.arXiv preprint arXiv:2501.14369, 2025

    Pith/arXiv arXiv 2025

  59. [59]

    Boostingcontinual learning of vision-language models via mixture-of-experts adapters.arXiv preprint arXiv:2403.11549, 2024

    JiazuoYu, YunzhiZhuge, LuZhang, PingHu, DongWang, HuchuanLu, andYouHe. Boostingcontinual learning of vision-language models via mixture-of-experts adapters.arXiv preprint arXiv:2403.11549, 2024

    Pith/arXiv arXiv 2024

  60. [60]

    Semantic drift compensation for class-incremental learning

    Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. Semantic drift compensation for class-incremental learning. InCVPR, 2020

    2020

  61. [61]

    Continual learning for abdominal multi-organ and tumor segmentation

    Yixiao Zhang, Xinyi Li, Huimiao Chen, Alan L Yuille, Yaoyao Liu, and Zongwei Zhou. Continual learning for abdominal multi-organ and tumor segmentation. InMICCAI, 2023

    2023

  62. [62]

    Learning without forgetting for vision-language models.TPAMI, 2025

    Da-Wei Zhou, Yuanhan Zhang, Yan Wang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Learning without forgetting for vision-language models.TPAMI, 2025

    2025

  63. [63]

    Anytime continual learning for open vocabulary classifica- tion.arXiv preprint arXiv:2409.08518, 2024

    Zhen Zhu, Yiming Gong, and Derek Hoiem. Anytime continual learning for open vocabulary classifica- tion.arXiv preprint arXiv:2409.08518, 2024

    Pith/arXiv arXiv 2024

  64. [64]

    Continual learning in open-vocabulary classification with complementary memory systems.arXiv preprint arXiv:2307.01430, 2024

    Zhen Zhu, Weijie Lyu, Yao Xiao, and Derek Hoiem. Continual learning in open-vocabulary classification with complementary memory systems.arXiv preprint arXiv:2307.01430, 2024

    Pith/arXiv arXiv 2024

  65. [65]

    How to teach large multimodal models new skills? InECCV, 2026

    Zhen Zhu, Yiming Gong, Yao Xiao, Yaoyao Liu, and Derek Hoiem. How to teach large multimodal models new skills? InECCV, 2026. 15 A Pair-Specific Interpretations A.1 Normal + EWC Letθ N denote the checkpoint obtained by normal sequential training, and letθE denote the checkpoint obtained by EWC. At a given phase, normal training approximately optimizes the ...

    2026