Pith. sign in

REVIEW 3 major objections 7 minor 40 references

Leveraging Pre-Trained Models for Multimodal Class-Incremental Learning under Adaptive Fusion

T0 review · 3 major / 7 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes a frozen-AudioCLIP continual learner whose adaptive fusion and MoE adapters yield the highest reported accuracy and lowest forgetting on three vision-audio-text benchmarks.

desk verdict Plausible engineering advance for audio-vision-text continual learning, but the mutual-information loss is undefined as written and the empirical claims lack error bars; the result is worth a serious look with major revision. read the letter →

arxiv 2506.09999 v1 pith:2JWEIPNQ submitted 2025-02-07 cs.LG cs.MMcs.SDeess.AS

Yukun Chen , Zihuan Qiu , Fanman Meng , Hongliang Li , Linfeng Xu , Qingbo Wu This is my paper · ORCID
classification cs.LGcs.MMcs.SDeess.AS
keywords multimodalclass-incrementallearningcatastrophicforgettingMixture-of-Expertsadaptiveaudio-visualfusionAudioCLIPcontrastivelossincrementalevaluationmetricsvision-audio-text
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that multimodal class-incremental learning (MCIL) can work across vision, audio, and text by keeping a pre-trained AudioCLIP backbone frozen and injecting new-task knowledge through lightweight Mixture-of-Experts adapters. It claims that an adaptive audio-visual fusion module, which masks out weak modality features and dynamically combines vision with audio, plus a multimodal contrastive loss, outperforms re-implemented continual learning baselines on three datasets. If correct, this would be one of the first demonstrations that a multimodal pre-trained model can be fine-tuned incrementally while exploiting complementary information from three modalities. The practical stake is systems that continuously learn in real settings where data arrive as images, sound, and text together, with classroom activity recognition as the running example.

What carries the argument

Three mechanisms carry the argument. The Multimodal Incremental Feature Extractor (MIFE) freezes the AudioCLIP encoders and inserts task-specific experts, implemented as LoRA-decomposed adapters, with routers controlling whether each expert is active; this is what limits forgetting while allowing knowledge sharing. The Adaptive Audio-Visual Fusion Module (AAVFM) computes a Pearson correlation between visual and audio features, masks out the audio channel when the correlation falls below a threshold, and otherwise fuses modalities through weighted concatenation and cross-attention; this is what handles imbalanced modality quality. The total loss combines a sample-weighted cross-entropy term with a mutual-information term meant to preserve modality detail in the fused feature, and the paper also proposes two metrics, M1 and M2, that assess accuracy, forgetting, and multimodal information use together.

What would settle it

A concrete check is to run the authors' training pipeline on miniARIC with T=3 and verify the reported 67.76% average Top-1 accuracy and that the total loss actually computes the mutual-information terms in Eqs. (6)--(8); if the numbers do not reproduce or the loss is implemented without an estimator for $I$, the central claim is not settled.

Watch

Extended reading notes

Core claim

The central claim is that a single frozen multimodal encoder can be extended incrementally by adding task-specific experts and an adaptive fusion head, and that this combination achieves higher average Top-1 accuracy and lower forgetting than the baselines re-implemented on AudioCLIP. Concretely, the paper reports average Top-1 accuracy of 67.76% versus 49.19% for the strongest baseline on miniARIC at T=3, with consistent gains on ImageNet-ESC-19 and ImageNet-ESC-27 across task splits. The authors attribute the gains to preserving the frozen encoders, per-sample gating of weak audio features, and a multimodal contrastive loss that keeps the fused feature aligned with both visual and audio information.

Load-bearing premise

The load-bearing premise is that the mutual-information terms in the training loss can actually be computed from high-dimensional features, but the paper supplies no estimator or implementation for $I$, so the stated training objective is not fully defined.

Editorial extensions

If this is right

  • Average Top-1 accuracy across all seen classes should improve by up to roughly 18 points over the strongest baseline on miniARIC, with smaller but consistent gains on the ImageNet-ESC splits.
  • Forgetting of old classes should be visibly reduced because only the MoE experts and fusion module are trained while the AudioCLIP encoders stay frozen.
  • The adaptive fusion module should help most when modalities are imbalanced, such as noisy audio, by falling back to the stronger modality below the masking threshold.
  • The proposed M1 and M2 metrics should make MCIL comparisons sensitive to task similarity, memory stability, plasticity, and multimodal information use rather than only average accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the masking-threshold idea could be adapted to missing-modality scenarios, where the model decides per sample whether one modality alone is enough for classification; the paper leaves that extension for future work.
  • Editorial extension: because the mutual-information loss lacks an explicit estimator, a practical implementation would likely replace $I$ with a variational or InfoNCE-style bound, and whether the reported gains survive that replacement is an open empirical question.
  • Editorial extension: the paper does not specify how the MoE routers select experts at inference time when task identity is unknown, so a deployed system would need a task-id estimator or a routing rule based on feature statistics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes a multimodal class-incremental learning (MCIL) method for vision, audio, and text using a pre-trained AudioCLIP backbone. It introduces a Mixture-of-Experts-based incremental feature extractor (MIFE), an adaptive audio-visual fusion module (AAVFM) with a masking threshold and cross-attention, a text-prompt diversity strategy, and a loss that combines a sample-weighted cross-entropy term with mutual-information terms. The authors report higher average Top-1 accuracy and lower forgetting than several re-implemented CIL baselines on miniARIC, ImageNet-ESC-19, and ImageNet-ESC-27, and propose two new MCIL evaluation metrics M1 and M2.

Significance. If the empirical claims hold, this would be a useful early demonstration of MCIL with three heterogeneous modalities on top of a multimodal pre-trained model. The main idea of gating fusion by data quality and using task-specific MoE adapters is reasonable and the reported gains, especially on miniARIC (67.76 vs. 49.19 for T=3), are substantial. The paper is not circular: the headline comparison is an accuracy benchmark against external and re-implemented baselines. However, the current form is not reproducible because a load-bearing component of the training loss is not actually specified, and the experimental section lacks error bars. The proposed metrics are also not fully defined. The contribution is therefore promising but requires a substantive revision before the results can be considered supported.

major comments (3)
  1. [Sec. II-C, Eqs. (6)-(9)] The total loss L = αLCW + (1−α)LMI depends on I(f_fusion; F_v) and I(f_fusion; F_a), but no mutual-information estimator, binning scheme, negative sampling strategy, or approximation is provided anywhere in the manuscript. Since f_fusion, F_v, and F_a are high-dimensional continuous feature vectors, these quantities cannot be evaluated as written. Consequently, the training procedure is not defined, the '+ L' ablation row in Table II cannot be implemented or checked, and the state-of-the-art numbers in Table I are not reproducible from the paper alone. The authors should specify an exact estimator (e.g., InfoNCE with temperature and negative samples, or a kernel-based estimator) and state how it is used in backpropagation.
  2. [Sec. III-C, Table II] The text states that each component of the method effectively improves classification performance, but Table II shows that adding AAVFM after Textual Diversity reduces average Top-1 accuracy from 58.73 to 42.44 and last Top-1 accuracy from 46.86 to 25.91. Only M1 and M2 improve. This contradicts the claimed contribution of AAVFM to the average accuracy metric. The ablation should be presented as add-one-in and leave-one-out experiments with clearly defined base configurations, and the claim about component effectiveness should be revised to match the table.
  3. [Sec. III-A and III-B, Tables I-III] No standard deviations, number of runs, or seeds are reported for any experiment. The claim in Sec. III-B that the method 'significantly outperforms others' is therefore not supported against run-to-run variability. The authors should provide at least three independent runs with means and standard deviations, or otherwise justify that the differences are outside noise. This is a standard requirement for empirical continual-learning papers.
minor comments (7)
  1. [Sec. II-C, Eqs. (4)-(5)] The definition of LCW is ambiguous: wij is indexed by both i and j, but the loss inside the double sum appears to depend only on i, so the expression reduces to a per-sample weighted CE loss with weight (1/n) Σ_j wij. Please clarify whether this is intended.
  2. [Sec. II-D, Eqs. (10)-(11)] The definitions of M1 and M2 use terms For_t, BWT_t, FWT_t, and wt, but the formal definitions of For_t, BWT_t, and FWT_t are not given, and the mapping from cosine similarity to wt is only described qualitatively. Moreover, NM I_{f−v} and NM I_{f−a} in Eq. (11) require an estimator for normalized mutual information on continuous features, which is also not specified.
  3. [Sec. II-B, Eq. (1)] The Pearson correlation coefficient r is computed between Fv(vt) and Fa(at), but the manuscript does not state over which dimension the sums are taken or how the threshold th is applied to a correlation coefficient per sample, which is needed for reproducibility.
  4. [Sec. II-A] The MoE structure is described only at a high level; the number of experts, the router architecture, the LoRA rank, and the criteria for activating or deactivating experts are not given. These details are needed to reproduce MIFE.
  5. [Sec. III-A] The experimental setup lists only N, th, and α. Learning rate, batch size, number of epochs, optimizer settings, and the full set of hyperparameters for the re-implemented baselines are missing.
  6. [Fig. 2] Figure 2 contains malformed placeholder text (e.g., '?????' and 'Calculate ?') and non-English characters. Please replace these with correct mathematical notation or English labels.
  7. [References] Reference [3] and reference [11] are the same GDumb paper but appear as two separate entries; references [38] and [39] also contain stray text in the journal/venue fields.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the central claim is an empirical accuracy comparison against external and re-implemented baselines, and no fitted quantity is repackaged as a prediction.

full rationale

The paper's main claim is an empirical accuracy comparison in Table I against external baselines and re-implemented methods on miniARIC, ImageNet-ESC-19, and ImageNet-ESC-27. The proposed components (MIFE, AAVFM, LCW, LMI) are defined independently of the measured outcomes; there is no fitted parameter that is then reported as a prediction, and no derivation reduces the reported accuracies to the method's own definitions. Equations (6)-(8) define LMI using mutual information I(.;.) without specifying an estimator or approximation, so the loss in Eq. (9) is not actually computable as written; this is a serious reproducibility/correctness omission, but not a circularity, because the results do not follow by construction from the definition of the loss. The self-citations [28], [29], [36] supply the MCIL task formulation and the miniARIC dataset; they are not load-bearing for the numerical superiority claim, which is also supported by the independently sourced ImageNet-ESC datasets [37]. The proposed metrics M1 and M2 are used only in ablations and do not substitute for the standard average/last Top-1 accuracy comparisons. No uniqueness theorem or ansatz is smuggled in through self-citation. Therefore no circular step is present; the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central method is an empirical construction on top of AudioCLIP; it introduces no new physics or formal objects. The main assumptions are the transferability of AudioCLIP, the validity of Pearson-correlation gating, and the computability of the mutual information losses. Free parameters (th, alpha, N and unreported training hyperparameters) are chosen by hand without sensitivity analysis. Since no code or data is shipped, the ledger cannot be resolved further.

free parameters (4)
  • masking threshold th = 0.8
    Fixed in Sec III-A; controls when the audio modality is dropped in AAVFM; no sensitivity analysis is reported, yet it directly affects whether audio features are fused.
  • loss weight alpha = 0.7
    Fixed in Sec III-A; balances LCW and LMI in Eq. (9); no sensitivity analysis is reported.
  • number of text prompt templates N = 35
    Chosen in Sec III-A to expand each class label; no analysis of how performance depends on N is given.
  • training hyperparameters (learning rate, batch size, epochs, LoRA rank, MoE expert count) = not reported
    These free choices are not given in Sec III-A, so the exact reported numbers cannot be reproduced without guessing.
assumptions (5)
  • domain assumption AudioCLIP's pretrained encoders provide an aligned cross-modal feature space suitable for incremental fine-tuning.
    The entire method builds on AudioCLIP and assumes its encoders transfer to the three datasets; this is standard practice but not proven for the MCIL setting.
  • ad hoc to paper The Pearson correlation between image and audio features indicates data quality agreement and can safely gate fusion.
    Eq. (1) and the threshold th in Sec II-B introduce this assumption without evidence that correlation is a reliable quality measure; a low correlation may indicate complementary information rather than noise.
  • ad hoc to paper The mutual information terms I(f_fusion; F_v) and I(f_fusion; F_a) are computable in practice.
    Eqs. (6)-(8) use I without specifying an estimator or approximation, so the loss is underspecified.
  • domain assumption Existing CLIP continual learning experience transfers to AudioCLIP's three-modality setting.
    MoE-CLIP and related prompt and LoRA methods are assumed to transfer; the paper relies on this to justify MIFE.
  • domain assumption The new M1 and M2 metrics are valid measures of MCIL quality.
    Proposed in Sec II-D without validation against existing benchmarks; they are then used in the ablation as evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging Pre-Trained Models for Multimodal Class-Incremental Learning under Adaptive Fusion." pith.science (2026). https://pith.science/paper/2JWEIPNQ

@misc{pith2026250609999,
  author       = {Pith},
  title        = {Pith review of: Leveraging Pre-Trained Models for Multimodal Class-Incremental Learning under Adaptive Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2JWEIPNQ}},
  note         = {Machine review of arXiv:2506.09999}
}
read the original abstract

Unlike traditional Multimodal Class-Incremental Learning (MCIL) methods that focus only on vision and text, this paper explores MCIL across vision, audio and text modalities, addressing challenges in integrating complementary information and mitigating catastrophic forgetting. To tackle these issues, we propose an MCIL method based on multimodal pre-trained models. Firstly, a Multimodal Incremental Feature Extractor (MIFE) based on Mixture-of-Experts (MoE) structure is introduced to achieve effective incremental fine-tuning for AudioCLIP. Secondly, to enhance feature discriminability and generalization, we propose an Adaptive Audio-Visual Fusion Module (AAVFM) that includes a masking threshold mechanism and a dynamic feature fusion mechanism, along with a strategy to enhance text diversity. Thirdly, a novel multimodal class-incremental contrastive training loss is proposed to optimize cross-modal alignment in MCIL. Finally, two MCIL-specific evaluation metrics are introduced for comprehensive assessment. Extensive experiments on three multimodal datasets validate the effectiveness of our method.

Figures

Figures reproduced from arXiv: 2506.09999 by the authors.

Figure 1
Figure 1. (a) Illustration of cross-modal matching in the feature [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall framework of our proposed method. visual modality data, then the obtained feature with clear clas￾sification boundaries are matched with text features obtained by a strategy for enhancing text diversity. To address the second limitation, a Multimodal Incremental Feature Extractor (MIFE) based on Mixture-of-Experts (MoE) structure [27], [32] is proposed to achieve effective incremental fine-tuning for AudioCL… view at source ↗
Figure 4
Figure 4. t-SNE visualization in the first incremental stage. TABLE III: Ablation study of the AAVFM. Best results - in bold. Method ImageNet-ESC-19 ImageNet-ESC-27 Avg. Acc. Last. Acc. Avg. Acc. Last. Acc. Partial Fine-tune 74.52 66.67 64.70 61.74 Baseline w/ AAVFM 95.12 92.00 95.70 91.30 Table II, demonstrate that each component of our method ef￾fectively improves classification performance in MCIL across average Top-1 accu… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 24 canonical work pages

  1. [1]

    icarl: Incre- mental classifier and representation learning,

    S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incre- mental classifier and representation learning,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017, pp. 5533– 5542

    work page 2017

  2. [2]

    Expe- rience replay for continual learning,

    D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne, “Expe- rience replay for continual learning,” in NeurIPS, 2019

    work page 2019

  3. [4]

    Gradient projection memory for continual learning,

    G. Saha, I. Garg, and K. Roy, “Gradient projection memory for continual learning,” arXiv preprint arXiv:2103.09762 , 2021

    arXiv 2021

  4. [5]

    Continual learning with foundation models: An empirical study of latent replay,

    O. Ostapenko, T. Lesort, P. Rodriguez, M. R. Arefin, A. Douillard, I. Rish, and L. Charlin, “Continual learning with foundation models: An empirical study of latent replay,” in Conference on lifelong learning agents. PMLR, 2022, pp. 60–91

    work page 2022

  5. [6]

    Learning without forgetting,

    Z. Li and D. Hoiem, “Learning without forgetting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 12, pp. 2935– 2947, 2018

    work page 2018

  6. [7]

    Rotate your networks: Better weight consolidation and less catastrophic forgetting,

    X. Liu, M. Masana, L. Herranz, J. V . de Weijer, A. M. Lopez, and A. D. Bagdanov, “Rotate your networks: Better weight consolidation and less catastrophic forgetting,” in ICPR, 2018, pp. 2262–2268

    work page 2018

  7. [8]

    Continual learning by asymmetric loss approximation with single-side overestimation,

    D. Park, S. Hong, B. Han, and K. M. Lee, “Continual learning by asymmetric loss approximation with single-side overestimation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 3335–3344

    work page 2019

  8. [9]

    Continual learning with extended kronecker-factored approximate curvature,

    J. Lee, H. G. Hong, D. Joo, and J. Kim, “Continual learning with extended kronecker-factored approximate curvature,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9001–9010

    work page 2020

Show all 40 references
  1. [10]

    Podnet: Pooled outputs distillation for small-tasks incremental learning,

    A. Douillard, M. Cord, C. Ollion, T. Robert, and E. Valle, “Podnet: Pooled outputs distillation for small-tasks incremental learning,” in Computer vision–ECCV 2020: 16th European conference, Glasgow, UK, August 23–28, 2020, proceedings, part XX 16 . Springer, 2020, pp. 86– 102

    2020

  2. [11]

    Gdumb: A simple approach that questions our progress in continual learning,

    A. Prabhu, P. H. Torr, and P. K. Dokania, “Gdumb: A simple approach that questions our progress in continual learning,” in Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16 . Springer, 2020, pp. 524–540

    2020

  3. [12]

    Few-shot class-incremental learning,

    X. Tao, X. Hong, X. Chang, S. Dong, X. Wei, and Y . Gong, “Few-shot class-incremental learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 12 183–12 192

    2020

  4. [13]

    Der: Dynamically expandable representation for class incremental learning,

    S. Yan, J. Xie, and X. He, “Der: Dynamically expandable representation for class incremental learning,” in CVPR, 2021, pp. 3014–3023

    2021

  5. [14]

    Dytox: Trans- formers for continual learning with dynamic token expansion,

    A. Douillard, A. Ram ´e, G. Couairon, and M. Cord, “Dytox: Trans- formers for continual learning with dynamic token expansion,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9285–9295

    2022

  6. [15]

    Dense network ex- pansion for class incremental learning,

    Z. Hu, Y . Li, J. Lyu, D. Gao, and N. Vasconcelos, “Dense network ex- pansion for class incremental learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 11 858–11 867

    2023

  7. [16]

    Compacter: Efficient low-rank hypercomplex adapter layers,

    R. Karimi Mahabadi, J. Henderson, and S. Ruder, “Compacter: Efficient low-rank hypercomplex adapter layers,” Advances in Neural Information Processing Systems, vol. 34, pp. 1022–1035, 2021

    2021

  8. [17]

    1% vs 100%: Parameter-efficient low rank adapter for dense predictions,

    D. Yin, Y . Yang, Z. Wang, H. Yu, K. Wei, and X. Sun, “1% vs 100%: Parameter-efficient low rank adapter for dense predictions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 20 116–20 126

    2023

  9. [18]

    A unified continual learning framework with general parameter-efficient tuning,

    Q. Gao, C. Zhao, Y . Sun, T. Xi, G. Zhang, B. Ghanem, and J. Zhang, “A unified continual learning framework with general parameter-efficient tuning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 11 483–11 493

    2023

  10. [19]

    Parameter-efficient fine- tuning for large models: A comprehensive survey,

    Z. Han, C. Gao, J. Liu, S. Q. Zhang et al. , “Parameter-efficient fine- tuning for large models: A comprehensive survey,” arXiv preprint arXiv:2403.14608, 2024

    2024 arXiv

  11. [20]

    Vmt-adapter: Parameter- efficient transfer learning for multi-task dense scene understanding,

    Y . Xin, J. Du, Q. Wang, Z. Lin, and K. Yan, “Vmt-adapter: Parameter- efficient transfer learning for multi-task dense scene understanding,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 14, 2024, pp. 16 085–16 093

    2024

  12. [21]

    Visual prompt tuning,

    M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, “Visual prompt tuning,” in European Conference on Computer Vision. Springer, 2022, pp. 709–727

    2022

  13. [22]

    Learning to prompt for continual learning,

    Z. Wang, Z. Zhang, C.-Y . Lee, H. Zhang, R. Sun, X. Ren, G. Su, V . Perot, J. Dy, and T. Pfister, “Learning to prompt for continual learning,” in CVPR, 2022, pp. 139–149

    2022

  14. [23]

    Dualprompt: Complementary prompting for rehearsal-free continual learning,

    Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y . Lee, X. Ren, G. Su, V . Perot, J. Dyet al., “Dualprompt: Complementary prompting for rehearsal-free continual learning,” in European Conference on Computer Vision. Springer, 2022, pp. 631–648

    2022

  15. [24]

    Coda-prompt: Contin- ual decomposed attention-based prompting for rehearsal-free continual learning,

    J. S. Smith, L. Karlinsky, V . Gutta, P. Cascante-Bonilla, D. Kim, A. Arbelle, R. Panda, R. Feris, and Z. Kira, “Coda-prompt: Contin- ual decomposed attention-based prompting for rehearsal-free continual learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision a...

    2023

  16. [25]

    Don’t stop learn- ing: Towards continual learning for the clip model,

    Y . Ding, L. Liu, C. Tian, J. Yang, and H. Ding, “Don’t stop learn- ing: Towards continual learning for the clip model,” arXiv preprint arXiv:2207.09248, 2022

    2022 arXiv

  17. [26]

    Preventing zero-shot transfer degradation in continual learning of vision-language models,

    Z. Zheng, M. Ma, K. Wang, Z. Qin, X. Yue, and Y . You, “Preventing zero-shot transfer degradation in continual learning of vision-language models,” arXiv preprint arXiv:2303.06628 , 2023

    2023 arXiv

  18. [27]

    Boosting continual learning of vision-language models via mixture-of-experts adapters,

    J. Yu, Y . Zhuge, L. Zhang, P. Hu, D. Wang, H. Lu, and Y . He, “Boosting continual learning of vision-language models via mixture-of-experts adapters,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 23 219–23 230

    2024

  19. [28]

    Towards continual egocentric activity recognition: A multi-modal egocentric activity dataset for continual learning,

    L. Xu, Q. Wu, L. Pan, F. Meng, H. Li, C. He, H. Wang, S. Cheng, and Y . Dai, “Towards continual egocentric activity recognition: A multi-modal egocentric activity dataset for continual learning,” IEEE Transactions on Multimedia , 2023

    2023

  20. [29]

    Vision-sensor attention based continual multimodal egocentric activity recognition,

    S. Cheng, C. He, K. Chen, L. Xu, H. Li, F. Meng, and Q. Wu, “Vision-sensor attention based continual multimodal egocentric activity recognition,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 6300–6304

    2024

  21. [30]

    Audioclip: Extending clip to image, text and audio,

    A. Guzhov, F. Raue, J. Hees, and A. Dengel, “Audioclip: Extending clip to image, text and audio,” in ICASSP, 2022, pp. 976–980

    2022

  22. [31]

    Mmg-ego4d: Multimodal generalization in egocentric action recognition,

    X. Gong, S. Mohan, N. Dhingra, J.-C. Bazin, Y . Li, Z. Wang, and R. Ranjan, “Mmg-ego4d: Multimodal generalization in egocentric action recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 6481–6491

    2023

  23. [32]

    A survey on mixture of experts,

    W. Cai, J. Jiang, F. Wang, J. Tang, S. Kim, and J. Huang, “A survey on mixture of experts,” arXiv preprint arXiv:2407.06204 , 2024

    2024 arXiv

  24. [33]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , 2021

    2021 arXiv

  25. [34]

    Multilayer perceptron (mlp),

    H. Taud and J.-F. Mas, “Multilayer perceptron (mlp),” Geomatic ap- proaches for modeling land change scenarios , pp. 451–455, 2018

    2018

  26. [35]

    Don’t forget, there is more than forgetting: new metrics for continual learning,

    N. D ´ıaz-Rodr´ıguez, V . Lomonaco, D. Filliat, and D. Maltoni, “Don’t forget, there is more than forgetting: new metrics for continual learning,” arXiv preprint arXiv:1810.13166 , 2018

    2018 arXiv

  27. [36]

    Aric: An activity recognition dataset in classroom surveillance images,

    L. Xu, F. Meng, Q. Wu, L. Pan, H. Qiu, L. Wang, K. Chen, K. Geng, Y . Qian, H. Wang, S. Zhou, S. Ling, Z. Liu, N. Chen, Y . Xu, S. Cheng, B. Tan, Z. Xu, and H. Li, “Aric: An activity recognition dataset in classroom surveillance images,” 2024. [Online]. Available: https://arxi...

    2024 arXiv

  28. [37]

    Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models,

    Z. Lin, S. Yu, Z. Kuang, D. Pathak, and D. Ramanan, “Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 19 325–19 337

    2023

  29. [38]

    An image is worth 16x16 words: Trans- formers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Trans- formers for image recognition at scale,” arXiv: Computer Vision and Pattern Re...

    2020

  30. [39]

    Esresne(x)t-fbsp: Learning robust time-frequency transformation of audio,

    A. Guzhov, F. Raue, J. Hees, and A. Dengel, “Esresne(x)t-fbsp: Learning robust time-frequency transformation of audio,” arXiv: Sound,arXiv: Sound, 2021

    2021

  31. [40]

    Decoupled weight decay regularization,

    I. Loshchilov, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  32. [41]

    Sgdr: Stochastic gradient descent with warm restarts,

    I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” arXiv preprint arXiv:1608.03983 , 2016

    2016 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.