Pith. sign in

REVIEW 2 major objections 6 minor 56 references

PAL: Prompting Analytic Learning with Missing Modality for Multi-Modal Class-Incremental Learning

T0 review · 2 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PAL combines modality-specific prompts with a recursive least-squares classifier so that multi-modal class-incremental learning stays accurate even when image or text is missing at train and test time.

desk verdict PAL is a solid empirical MMCIL contribution with an overclaimed theoretical guarantee that only holds under a fixed feature extractor, which the method itself violates. read the letter →

arxiv 2501.09352 v1 pith:QS5W7RSN submitted 2025-01-16 cs.LG cs.MMeess.IV

Xianghu Yue , Yiming Chen , Xueyi Zhang , Xiaoxue Gao , Mengling Feng , Mingrui Lao , Huiping Zhuang , Haizhou Li This is my paper · ORCID
classification cs.LGcs.MMeess.IV
keywords multi-modalclass-incrementallearningmissingmodalitypromptanalyticrecursiveleastsquarescatastrophicforgettingexemplar-freecontinualimage-text
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that multi-modal class-incremental learning can be made robust to missing modalities by pairing modality-specific prompts with an analytic recursive least-squares classifier. It argues that prompts preserve a holistic representation when image or text is absent, and that the recursive update of the classifier is exactly equivalent to joint training on all data seen so far, so forgetting is controlled without storing exemplars. If true, continual learners on audio-visual or image-text streams could be deployed in realistic settings where sensors or inputs fail, rather than assuming every sample arrives complete.

What carries the argument

The load-bearing object is the recursive least-squares classifier fed by a frozen multi-modal transformer backbone with modality-specific prompt pools. The RLS recursion, rooted in the Woodbury matrix identity, carries the anti-forgetting guarantee: if the feature mapping is fixed, incremental weight updates equal the weights from joint training on all past and current data, so old knowledge is preserved without exemplars. The prompt pools, one for image and one for text, selected by attention-weighted cosine similarity, do the complementary job of reconstructing missing-modality query embeddings and maintaining a holistic input representation, which addresses the under-fitting that a frozen analytic backbone would otherwise suffer on complex multi-modal data.

What would settle it

Run PAL on UPMC-Food101 with missing-both at 70% missing, then after each step compare the recursively updated classifier weights against the closed-form solution of Eq. (12) computed from all data seen so far; if the norm of the difference is not near machine precision, Theorem 1's premise fails under prompt retraining.

Watch

Extended reading notes

Core claim

The central discovery is that multi-modal class-incremental learning under missing modality can be reformulated as a Recursive Least-Squares problem and solved analytically. With the pre-trained multi-modal backbone frozen, the classifier at step k is updated as $\hat{W}_k = \hat{W}_{k-1} - R_k H_k^T H_k \hat{W}_{k-1} + R_k H_k^T Y_k$, with $R_k$ updated by the Woodbury identity; Theorem 1 proves this recursion reproduces the joint-training solution. Modality-specific prompt pools, tuned by backpropagation with a reconstruction loss, compensate for absent inputs so the frozen backbone still sees a holistic representation. The paper reports that this combination beats prior prompt-based, replay-based, and analytic baselines on UPMC-Food101 and N24News across missing rates from 10% to 90%, with lower forgetting and accuracy that stays roughly constant as the number of incremental steps grows from 5 to 50.

Load-bearing premise

The recursive classifier update is exact only if the frozen backbone produces the same features for a given input at every incremental step, yet PAL retrains the prompt pools with backpropagation at each task, so the feature map shifts and the claimed equivalence to joint training becomes approximate.

Editorial extensions

If this is right

  • Deploying continual learners on real image-text or audio-visual streams where modalities drop out becomes feasible without storing old exemplars.
  • The method's accuracy and forgetting stay nearly constant from 5 to 50 incremental steps, so fine-grained class arrivals do not degrade old knowledge.
  • Missing rates up to 90% are handled: the paper reports 69.69% accuracy on UPMC-Food101 in the 90% missing-both scenario, with lower forgetting than competing baselines.
  • The framework removes the modality-completeness assumption that current multi-modal class-incremental methods rely on, opening the same design to other paired modalities.
  • The authors state tri-modal learning and beyond as the planned next step, suggesting the prompt-pool-plus-analytic-classifier structure is intended to generalize.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The equivalence in Theorem 1 likely degrades when prompt pools shift features across tasks, since the recursion assumes a fixed feature mapping; a testable extension is to add a correction term to the RLS update when prompts evolve.
  • Because the classifier is analytic, per-task prompt retraining may need fewer epochs, and the reconstruction loss acts as a self-supervised regularizer that could transfer to uni-modal class-incremental learning.
  • The step-invariance result suggests the remaining bottleneck in long-run continual learning here is representation quality rather than storage, so a direct comparison against replay methods with large buffers would clarify where PAL's advantage comes from.
  • The same two-module design with additional prompt pools may extend to audio-visual pairs or tri-modal inputs, though the paper does not yet demonstrate that extension.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes PAL, an exemplar-free framework for multi-modal class-incremental learning in which one modality may be missing at training or test time. PAL combines two modules: modality-specific prompt pools that are tuned via backpropagation, and an analytic learning module in which the classifier is retrained by recursive least squares. The theoretical centerpiece is Theorem 1, which states that the RLS recursion yields the same classifier as joint training on all seen tasks. Experiments on UPMC-Food101 and N24News compare PAL with prompt-based and analytic baselines under missing rates from 10% to 90%, along with ablations of prompt structure, AL up-sampling, regularization, and large-step variants.

Significance. If the claims are established, PAL addresses a relevant and under-explored problem: multi-modal incremental learning under missing modalities, without rehearsal. The combination of prompt tuning with analytic learning is a sensible way to mitigate the under-fitting that arises when the backbone is frozen. The paper includes a self-contained derivation of the RLS recursion, broad experiments over missing-rate configurations, and ablations that isolate the prompt and AL components. The main reservation is that Theorem 1 is not actually satisfied by the proposed algorithm, because the prompt module—which is part of the feature extractor—is retrained at every increment, so the exact joint-training equivalence is not achieved. The empirical contribution is potentially strong, but the absence of error bars prevents significance claims.

major comments (2)
  1. [III.D.2, Algorithm 1, Theorem 1] Theorem 1 is derived for fixed feature matrices H_1,...,H_k (Appendix, Eqs. (1)-(15)), and the proof is algebraically correct under that assumption. However, in PAL the feature mapping is not fixed: Algorithm 1 step 4 updates P_v and P_t by backpropagation at every task, and Eq. (7) extracts H_k with the newly updated prompts, while the stored H_1,...,H_{k-1} come from earlier, different prompt states. Therefore Eqs. (13)-(14) do not minimize the joint objective Eq. (11) under the final feature mapping, and the claim after Theorem 1 that 'the model trained incrementally yields the same weights as that trained on both current and all previous data' is not justified by the proof. Since the paper uses this equivalence to explain anti-forgetting and calls Table V a demonstration of a 'step-invariant property,' this mismatch is load-bearing. Please restrict the theorem to the fixed-prompt condition and explicitly treat prompt drift as an approximation, or modify the algorithm so all historical features are produced by a single fixed mapping (e.g., freeze prompts after task 1, or recompute historical H_i after each prompt update if memory allows).
  2. [IV.B.1, Tables I-IV] The paper states that results are averaged over 3 runs but reports no standard deviations, confidence intervals, or significance tests. Several comparisons are close (e.g., Table I, eta=90% missing-both, PAL Acc 69.69 vs ACIL 65.93, while PAL FG 5.32 vs ACIL 4.33 is worse), so without variance information the claim that PAL 'significantly outperforms' all baselines is not supportable. Please report error bars or other variability measures and temper the significance language accordingly.
minor comments (6)
  1. [III.C] In Eq. (2), 'sim(·, ·) measures consine similarity' should be 'cosine similarity.'
  2. [V.A] The dataset name is written as 'UMPC-Food101' in some places and 'UPMC-Food101' elsewhere; please standardize.
  3. [Algorithm 1] Step 5 says 'Update R_k and W_k via AL using Eq. (13) and Eq. (14), respectively,' but Eq. (13) updates W_k and Eq. (14) updates R_k; the mapping in the sentence is reversed.
  4. [References] References [23] and [51] are the same DS-AL paper, which causes duplicate citation ambiguity. Also, reference [54] is attributed to Dosovitskiy et al. but lists the ViLT title; the citation should point to the Vision Transformer paper ('An Image is Worth 16x16 Words') for the statement in Section IV.B.1.
  5. [III.D.1, Algorithm 1] The manuscript is ambiguous about the state of the classifier when each new task begins. Section III.D.1 says the prompt module and classifier are trained via BP, but Algorithm 1 step 4 says only 'Update P_v and P_t via BP,' and step 5 re-trains the classifier via AL. Clarify whether the AL-trained classifier from step k-1 is used to initialize the BP stage at step k, or whether a fresh classifier is used, and how the two linear layers (W_FNN in Eq. (6) and the AL weights W_up and W) interact after re-training.
  6. [V.A] The conclusion that PAL achieves the lowest forgetting should be read against Table I: at eta=90% missing-both, ACIL reports FG 4.33 vs PAL 5.32. The text says 'lowest values across most settings,' which is careful, but the abstract and conclusion should not claim universal superiority on forgetting without qualification.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the RLS equivalence is proven in the appendix from the Woodbury identity, and the empirical claims are benchmarked against external baselines.

full rationale

The central derivation chain is Theorem 1 (Eqs. 13-14), whose proof in the Appendix computes R_k via the Woodbury identity and then obtains W_k by substitution; this is a self-contained algebraic derivation that does not depend on the prompt module, the reconstruction loss, or any fitted value. The prompt module (Eqs. 2-4) and the RLS module are defined independently, and the empirical gains in Tables I-II compare PAL to external baselines (MAP, MSP, L2P, DualPrompt, RebQ, ACIL) on standard datasets, so the main claims are not constructed from their own outputs. The paper's self-citations to ACIL, DS-AL, BRMP, GKEAL, and MMAL are used for background and for specific design choices, such as the random initialization of Wup and ReLU, rather than to justify the central equivalence; the key theorem is proved locally in the appendix. A reviewer-level concern is that prompts are retrained by backpropagation at each step while Theorem 1's interpretation as equivalence to joint training assumes a single fixed feature mapping for all stored H_i, but this is an assumption-validity issue, not circularity: the recursion is still the exact least-squares solution for the stored H_i. No circular step is exhibited.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The method is a combination of existing prompt learning and analytic learning machinery. It depends on hand-chosen hyperparameters (prompt length, pool size, layers, upsampling dimension, loss weights) and on domain assumptions about ViLT transfer and about the compatibility of prompt retraining with the fixed-feature RLS derivation. No fundamentally new entity is postulated.

free parameters (8)
  • reconstruction weight λ = 0.01
    Balances reconstruction loss against classification loss; no sensitivity analysis reported beyond the chosen value.
  • regularization weight η = 1
    Regularizes the analytic least-squares solution; ablation (Table IV) shows 0.1 degrades accuracy.
  • upsampling size (dimension of W_up) = 15k
    Random-feature projection dimension; ablation (Table IV) shows 15k is best and 20k degrades.
  • prompt length = 8
    Number of tokens per prompt; ablation (Fig. 8) shows 8 is optimal.
  • number of prompt layers = 8
    Transformer layers where prompts are inserted; ablation (Fig. 6) shows more layers helps.
  • prompt pool size = 128
    Number of prompts per modality pool; ablation (Fig. 7) shows 128 is best.
  • learning rate = 1e-4
    AdamW learning rate for BP training of prompts and classifier; no tuning reported.
  • batch size = 4
    Batch size for BP training; no tuning reported.
assumptions (6)
  • standard math Woodbury matrix identity and standard regularized least squares are valid.
    Used in the Appendix to derive the recursive updates for R_k and W_k.
  • domain assumption Frozen ViLT pre-trained on MS-COCO and Visual Genome transfers to food and news image-text classification.
    Section IV.B freezes the backbone and relies entirely on its representations.
  • domain assumption Dummy inputs (empty pixels/strings) are an adequate representation of missing modalities, and the CLS-query reconstruction loss trains prompts to fill missing information.
    Sections III.A and III.C introduce this without justification beyond prior work [36].
  • domain assumption The block-diagonal label matrix Y_{1:k-1} (Eq. 11), with one block per task, correctly represents the joint training objective for disjoint class-incremental tasks.
    Adopted from ACIL-style analytic learning; central to the AL formulation.
  • ad hoc to paper Retraining prompts at each incremental step does not break the RLS recursion's fixed-feature assumption.
    Section III.D.1 trains prompts via BP at every task, while Theorem 1 requires a fixed feature mapping H; the paper does not analyze the resulting approximation error.
  • domain assumption A randomly initialized, frozen upsampling layer with ReLU yields a suitable random feature space for the analytic classifier.
    Section III.D.2 follows prior analytic learning works without additional justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PAL: Prompting Analytic Learning with Missing Modality for Multi-Modal Class-Incremental Learning." pith.science (2026). https://pith.science/paper/QS5W7RSN

@misc{pith2026250109352,
  author       = {Pith},
  title        = {Pith review of: PAL: Prompting Analytic Learning with Missing Modality for Multi-Modal Class-Incremental Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QS5W7RSN}},
  note         = {Machine review of arXiv:2501.09352}
}
read the original abstract

Multi-modal class-incremental learning (MMCIL) seeks to leverage multi-modal data, such as audio-visual and image-text pairs, thereby enabling models to learn continuously across a sequence of tasks while mitigating forgetting. While existing studies primarily focus on the integration and utilization of multi-modal information for MMCIL, a critical challenge remains: the issue of missing modalities during incremental learning phases. This oversight can exacerbate severe forgetting and significantly impair model performance. To bridge this gap, we propose PAL, a novel exemplar-free framework tailored to MMCIL under missing-modality scenarios. Concretely, we devise modality-specific prompts to compensate for missing information, facilitating the model to maintain a holistic representation of the data. On this foundation, we reformulate the MMCIL problem into a Recursive Least-Squares task, delivering an analytical linear solution. Building upon these, PAL not only alleviates the inherent under-fitting limitation in analytic learning but also preserves the holistic representation of missing-modality data, achieving superior performance with less forgetting across various multi-modal incremental scenarios. Extensive experiments demonstrate that PAL significantly outperforms competitive methods across various datasets, including UPMC-Food101 and N24News, showcasing its robustness towards modality absence and its anti-forgetting ability to maintain high incremental accuracy.

Figures

Figures reproduced from arXiv: 2501.09352 by the authors.

Figure 1
Figure 1. Illustrate of multi-modal class-incremental learning (MMCIL) with [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The basic backbone of PAL, including a pre-trained multi-modal [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The conceptual illustration of our proposed PAL framework during multi-modal CIL procedures, consisting of two steps of training. The first step [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Testing accuracy at each incremental step on the UPMC-Food101 dataset, when the missing rate [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Testing accuracy at each incremental step on the N24News dataset, when the missing rate [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 8
Figure 8. Figure 8: Model performance comparison when we vary the prompt length. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: The trade-off between stability and plasticity via AL module. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 51 canonical work pages

  1. [1]

    Deep audio-visual speech recognition,

    T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Deep audio-visual speech recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 44, no. 12, pp. 8717–8727, 2018

    work page 2018

  2. [2]

    Audio-visual efficient conformer for robust speech recognition,

    M. Burchi and R. Timofte, “Audio-visual efficient conformer for robust speech recognition,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , 2023, pp. 2258–2267

    work page 2023

  3. [3]

    End-to-end audiovisual speech recognition system with multitask learning,

    F. Tao and C. Busso, “End-to-end audiovisual speech recognition system with multitask learning,” IEEE Transactions on Multimedia, vol. 23, pp. 1–11, 2020

    work page 2020

  4. [4]

    AKVSR: Audio knowledge empowered visual speech recognition by compressing audio knowledge of a pretrained model,

    J. H. Yeo, M. Kim, J. Choi, D. H. Kim, and Y . M. Ro, “AKVSR: Audio knowledge empowered visual speech recognition by compressing audio knowledge of a pretrained model,” IEEE Transactions on Multimedia , 2024

    work page 2024

  5. [5]

    Revisiting classifier: Transferring vision-language models for video recognition,

    W. Wu, Z. Sun, and W. Ouyang, “Revisiting classifier: Transferring vision-language models for video recognition,” in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , vol. 37, no. 3, 2023, pp. 2847–2855

    work page 2023

  6. [6]

    Image-text embedding learning via visual and textual semantic reasoning,

    K. Li, Y . Zhang, K. Li, Y . Li, and Y . Fu, “Image-text embedding learning via visual and textual semantic reasoning,”IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 1, pp. 641–656, 2023

    work page 2023

  7. [7]

    Deep audio-visual beam- forming for speaker localization,

    X. Qian, Q. Zhang, G. Guan, and W. Xue, “Deep audio-visual beam- forming for speaker localization,” IEEE Signal Processing Letters , vol. 29, pp. 1132–1136, 2022

    work page 2022

  8. [8]

    Audio-visual cross- attention network for robotic speaker tracking,

    X. Qian, Z. Wang, J. Wang, G. Guan, and H. Li, “Audio-visual cross- attention network for robotic speaker tracking,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 550–562, 2022

    work page 2022

Show all 56 references
  1. [9]

    Localizing visual sounds the hard way,

    H. Chen, W. Xie, T. Afouras, A. Nagrani, A. Vedaldi, and A. Zisser- man, “Localizing visual sounds the hard way,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 16 867–16 876

    2021

  2. [10]

    Sound localization by self- supervised time delay estimation

    Z. Chen, D. F. Fouhey, and A. Owens, “Sound localization by self- supervised time delay estimation.” Springer, 2022, pp. 489–508

    2022

  3. [11]

    Deep multimodal clustering for unsupervised audiovisual learning,

    D. Hu, F. Nie, and X. Li, “Deep multimodal clustering for unsupervised audiovisual learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 9248–9257

    2019

  4. [12]

    Learning to localize sound source in visual scenes,

    A. Senocak, T.-H. Oh, J. Kim, M.-H. Yang, and I. S. Kweon, “Learning to localize sound source in visual scenes,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2018, pp. 4358–4366

    2018

  5. [13]

    icarl: Incremental classifier and representation learning,

    S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2001–2010

    2017

  6. [14]

    Learning without forgetting,

    Z. Li and D. Hoiem, “Learning without forgetting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 12, pp. 2935– 2947, 2017

    2017

  7. [15]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al. , “Overcoming catastrophic forgetting in neural networks,” Pro- ceedings of the National Academy of Sciences , vol. 114, no. 13, pp. 3521...

    2017

  8. [16]

    Less-forgetting learning in deep neural networks,

    H. Jung, J. Ju, M. Jung, and J. Kim, “Less-forgetting learning in deep neural networks,” in arXiv preprint arXiv:1607.00122 , 2016

    2016 arXiv

  9. [17]

    Class-incremental learning via deep model consolidation,

    J. Zhang, J. Zhang, S. Ghosh, D. Li, S. Tasci, L. Heck, H. Zhang, and C.- C. J. Kuo, “Class-incremental learning via deep model consolidation,” in Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2020, pp. 1131–1140. JOURNAL OF LATEX CLASS FILE...

    2020

  10. [18]

    Podnet: Pooled outputs distillation for small-tasks incremental learning,

    A. Douillard, M. Cord, C. Ollion, T. Robert, and E. Valle, “Podnet: Pooled outputs distillation for small-tasks incremental learning,” in Computer vision–ECCV 2020: 16th European conference, Glasgow, UK, August 23–28, 2020, proceedings, part XX 16 . Springer, 2020, pp. 86– 102

    2020

  11. [19]

    Adaptive aggregation networks for class- incremental learning,

    Y . Liu, B. Schiele, and Q. Sun, “Adaptive aggregation networks for class- incremental learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), 2021, pp. 2544–2553

    2021

  12. [20]

    Rmm: Reinforced memory management for class-incremental learning,

    ——, “Rmm: Reinforced memory management for class-incremental learning,” in Proceedings of Advances in Neural Information Processing Systems, vol. 34, 2021, pp. 3478–3490

    2021

  13. [21]

    Mnemonics training: Multi-class incremental learning without forgetting,

    Y . Liu, Y . Su, A.-A. Liu, B. Schiele, and Q. Sun, “Mnemonics training: Multi-class incremental learning without forgetting,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 12 245–12 254

    2020

  14. [22]

    Acil: Analytic class-incremental learning with absolute memorization and privacy protection,

    H. Zhuang, Z. Weng, H. Wei, R. Xie, K.-A. Toh, and Z. Lin, “Acil: Analytic class-incremental learning with absolute memorization and privacy protection,” in Proceedings of Advances in Neural Information Processing Systems(NeurIPS), vol. 35, 2022, pp. 11 602–11 614

    2022

  15. [24]

    Class-incremental grouping network for continual audio-visual learning,

    S. Mo, W. Pian, and Y . Tian, “Class-incremental grouping network for continual audio-visual learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision(ICCV) , 2023, pp. 7788– 7798

    2023

  16. [25]

    Audio-visual class-incremental learning,

    W. Pian, S. Mo, Y . Guo, and Y . Tian, “Audio-visual class-incremental learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision(ICCV), 2023, pp. 7799–7811

    2023

  17. [26]

    Reconstruct before query: Continual missing modality learning with decomposed prompt collaboration,

    S. Zhao, X. Zou, T. Yu, and H. Xu, “Reconstruct before query: Continual missing modality learning with decomposed prompt collaboration,” in arXiv preprint arXiv: 2403.11373 , 2024

    2024 arXiv

  18. [27]

    Mmal: Multi-modal analytic learning for exemplar-free audio-visual class incremental tasks,

    X. Yue, X. Zhang, Y . Chen, C. Zhang, M. Lao, H. Zhuang, X. Qian, and H. Li, “Mmal: Multi-modal analytic learning for exemplar-free audio-visual class incremental tasks,” in Proceedings of the 32nd ACM International Conference on Multimedia , 2024, p. 2428–2437

    2024

  19. [28]

    Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition,

    Z. Guo, T. Jin, and Z. Zhao, “Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition,” in Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024, pp. 1726–1736

    2024

  20. [29]

    Multimodal machine learning: A survey and taxonomy,

    T. Baltru ˇsaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 41, no. 2, pp. 423–443, 2019

    2019

  21. [30]

    Deep multimodal learning: A survey on recent advances and trends,

    D. Ramachandram and G. W. Taylor, “Deep multimodal learning: A survey on recent advances and trends,” IEEE Signal Processing Magazine, vol. 34, no. 6, pp. 96–108, 2017

    2017

  22. [31]

    Vilt: Vision-and-language transformer without convolution or region supervision,

    W. Kim, B. Son, and I. Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” in Proceedings of the International Conference on Machine Learning (ICML) , 2021

    2021

  23. [32]

    V ATT: Transformers for multimodal self-supervised learning from raw video, audio and text,

    H. Akbari, L. Yuan, R. Qian, W.-H. Chuang, S.-F. Chang, Y . Cui, and B. Gong, “V ATT: Transformers for multimodal self-supervised learning from raw video, audio and text,” in Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) , 2021

    2021

  24. [33]

    Multi-modal Trans- former for Video Retrieval,

    V . Gabeur, C. Sun, K. Alahari, and C. Schmid, “Multi-modal Trans- former for Video Retrieval,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2020

    2020

  25. [34]

    End-to-end referring video object segmentation with multimodal transformers,

    A. Botach, E. Zheltonozhskii, and C. Baskin, “End-to-end referring video object segmentation with multimodal transformers,” in Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 4975–4985

    2022

  26. [35]

    Are multi- modal transformers robust to missing modality?

    M. Ma, J. Ren, L. Zhao, D. Testuggine, and X. Peng, “Are multi- modal transformers robust to missing modality?” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 18 156–18 165

    2022

  27. [36]

    Multimodal prompt- ing with missing modalities for visual recognition,

    Y .-L. Lee, Y .-H. Tsai, W.-C. Chiu, and C.-Y . Lee, “Multimodal prompt- ing with missing modalities for visual recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 14 943–14 952

    2023

  28. [37]

    Towards good practices for missing modality robust action recognition,

    S. Woo, S. Lee, Y . Park, M. A. Nugroho, and C. Kim, “Towards good practices for missing modality robust action recognition,” 2023

    2023

  29. [38]

    Missing modality imagination network for emotion recognition with uncertain missing modalities,

    J. Zhao, R. Li, and Q. Jin, “Missing modality imagination network for emotion recognition with uncertain missing modalities,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Langu...

    2021

  30. [39]

    Towards robust multimodal prompting with missing modalities,

    J. Jang, Y . Wang, and C. Kim, “Towards robust multimodal prompting with missing modalities,” in Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024, pp. 8070– 8074

    2024

  31. [40]

    Learning to prompt for continual learning,

    Z. Wang, Z. Zhang, C.-Y . Lee, H. Zhang, R. Sun, X. Ren, G. Su, V . Perot, J. Dy, and T. Pfister, “Learning to prompt for continual learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, p. 139–149

    2022

  32. [41]

    Dualprompt: Complementary prompting for rehearsal-free continual learning,

    Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y . Lee, X. Ren, G. Su, V . Perot, J. Dy, and T. Pfister, “Dualprompt: Complementary prompting for rehearsal-free continual learning,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2022, p. 631–648

    2022

  33. [42]

    Rainbow memory: Continual learning with a memory of diverse samples,

    J. Bang, H. Kim, Y . Yoo, J.-W. Ha, and J. Choi, “Rainbow memory: Continual learning with a memory of diverse samples,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 8218–8227

    2021

  34. [43]

    Memory-efficient class- incremental learning for image classification,

    H. Zhao, H. Wang, Y . Fu, F. Wu, and X. Li, “Memory-efficient class- incremental learning for image classification,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 10, pp. 5966–5977, 2021

    2021

  35. [44]

    End-to-end incremental learning,

    F. M. Castro, M. J. Mar ´ın-Jim´enez, N. Guil, C. Schmid, and K. Alahari, “End-to-end incremental learning,” in Proceedings of the European Conference on Computer Vision , 2018, pp. 233–248

    2018

  36. [45]

    Learning a unified classifier incrementally via rebalancing,

    S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin, “Learning a unified classifier incrementally via rebalancing,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 831–839

    2019

  37. [46]

    Generating instance-level prompts for rehearsal-free continual learning,

    D. Jung, D. Han, J. Bang, and H. Song, “Generating instance-level prompts for rehearsal-free continual learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023, pp. 11 813–11 823

    2023

  38. [47]

    Blockwise recursive moore– penrose inverse for network learning,

    H. Zhuang, Z. Lin, and K.-A. Toh, “Blockwise recursive moore– penrose inverse for network learning,” IEEE Trans on Systems, Man, and Cybernetics: Systems , vol. 52, no. 5, pp. 3237–3250, 2021

    2021

  39. [48]

    A pseudoinverse learning algorithm for feedforward neural networks with stacked generalization applications to software reliability growth data,

    P. Guo and M. R. Lyu, “A pseudoinverse learning algorithm for feedforward neural networks with stacked generalization applications to software reliability growth data,” Neurocomputing, vol. 56, pp. 101–121, 2004

    2004

  40. [49]

    Gkeal: Gaussian kernel embedded analytic learning for few-shot class incremental task,

    H. Zhuang, Z. Weng, R. He, Z. Lin, and Z. Zeng, “Gkeal: Gaussian kernel embedded analytic learning for few-shot class incremental task,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 7746–7755

    2023

  41. [50]

    Coda-prompt: Contin- ual decomposed attention-based prompting for rehearsal-free continual learning,

    J. S. Smith, L. Karlinsky, V . Gutta, P. Cascante-Bonilla, D. Kim, A. Arbelle, R. Panda, R. Feris, and Z. Kira, “Coda-prompt: Contin- ual decomposed attention-based prompting for rehearsal-free continual learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision a...

    2023

  42. [51]

    Ds-al: A dual-stream analytic learning for exemplar-free class-incremental learn- ing,

    H. Zhuang, R. He, K. Tong, Z. Zeng, C. Chen, and Z. Lin, “Ds-al: A dual-stream analytic learning for exemplar-free class-incremental learn- ing,” in Proceedings of the AAAI Conference on Artificial Intelligence , 2024, pp. 17 237–17 244

    2024

  43. [52]

    Recipe recognition with large multimodal food dataset,

    X. Wang, D. Kumar, N. Thome, M. Cord, and F. Precioso, “Recipe recognition with large multimodal food dataset,” in Proceedings of the IEEE International Conference on Multimedia & Expo Workshops (ICMEW), 2015, pp. 1–6

    2015

  44. [53]

    N24News: A new dataset for multimodal news classification,

    Z. Wang, X. Shan, X. Zhang, and J. Yang, “N24News: A new dataset for multimodal news classification,” in Proceedings of the Thirteenth Language Resources and Evaluation Conference , 2022, pp. 6768–6775

    2022

  45. [54]

    Vilt: Vision-and-language transformer without convolution or region supervision,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “Vilt: Vision-and-language transformer without convolution or region supervision,” in Proceedings of the Internationa...

    2021

  46. [55]

    Microsoft COCO: common objects in context,

    T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Doll´ar, and C. L. Zitnick, “Microsoft COCO: common objects in context,” in Proceedings of the Computer Vision–ECCV 2014, 2014

    2014

  47. [56]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations,

    R. Krishna, Y . Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y . Kalantidis, L. Li, D. A. Shamma, M. S. Bernstein, and L. Fei-Fei, “Visual genome: Connecting language and vision using crowdsourced dense image annotations,” International Journal of Computer Vision , 2016

    2016

  48. [57]

    Fixing weight decay regularization in adam,

    I. Loshchilov and F. Hutter, “Fixing weight decay regularization in adam,” in Proceedings of the International Conference on Learning Representations (ICLR), 2017. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 11 APPENDIX PROOF OF THEOREM 1 Here we give the proof o...

    2017

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.