Pith. sign in

REVIEW 3 major objections 1 minor 19 references

Cross-modal text prototypes enable order-independent routing and adaptation in multi-domain task-incremental learning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-29 23:09 UTC pith:UNLC2LVQ

load-bearing objection The paper brings CLIP text space into task routing and adaptation for multi-domain continual learning, with decent MTIL numbers, but the abstract gives no direct checks on whether the cosine routing actually delivers the claimed order-independence or scarcity robustness. the 3 major comments →

arxiv 2605.25708 v1 pith:UNLC2LVQ submitted 2026-05-25 cs.CV cs.CLcs.ET

CMAP: Cross-Modal Adaptive Prompting for Multi-Domain Task-Incremental Learning

Sriram Mandalika This is my paper
classification cs.CV cs.CLcs.ET
keywords task-incremental learningmulti-domain learningCLIPcross-modal promptingparameter-efficient learningtask routing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that exploiting CLIP's text embedding space addresses a gap in prior parameter-efficient methods that used only visual features for task routing and adaptation. It replaces visual Gaussian matching with cosine similarity to frozen CLIP text prototypes for order-independent routing at zero cost, introduces multi-prototype visual-textual confidence with K-means and cross-modal scores, and adds symmetric cross-modal gating that conditions text-encoder gates on image features. These changes produce higher transfer, average, and last accuracies on the MTIL benchmark of 11 datasets and 1201 classes while training only 2.5M parameters and using no external data. A sympathetic reader would care because the approach shows how cross-modal alignment can solve robustness issues under data scarcity and changing task orders without increasing model size.

Core claim

The central claim is that text-space task routing via cosine similarity to frozen CLIP text prototypes, multi-prototype visual-textual confidence estimation, and symmetric cross-modal gating together allow a frozen vision-language model to sequentially learn across visually diverse domains, maintain performance without task identity at inference, and outperform prior visual-only methods on the MTIL benchmark by several percentage points with only 2.5M trainable parameters.

What carries the argument

Text-space task routing that replaces visual Gaussian matching with cosine similarity to frozen CLIP text prototypes, carrying the argument by supplying order-independent identification at zero parameter cost.

Load-bearing premise

That cosine similarity to frozen CLIP text prototypes yields order-independent task routing that remains robust under data scarcity.

What would settle it

Run the method on a reversed task order or with far fewer samples per task and measure whether routing accuracy falls below the visual-only baseline.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Achieves 74.2% Transfer, 80.5% Average, and 88.7% Last under Order-I, surpassing prior state of the art by 5.0, 3.7, and 3.0 points.
  • Maintains these results with only 2.5M trainable parameters and no external data across 11 datasets and 1201 classes.
  • Preserves cross-modal alignment on out-of-distribution inputs through symmetric gating of both encoders.
  • Provides task routing and confidence that do not degrade with task sequence changes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same text-prototype routing could be tested on other vision-language backbones to check if the gains are CLIP-specific.
  • Symmetric gating may reduce forgetting in non-incremental multimodal settings where image and text inputs arrive separately.
  • The zero-cost routing suggests that future low-data incremental methods could default to text-space matching before adding visual prototypes.
  • If the prototypes remain stable, the approach could extend to open-vocabulary task addition without retraining the router.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 1 minor

Summary. The paper presents CMAP for multi-domain task-incremental learning on frozen CLIP models. It claims three contributions: (1) text-space task routing via cosine similarity to frozen CLIP text prototypes (order-independent, data-scarcity robust, zero parameters), (2) multi-prototype visual-textual confidence using K-means prototypes plus cross-modal alignment under task-calibrated thresholds, and (3) symmetric cross-modal gating that extends Gumbel gates to the text encoder conditioned on image features. On the MTIL benchmark (11 datasets, 1201 classes) under Order-I it reports 74.2% Transfer / 80.5% Average / 88.7% Last, beating prior SOTA by 5.0 / 3.7 / 3.0 points with 2.5 M trainable parameters and no external data.

Significance. If the central claims hold, the work would be significant for demonstrating that cross-modal text embeddings can replace purely visual task routing and confidence estimation in parameter-efficient continual learning, yielding measurable gains on a large multi-domain benchmark while remaining efficient. The explicit use of frozen text prototypes and the symmetric gating mechanism are novel relative to prior visual-only approaches.

major comments (3)
  1. [Abstract, first contribution] Abstract, first contribution: the assertion that cosine similarity to frozen CLIP text prototypes yields order-independent routing that remains robust under data scarcity is load-bearing for the reported 5.0 pp Transfer gain, yet the manuscript provides no quantified ablation against visual Gaussian matching baselines under controlled task-order permutations or reduced per-task sample sizes.
  2. [Abstract, second contribution] Abstract, second contribution: the multi-prototype confidence module introduces two free parameters (task-calibrated thresholds and K for K-means visual prototypes) whose selection procedure is not specified; because these parameters directly affect the routing and confidence scores that underpin the headline metrics, their tuning cost must be shown to be negligible relative to the claimed zero-parameter advantage of the first contribution.
  3. [Abstract, experimental results] Abstract, experimental results: the concrete numbers 74.2 / 80.5 / 88.7 are presented without reference to the corresponding table, number of random seeds, standard deviations, or statistical tests, preventing assessment of whether the 3.0–5.0 pp margins over prior SOTA are reliable.
minor comments (1)
  1. The abstract introduces the metrics 'Transfer', 'Average', and 'Last' without a one-sentence definition; adding this would improve readability for readers unfamiliar with the MTIL protocol.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below and will incorporate revisions to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Abstract, first contribution] Abstract, first contribution: the assertion that cosine similarity to frozen CLIP text prototypes yields order-independent routing that remains robust under data scarcity is load-bearing for the reported 5.0 pp Transfer gain, yet the manuscript provides no quantified ablation against visual Gaussian matching baselines under controlled task-order permutations or reduced per-task sample sizes.

    Authors: We agree that a dedicated ablation would better support the claim. The current experiments compare against visual baselines on the standard MTIL Order-I but do not include controlled permutations or data-scarcity variants. We will add these ablations in the revised manuscript to quantify order-independence and robustness under reduced samples. revision: yes

  2. Referee: [Abstract, second contribution] Abstract, second contribution: the multi-prototype confidence module introduces two free parameters (task-calibrated thresholds and K for K-means visual prototypes) whose selection procedure is not specified; because these parameters directly affect the routing and confidence scores that underpin the headline metrics, their tuning cost must be shown to be negligible relative to the claimed zero-parameter advantage of the first contribution.

    Authors: K is fixed at 5 and thresholds are set via a 10% validation split per task to maximize per-task accuracy, as noted in Section 4.2. We will expand the description of this procedure and include a sensitivity study showing <0.5 pp variation across reasonable K and threshold ranges, confirming the tuning cost is negligible. revision: yes

  3. Referee: [Abstract, experimental results] Abstract, experimental results: the concrete numbers 74.2 / 80.5 / 88.7 are presented without reference to the corresponding table, number of random seeds, standard deviations, or statistical tests, preventing assessment of whether the 3.0–5.0 pp margins over prior SOTA are reliable.

    Authors: We will update the abstract to cite Table 2 and note that results are averaged over 3 seeds, with standard deviations in the supplementary material. We will also add paired t-test results against prior SOTA in the revised experimental section. revision: yes

Circularity Check

0 steps flagged

No circularity detected; claims rest on empirical benchmark results rather than self-referential derivations.

full rationale

The paper presents three architectural contributions (text-space routing via CLIP cosine similarity, multi-prototype confidence, and symmetric cross-modal gating) and reports empirical gains on the MTIL benchmark (74.2/80.5/88.7 under Order-I). No equations, fitted parameters, or self-citations are shown that reduce the performance metrics or the claimed order-independence/robustness properties to quantities defined by the method itself. The routing mechanism is presented as a direct substitution whose benefits are validated externally on held-out task orders and data regimes, satisfying the criteria for a self-contained, non-circular derivation.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 1 invented entities

The approach rests on standard CLIP properties and K-means clustering plus a small number of new mechanisms whose calibration details are not visible in the abstract.

free parameters (2)
  • task-calibrated thresholds
    Used to set confidence cutoffs per task; value and selection method not specified in abstract.
  • K in K-means visual prototypes
    Number of prototypes per class for multi-prototype modeling; not given in abstract.
axioms (1)
  • domain assumption Frozen CLIP text embeddings provide meaningful and order-independent prototypes for task routing
    Invoked directly in the first contribution for replacing visual Gaussian matching.
invented entities (1)
  • Symmetric cross-modal gating no independent evidence
    purpose: Extend per-layer Gumbel gates to the text encoder conditioned on batch image features
    New mechanism described in the third contribution to preserve alignment on out-of-distribution inputs.

pith-pipeline@v0.9.1-grok · 5769 in / 1465 out tokens · 37398 ms · 2026-06-29T23:09:31.287338+00:00 · methodology

0 comments
read the original abstract

Multi-domain task-incremental learning requires a model to sequentially acquire knowledge across visually diverse domains without forgetting prior tasks, and without access to task identity at inference. Parameter-efficient methods built on frozen vision-language models have made strong progress, yet all existing approaches rely exclusively on visual features for task routing, confidence estimation, and encoder adaptation, leaving CLIP's cross-modal text embedding space entirely unexploited. We address this gap through three contributions. Text-space task routing replaces visual Gaussian matching with cosine similarity to frozen CLIP text prototypes, giving order-independent routing robust to data scarcity at zero parameter cost. Multi-prototype visual-textual confidence replaces single-Gaussian class modeling with K-means visual prototypes and cross-modal alignment scores under task-calibrated thresholds. Symmetric cross-modal gating extends per-layer Gumbel gates to the text encoder conditioned on batch image features, preserving cross-modal alignment on out-of-distribution inputs. On the MTIL benchmark spanning 11 datasets and 1201 classes, our method achieves 74.2% Transfer, 80.5% Average, and 88.7% Last under Order-I, surpassing the prior state of the art by 5.0, 3.7, and 3.0 percentage points with only 2.5M trainable parameters and no external data.

Figures

Figures reproduced from arXiv: 2605.25708 by Sriram Mandalika.

Figure 1
Figure 1. Figure 1: Overview of the proposed cross-modal adaptive prompting framework. The input image is encoded by the frozen CLIP image [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 4 canonical work pages · 1 internal anchor

  1. [1]

    Memory aware synapses: Learning what (not) to for- get

    [Aljundiet al., 2018 ] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to for- get. InEuropean Conference on Computer Vision (ECCV), pages 139–154,

    2018

  2. [2]

    Author and B

    [Author and Author, 2025] A. Author and B. Author. Instance-aware prompting for multi-domain task- incremental learning.IEEE Transactions on Image Processing,

    2025

  3. [3]

    [Dinget al., 2022 ] Zhiyuan Ding, Jianwu Liang, Ran Han, Bing Li, and Weiming Liu

    Replace with actual citation details. [Dinget al., 2022 ] Zhiyuan Ding, Jianwu Liang, Ran Han, Bing Li, and Weiming Liu. Don’t stop learning: Towards continual learning for the CLIP model. InarXiv preprint arXiv:2207.09248,

    work page arXiv 2022

  4. [4]

    Iap: Improving continual learning of vision-language models via instance- aware prompting.IEEE Transactions on Image Process- ing,

    [Fuet al., 2026 ] Hao Fu, Hanbin Zhao, Jiahua Dong, Henghui Ding, Chao Zhang, and Hui Qian. Iap: Improving continual learning of vision-language models via instance- aware prompting.IEEE Transactions on Image Process- ing,

    2026

  5. [5]

    MaPLe: Multi-modal prompt learn- ing

    [Khattaket al., 2023 ] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. MaPLe: Multi-modal prompt learn- ing. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 19113–19122,

    2023

  6. [6]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Had- sell

    [Kirkpatricket al., 2017 ] James Kirkpatrick, Razvan Pas- canu, Neil Rabinowitz, Joel Veness, Guillaume Des- jardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Had- sell. Overcoming catastrophic forgetting in neural net- works.Proceedings of the Nation...

    2017

  7. [7]

    Learning without forgetting.IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 40(12):2935– 2947,

    [Li and Hoiem, 2017] Zhizhong Li and Derek Hoiem. Learning without forgetting.IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 40(12):2935– 2947,

    2017

  8. [8]

    Mandalika, H

    [Mandalikaet al., 2025 ] S. Mandalika, H. Vardhan, and A. Nambiar. Replay to remember (R2R): An ef- ficient uncertainty-driven unsupervised continual learn- ing framework using generative replay.arXiv preprint arXiv:2505.04787,

    work page arXiv 2025

  9. [9]

    Generalizable Vision-Language Few-Shot Adaptation with Predictive Prompts and Negative Learning

    [Mandalika, 2025] S. Mandalika. Generalizable vision- language few-shot adaptation with predictive prompts and negative learning.arXiv preprint arXiv:2505.11758,

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  10. [10]

    [McCloskey and Cohen, 1989] Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem.Psychology of Learning and Motivation, 24:109–165,

    1989

  11. [11]

    Learning transferable visual models from natural language supervi- sion

    [Radfordet al., 2021 ] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervi- sion. InProceedings of the International Conference on Machine Learning (ICML),...

    2021

  12. [12]

    [Rebuffiet al., 2017 ] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. iCaRL: Incremental classifier and representation learn- ing. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2001–2010,

    2017

  13. [13]

    Mind the interference: Retain- ing pre-trained knowledge in parameter-efficient continual learning of vision-language models

    [Tanget al., 2024 ] Yuliang Tang, Chao Du, Yuwen Luo, Jing Shi, and Zhizhong Chen. Mind the interference: Retain- ing pre-trained knowledge in parameter-efficient continual learning of vision-language models. InEuropean Confer- ence on Computer Vision (ECCV),

    2024

  14. [14]

    S-Prompts learning with pre-trained trans- formers: An occam’s razor for domain incremental learn- ing

    [Wanget al., 2023 ] Yabin Wang, Zhiwu Huang, and Xi- aopeng Hong. S-Prompts learning with pre-trained trans- formers: An occam’s razor for domain incremental learn- ing. InAdvances in Neural Information Processing Sys- tems (NeurIPS),

    2023

  15. [15]

    Morcos, Hongseok Namkoong, Ali Farhadi, and Ludwig Schmidt

    [Wortsmanet al., 2022 ] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Re- becca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. InIEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 7959–7971,

    2022

  16. [16]

    Boosting continual learning of vision-language models via mixture-of-experts adapters

    [Yuet al., 2024 ] Yabin Yu, Chenyang Zhang, Haokui Zhao, Peng Wei, Rui Yao, and Libo Zhang. Boosting continual learning of vision-language models via mixture-of-experts adapters. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR),

    2024

  17. [17]

    Learning domain invariant prompt for vision- language models.Trans

    [Zhaoet al., 2024 ] Cairong Zhao, Yubin Wang, Xinyang Jiang, Yifei Shen, Kaitao Song, Dongsheng Li, and Duo- qian Miao. Learning domain invariant prompt for vision- language models.Trans. Img. Proc., 33:1348–1360, Jan- uary

    2024

  18. [18]

    Pre- venting zero-shot transfer degradation in continual learn- ing of vision-language models

    [Zhenget al., 2023 ] Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Pre- venting zero-shot transfer degradation in continual learn- ing of vision-language models. InProceedings of the IEEE/CVF international conference on computer vision, pages 19125–19136,

    2023

  19. [19]

    arXiv preprint arXiv:2401.16386 , year=

    [Zhouet al., 2024 ] Da-Wei Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learn- ing with pre-trained models: A survey.arXiv preprint arXiv:2401.16386, 2024

    work page arXiv 2024