REVIEW 4 major objections 6 minor 30 references
Multiple Stochastic Prompt Tuning for Few-shot Adaptation under Extreme Domain Shift
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read MIST claims that two stochastic text prompts per class—one anchored to the frozen 'A photo' prompt and one fully learned—let a frozen CLIP adapt to extreme visual and semantic shifts with only a few labeled examples from all classes at…
desk verdict A plausible incremental advance in few-shot CLIP adaptation, but an underspecified visual-prompt inference path currently blocks independent reproduction of the headline results. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a pair of stochastic text prompts per class inside a deep multimodal prompt-tuning adapter. For each class, prompt one is sampled as $\theta^1_t \sim \mathcal{N}(\bar{\mu}_0, \sigma_0)$ with $\bar{\mu}_0$ fixed to the 'A photo' embedding and $\sigma_0$ learnable, while prompt two is sampled as $\theta^2_t \sim \mathcal{N}(\mu_1, \sigma_1)$ with both parameters learnable. The text prompts pass through a projection $f_\phi$ to produce visual prompts, and the reparameterization trick $\theta = \mu + \mathcal{N}(0, I) \odot \sigma$ lets gradients reach the distribution parameters. Training minimizes $\mathcal{L}_{mp}$ (assigning each image to its closest text prototype) plus $\mathcal{L}_{reg}$ (pulling the image toward the centroid of its class's two prompts), and inference takes the maximum logit over the two prompts for each class.
What would settle it
Construct or find a target class whose embedding space splits into three well-separated visual subgroups; if MIST with its fixed two prompts per class cannot match a three-prompt variant on that class, the two-prototype capacity assumption is falsified.
Extended reading notes
Core claim
The paper claims that the degraded few-shot performance of prompt-tuned CLIP under extreme domain and semantic shift comes mainly from modeling each class with a single text prompt, and that two stochastic text prototypes per class remove that limitation. MIST keeps the CLIP encoders frozen and optimizes only the Gaussian prompt parameters plus a projection layer, with the first prompt sampled from a Gaussian whose mean is fixed to the pretrained 'A photo' embedding and the second sampled from a fully learnable Gaussian. Each image is assigned to the nearer of its class's two text embeddings during training, and a regularization term pulls the image toward the centroid of the two, stopping the prompts from collapsing onto one mode. On the BSCDFSL benchmark the authors report average gains of 3.19%, 2.01%, 1.79%, 3.19%, and 1.60% over the best prior methods at 1, 2, 4, 8, and 16 shots, and lower variance across seeds.
Load-bearing premise
Two stochastic text prototypes per class are enough to cover the visual modes a class can have under extreme domain shift, and the few labeled support images reveal those modes.
Editorial extensions
If this is right
- The largest reported gain comes at 1-shot, supporting the paper's claim that sampling around a fixed pretrained prompt anchors CLIP to its prior knowledge and suppresses overfitting when each class has a single image.
- Because all classes are trained together rather than in sampled episodes, the method applies directly to deployment settings where the complete target label set is known at once.
- The class-imbalanced experiment and the worst-class-bin analysis imply that the two-stochastic-prompt design improves coverage of difficult and visually similar classes, not just average accuracy.
- The acknowledged weaker result on grayscale chest X-rays indicates that visual prompts can hurt when the shift is largely a change in low-level image statistics, so the method is not uniformly superior across all domain shifts.
Reading between the lines
- Editorial inference: if two prompts per class are the reason for the gains, a data-dependent number of prompts chosen by clustering the support-set embeddings could push accuracy further; the paper itself shows performance falls after three prompts, so the fixed choice of two may leave modes uncaptured in some classes.
- Editorial inference: the combination of one anchored prompt and one fully learned prompt is a general regularization recipe that could transfer to adapter tuning or to other frozen foundation models, not just CLIP text prompts.
- Editorial inference: since MIST trains without a validation set and without episodes, it could be paired with validation-free model selection to address the separate problem of choosing hyperparameters in few-shot adaptation, a direction this paper does not explore.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MIST, a few-shot CLIP adaptation method for target datasets with large visual appearance and label-semantic shifts, in a non-episodic setting where all classes are present. MIST augments a multimodal deep-prompting base network (MaPLe-style) with two stochastic text prompts per class: one sampled from a Gaussian with fixed mean 'A photo' and learnable variance, and one sampled from a fully learnable Gaussian. Visual prompts are obtained from text prompts through a projection layer. Training uses a per-image loss that assigns the image to the closest of its two class prompts and a centroid-similarity regularizer. Experiments on the BSCDFSL benchmark (EuroSAT, ISIC, Plant Disease, ChestX) at 1, 2, 4, 8, and 16 shots report average accuracy gains over CoOp, TaskRes, MaPLe, PromptSRC, CLAP, and TCP, plus a class-imbalance study, variance analysis, and ablations.
Significance. If the claims are correct, MIST addresses a practical gap: adapting CLIP to non-episodic few-shot settings with extreme domain and semantic shift while all classes are handled at once. The combination of multiple prompts per class and stochastic prompt sampling is a sensible way to regularize scarce data, and the paper provides useful ablations (Table 5), a class-imbalance experiment (Table 3), and variance analysis (Table 4). The reported average gains in Table 2 are internally consistent with the stated margins. However, two load-bearing points are unresolved: the class-specific visual-prompt path is underspecified, and the regularizer claimed to enforce prompt diversity does not appear to implement that goal. These issues must be resolved before the empirical claims can be accepted.
major comments (4)
- [Sec. 3.1 and Sec. 4.2, Eq. (2)] The visual-prompt pathway is undefined once text prompts become class-specific. Section 3.1 defines the visual prompts as θv = fφ(θt) for a single text prompt. In Section 4.2, the text prompt is replaced by two class-specific prompts θi_t per class in Eq. (2), but the paper never states whether θv is class-specific or shared. If θv is class-specific, the image embedding z̃v used in Eqs. (3)-(4) cannot be computed for a test image without knowing its class; the method would need 2C image-encoder forward passes per test image or it would leak the label at inference. If θv is shared, the paper does not say which prompt is passed through fφ. The text and Figure 3 are ambiguous and no code is provided. This blocks reproduction of Table 2 and must be resolved by specifying the exact training and inference image-encoding path.
- [Sec. 4.2, Eq. (5)] The regularizer Lreg does not enforce diversity between the two prompts. Minimizing -sim(z̃v, (z̃1_t+z̃2_t)/2) encourages the image embedding to be close to the centroid of the two class text embeddings, but it contains no term that penalizes similarity between z̃1_t and z̃2_t. If the two prompts collapse to the same embedding, the centroid is that embedding and the regularizer simply rewards closeness to it; it imposes no penalty on the collapse. For an image close to one prompt, moving the other prompt toward that image tends to increase sim(z̃v, centroid), so Eq. (5) can actively encourage collapse. The stated purpose 'prevents the two prompts from collapsing' is therefore not implemented by the given objective, and the ablation in Table 5 does not isolate a diversity effect. Please add an explicit repulsion/diversity term or reframe what Lreg actually does.
- [Sec. 5.1 and Limitations] The claim of superiority is stated too broadly. The 'consistent average gains' in Section 5.1 are average gains, not per-dataset wins: Table 2 shows MIST below the best baseline in several cells, e.g., 1-shot PDisease (50.27 vs MaPLe 51.53), 1-shot ChestX (17.10 vs CoOp 22.83), 2-shot ISIC (36.37 vs TCP 36.87), 4-shot ChestX (18.67 vs MaPLe 19.73), and 16-shot ChestX (14.77 vs CoOp 18.80). The Limitations section states 'MIST outperforms state-of-the-art methods across all datasets' and then immediately says its performance 'slightly drops on the grayscale ChestX dataset,' which is internally inconsistent. Please qualify the claims to refer to average performance and acknowledge the per-dataset exceptions.
- [Sec. 4.2, Inference] The inference-time sampling protocol is underspecified. The paper says that at inference 'we can sample weights' for the two text prompts and 'take the maximum logit among the two text prompts,' but it does not state how many samples are drawn per class or whether predictions are averaged over multiple samples. Because the test-time text embeddings are random draws from learned Gaussians, the reported accuracies in Table 2 are not reproducible without fixing a seed, a sample count, or an averaging rule. Please specify the exact inference protocol and, ideally, report the variance due to inference sampling.
minor comments (6)
- [Sec. 3, Eq. (7)] Equation (7) writes the expectation over D_tgt, but only the support set is available for training; please write the empirical expectation over the support set S.
- [Sec. 4, introductory paragraph] There is a typo: 'descrie' should be 'describe.' Also, 'upto' in the implementation details should be 'up to.'
- [Sec. 4.2, Inference] The phrase 'take the maximum logit among the two text prompts' is ambiguous because each text prompt is stochastic; please define the class score explicitly in terms of the sampled text embeddings.
- [Sec. 5.2 and Figure 5] Figure 5 would be easier to interpret with numeric x-axis values and error bars; the statement that accuracy 'decreases after a point' is not quantitative.
- [Sec. 5.2, Table 4] Variance across seeds is reported only for EuroSAT and ISIC; please also report it for PDisease and ChestX, or justify why those datasets are omitted.
- [Sec. 2] The related-work discussion mentions Bayesian prompt learning [5] but does not clarify how MIST differs from it in the stochastic-prompt design; a brief comparison would help position the contribution.
Circularity Check
No significant circularity: MIST is an empirical few-shot adaptation method evaluated on held-out test sets.
full rationale
MIST is an empirical adaptation method. The paper trains Gaussian prompt parameters (mu_1, sigma_0, sigma_1) and the projection f_phi on the support set, then reports accuracy on the full test set. The training objective in Eqs. (3)-(7) is a standard supervised loss; the reported accuracies in Table 2 are independent test-set evaluations against baselines run with official code. No theoretical constant is derived, and no reported prediction is statistically forced by a fitted parameter. The design choice of two prompts per class is motivated by Table 1 and by external mixture-prototype work [1,2], not by a self-citation chain. The only self-citation [10] appears in related work as an example of stochastic classifiers and is not load-bearing. The underspecification of whether visual prompts are class-specific (Sec. 3.1 vs 4.2) is a reproducibility and correctness risk, not a circularity, because it does not make the accuracy result equivalent to an input by construction.
Assumptions & free parameters
free parameters (4)
- mu_1 =
not reported
- sigma_0, sigma_1 =
not reported
- projection layer f_phi =
not reported
- prompt length, prompts per class, deep prompt depth, epochs, learning rate, batch size =
2, 2, 9, 150, 0.0035, 4
assumptions (5)
- domain assumption CLIP's text and image encoders produce comparable embeddings, so cosine similarity between image and text prototypes is a valid classification score.
- domain assumption Under extreme domain shift, classwise visual features are multimodal and two prompts per class can cover the main modes.
- ad hoc to paper The frozen "A photo" embedding is a useful fixed mean for one prompt distribution even under semantic shift.
- standard math Gaussian reparameterization gives unbiased differentiable gradients for sampling.
- domain assumption The few-shot support set is representative of each class in the target domain.
Cite this review
Pith. "Pith review of Multiple Stochastic Prompt Tuning for Few-shot Adaptation under Extreme Domain Shift." pith.science (2026). https://pith.science/paper/ZATHFPLS
@misc{pith2026250603926,
author = {Pith},
title = {Pith review of: Multiple Stochastic Prompt Tuning for Few-shot Adaptation under Extreme Domain Shift},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZATHFPLS}},
note = {Machine review of arXiv:2506.03926}
}
read the original abstract
Foundation Vision-Language Models (VLMs) like CLIP exhibit strong generalization capabilities due to large-scale pretraining on diverse image-text pairs. However, their performance often degrades when applied to target datasets with significant distribution shifts in both visual appearance and class semantics. Recent few-shot learning approaches adapt CLIP to downstream tasks using limited labeled data via adapter or prompt tuning, but are not specifically designed to handle such extreme domain shifts. Conversely, some works addressing cross-domain few-shot learning consider such domain-shifted scenarios but operate in an episodic setting with only a few classes per episode, limiting their applicability to real-world deployment, where all classes must be handled simultaneously. To address this gap, we propose a novel framework, MIST (Multiple Stochastic Prompt Tuning), for efficiently adapting CLIP to datasets with extreme distribution shifts using only a few labeled examples, in scenarios involving all classes at once. Specifically, we introduce multiple learnable prompts per class to effectively capture diverse modes in visual representations arising from distribution shifts. To further enhance generalization, these prompts are modeled as learnable Gaussian distributions, enabling efficient exploration of the prompt parameter space and reducing overfitting caused by limited supervision. Extensive experiments and comparisons with state-of-the-art methods demonstrate the effectiveness of the proposed framework.
Figures
Reference graph
Works this paper leans on
-
[5]
Bayesian prompt learning for image-language model generaliza- tion
Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor G Turrisi da Costa, Cees GM Snoek, Georgios Tzimiropoulos, and Brais Martinez. Bayesian prompt learning for image-language model generaliza- tion. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 15237–15246, 2023
work page 2023
-
[15]
Prompt distribution learning
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5206–5215, 2022
2022
-
[1]
Mixture-based feature space learning for few-shot image classification
Arman Afrasiyabi, Jean-Franc ¸ois Lalonde, and Chris- tian Gagn ´e. Mixture-based feature space learning for few-shot image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9041–9051, 2021
work page 2021
-
[2]
Infinite mixture prototypes for few-shot learning
Kelsey Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. In International conference on machine learn- ing, pages 232–241. PMLR, 2019
work page 2019
-
[3]
Weight uncertainty in neural net- work
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural net- work. In International conference on machine learning, pages 1613–1622. PMLR, 2015
work page 2015
-
[4]
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the interna- tional skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368, 2019
arXiv 2018
-
[6]
Clip-adapter: Better vision-language models with fea- ture adapters
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with fea- ture adapters. International Journal of Computer Vision, 132(2):581–595, 2024
2024
-
[7]
A broader study of cross-domain few- shot learning
Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study of cross-domain few- shot learning. In Computer Vision–ECCV 2020: 16th Eu- ropean Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16 , pages 124–141. Springer, 2020
work page 2020
Show all 30 references
-
[8]
Eurosat: A novel dataset and deep learn- ing benchmark for land use and land cover classification
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learn- ing benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Obser- vations and Remote Sensing, 12(7):2217–2226, 2019
2019
-
[9]
Scaling up visual and vision- language representation learning with noisy text super- vision
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- language representation learning with noisy text super- vision. In International conference on machine learning, pages 4904–49...
2021
-
[10]
S3c: Self-supervised stochastic classifiers for few-shot class-incremental learning
Jayateja Kalla and Soma Biswas. S3c: Self-supervised stochastic classifiers for few-shot class-incremental learning. In European Conference on Computer Vision, pages 432–448. Springer, 2022
2022
-
[11]
Maple: Multi-modal prompt learning
Muhammad Uzair Khattak, Hanoona Rasheed, Muham- mad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 19113–19122, 2023
2023
-
[12]
Self-regulating prompts: Founda- tional model adaptation without forgetting
Muhammad Uzair Khattak, Syed Talal Wasim, Muzam- mal Naseer, Salman Khan, Ming-Hsuan Yang, and Fa- had Shahbaz Khan. Self-regulating prompts: Founda- tional model adaptation without forgetting. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pag...
2023
-
[13]
Auto-encoding variational bayes, 2013
Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013
2013
-
[14]
Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In International conference on machine learning , pages 12888–12900. PMLR, 2022
2022
-
[16]
Stochastic classifiers for un- supervised domain adaptation
Zhihe Lu, Yongxin Yang, Xiatian Zhu, Cong Liu, Yi- Zhe Song, and Tao Xiang. Stochastic classifiers for un- supervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9111–9120, 2020
2020
-
[17]
Using deep learning for image-based plant dis- ease detection
Sharada P Mohanty, David P Hughes, and Marcel Salath´e. Using deep learning for image-based plant dis- ease detection. Frontiers in plant science , 7:215232, 2016
2016
-
[18]
Bayesian learning for neural networks, volume 118
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012
2012
-
[19]
Learning transferable visual models from natural lan- guage supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural lan- guage supervision. In International conference on ma- chine learni...
2021
-
[20]
A closer look at the few-shot adaptation of large vision-language models
Julio Silva-Rodriguez, Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. A closer look at the few-shot adaptation of large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23681–23690, 2024. 8
2024
-
[21]
Dynamic mixed- prototype model for incremental deepfake detection
Jiahe Tian, Cai Yu, Xi Wang, Peng Chen, Zihao Xiao, Jizhong Han, and Yesheng Chai. Dynamic mixed- prototype model for incremental deepfake detection. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 8129–8138, 2024
2024
-
[22]
Chestx- ray8: Hospital-scale chest x-ray database and bench- marks on weakly-supervised classification and localiza- tion of common thorax diseases
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mo- hammadhadi Bagheri, and Ronald M Summers. Chestx- ray8: Hospital-scale chest x-ray database and bench- marks on weakly-supervised classification and localiza- tion of common thorax diseases. In Proceedings of the IEEE conferenc...
2017
-
[23]
Semantic- guided robustness tuning for few-shot transfer across ex- treme domain shift
Kangyu Xiao, Zilei Wang, and Junjie Li. Semantic- guided robustness tuning for few-shot transfer across ex- treme domain shift. In European Conference on Com- puter Vision, pages 303–320. Springer, 2024
2024
-
[24]
Tcp: Textual-based class-aware prompt tuning for visual- language model
Hantao Yao, Rui Zhang, and Changsheng Xu. Tcp: Textual-based class-aware prompt tuning for visual- language model. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition , pages 23438–23448, 2024
2024
-
[25]
Task residual for tuning vision-language mod- els
Tao Yu, Zhihe Lu, Xin Jin, Zhibo Chen, and Xinchao Wang. Task residual for tuning vision-language mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10899– 10909, 2023
2023
-
[26]
Robust person re- identification by modelling feature uncertainty
Tianyuan Yu, Da Li, Yongxin Yang, Timothy M Hospedales, and Tao Xiang. Robust person re- identification by modelling feature uncertainty. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pages 552–561, 2019
2019
-
[27]
Tip-adapter: Training-free clip-adapter for better vision- language modeling
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision- language modeling. arXiv preprint arXiv:2111.03930 , 2021
2021 arXiv
-
[28]
Semi-supervised domain generalization with stochastic stylematch
Kaiyang Zhou, Chen Change Loy, and Ziwei Liu. Semi-supervised domain generalization with stochastic stylematch. International Journal of Computer Vision , 131(9):2377–2387, 2023
2023
-
[29]
Learning to prompt for vision-language models
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337–2348, 2022
2022
-
[30]
Prompt as free lunch: Enhancing di- versity in source-free cross-domain few-shot learning through semantic-guided prompting
Linhai Zhuo, Zheng Wang, Yuqian Fu, and Tian- wen Qian. Prompt as free lunch: Enhancing di- versity in source-free cross-domain few-shot learning through semantic-guided prompting. arXiv preprint arXiv:2412.00767, 2024. 9
2024 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.