REVIEW 4 major objections 5 minor 42 references
Bayesian Principles Improve Prompt Learning In Vision-Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Treating classifier logits as Bayesian random variables, with one-vs-each softmax and Pólya-Gamma augmentation, improves prompt-learning generalization in CLIP.
desk verdict The empirical OVE-PG recipe shows consistent gains on unseen classes, but the Bayesian derivation does not survive contact with the A≈I approximation, and the paper's own appendix admits the KL term is an L2 penalty. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a closed-form posterior sampler for logits built on the one-vs-each softmax bound and Pólya-Gamma augmentation. The pairwise-difference matrix $A \in \mathbb{R}^{C\times C\times C}$ maps logits $f$ to comparisons $\psi = Af$, with entries $f_{ni} - f_{nj}$; each sigmoid $\sigma(\psi_{nij})$ in the one-vs-each likelihood is augmented by a Pólya-Gamma variable $\omega$, which makes the conditional likelihood Gaussian in $\psi$. The paper approximates $A \approx I$ so the posterior covariance becomes diagonal, $(\alpha I + \Omega)^{-1}$, and samples fine-tuned logits from $\mathcal{N}(\mu_\theta, (\alpha I + \Omega)^{-1})$ while drawing $\omega$ from $\mathrm{PG}(1, \psi)$ with $\psi$ built from the pretrained mean. These samples feed a negative log-likelihood loss, and the KL term $\|\mu_\theta - \mu\|_2^2$ anchors the fine-tuned logits to the pretrained logits. The diagonal approximation is what makes the method computationally cheap, and it is also the step that carries the claim's Bayesian interpretation.
What would settle it
Re-run the CoOp experiments with the exact posterior covariance $(\alpha I + A^\top \Omega A)^{-1}$ in place of the diagonal approximation $(\alpha I + \Omega)^{-1}$: if unseen-class accuracy is unchanged, the diagonal approximation is not producing the gain; if training becomes unstable or accuracy drops, the method's stated posterior is not what it samples from.
Extended reading notes
Core claim
The paper's central claim is that a Bayesian treatment of the logit function itself—rather than of the prompt parameters—reduces overfitting in prompt learning for vision-language models. The paper puts a Gaussian prior on the logits whose mean is the pretrained model's logit output $\mu = I(x)^\top T(p^c)$, and models the fine-tuned logits as a Gaussian centered at $\mu_\theta = I(x)^\top T(p^c_\theta + r^c_\theta(x))$. It replaces the softmax likelihood with the one-vs-each approximation, a product of sigmoids over pairwise logit differences, and adds Pólya-Gamma auxiliary variables so the conditional likelihood is Gaussian and posterior sampling is closed form. A KL-divergence penalty $\beta\|\mu_\theta - \mu\|_2^2$ keeps the fine-tuned logits close to the pretrained ones. The combination, OVE-PG, is claimed to improve unseen-class accuracy and cross-dataset transfer across CoOp, CoCoOp, MaPLe, and APEX while adding no extra parameters or separate regularization terms.
Load-bearing premise
The load-bearing assumption is that the matrix $A$ that forms every pairwise class comparison can be treated as the identity, so the sampled logits are uncorrelated and the posterior covariance is diagonal; if this approximation is unreliable, the sampling procedure no longer draws from the stated Bayesian posterior.
Editorial extensions
If this is right
- Unseen-class accuracy improves for every prompt-learning method tested, with CoOp rising from 71.05 to 75.04 on average and EuroSAT jumping from 55.20 to 71.23.
- Cross-dataset transfer improves on average for both CoOp and CoCoOp, with the largest target gains on EuroSAT and DTD.
- The Pólya-Gamma augmentation, not the OVE approximation by itself, is what produces the generalization gain; plain OVE can even hurt on the hardest datasets.
- The KL-anchoring strength $\beta$ is robust across 0.2–0.5, so the method does not depend on a finely tuned hyperparameter.
- Because the change is confined to the likelihood and sampling step, it can be added to existing prompt-learning methods without extra parameters or architectural changes.
Reading between the lines
- This recipe could generalize beyond prompt tokens: any fine-tuning scheme that puts a pretrained linear classifier or logit head on a frozen encoder could anchor its logits to the pretrained output and use OVE-PG sampling, separating the benefit of the Bayesian logit prior from prompt parameterization.
- The large gain on EuroSAT, a domain far from the pretraining distribution, suggests the regularizer may be most valuable under distribution shift; a controlled sweep over shift magnitude would test whether the advantage scales with domain distance.
- The posterior logits are softened by Pólya-Gamma noise, so OVE-PG may also improve calibration or out-of-distribution detection on unseen classes; the paper does not report those metrics.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes OVE-PG, a prompt-learning method for vision-language models that replaces the standard softmax likelihood with a one-vs-each (OVE) softmax approximation and uses Pólya-Gamma augmentation to make the resulting sigmoid factors conjugate with Gaussian priors over logits. The prior mean is set to the logits of the frozen pretrained CLIP model, and the posterior mean to the logits of the trainable prompt model; a KL-type term between prior and posterior is added as a regularizer. The method is evaluated on seen-to-unseen generalization across ten datasets and on cross-dataset transfer, integrated into CoOp, CoCoOp, MaPLe, and APEX. The reported results show consistent improvements over the corresponding softmax baselines, with the largest gains on EuroSAT and FGVC Aircraft for CoOp.
Significance. If the empirical results hold, the contribution is practically useful: it is a simple plug-in modification that improves unseen-class and cross-dataset accuracy across several prompt-learning frameworks without adding network parameters, and the code is publicly available. The reported improvements are large on some datasets (e.g., EuroSAT from 55.20 to 71.23 for CoOp) and consistent across baselines and seeds, with standard deviations reported in the appendix. However, the paper's central theoretical claim, that the objective is derived from Bayesian principles, is not currently established: the derivation relies on an acknowledged diagonal-covariance approximation and on collapsing a KL divergence to an L2 term, and several admitted deviations place the method closer to a heuristic regularized logit-matching procedure than to a posterior-based method. The significance of the paper therefore depends on whether the authors can either repair the derivation or honestly reframe the contribution as an empirical regularizer with PG noise.
major comments (4)
- [Section 3.4, Eq. (17) and Appendix C.4.3]
- [Section 3.4 and Appendix C.3, Eq. (C.44)]
- [Algorithm 1, line 'fθ^(m) ← ∑_C ψθ^(m)']
- [Appendix C.4.2]
minor comments (5)
- [Section 1]
- [Section 5.1]
- [References]
- [Appendix F, Table F.6]
- [Appendix D]
Circularity Check
No significant circularity: OVE-PG's derivation and held-out evaluation are self-contained; the main caveats are approximations, not input-output equivalences.
full rationale
The central derivation is a variational objective: a Polya-Gamma augmented one-vs-each likelihood, a Gaussian prior centered at the frozen CLIP logits, and a Gaussian variational posterior whose mean is the tunable prompt logits. The KL term is reduced to ||μθ−μ||^2_2 by dropping terms that do not depend on the prompt parameters (Section 3.4, Eq. 17), and Appendix C.4.3 explicitly shows the rigorous KL expression before the reduction. This is an approximation, not a relabeling of a fitted quantity as a prediction. The benchmark claims are evaluated on held-out unseen classes and cross-dataset transfer (Tables 1 and 2), so the empirical result is not forced by the loss definition. The A≈I diagonalization in Section 3.4 and Appendix C.3 is a genuine modeling assumption, acknowledged in C.4.1 as 'motivated by purely computational reasons'; it affects whether the sampler matches the stated posterior, but it does not make the derivation circular. Appendix C.4.2 candidly notes that the construction 'deviates from the usual Bayesian construction,' which is a stated limitation of the Bayesian framing rather than evidence that the prediction reduces to its inputs. The only self-citation is Yang et al. 2024 (APEX, co-authored by Jongwoo Ko), used as a baseline and integration target; it is not load-bearing for the core derivation. No circular step meets the quoted-reduction bar.
Assumptions & free parameters
free parameters (3)
- β (KL weight) =
not stated for main tables (sensitivity over 0.1-0.7, robust 0.2-0.5)
- α (prior precision) =
not stated for CLIP experiments (synthetic uses 1 and 100)
- M (number of Gibbs chains) =
not stated
assumptions (5)
- standard math Polya-Gamma augmentation identity (Eq. 1) expresses logistic likelihood terms as Gaussian mixtures over PG variables.
- standard math One-vs-each product lower-bounds the softmax (Eq. 9 and Appendix A).
- domain assumption Frozen CLIP logits are a sensible prior mean for downstream tasks.
- ad hoc to paper A ≈ I and component-wise independence of logits.
- ad hoc to paper PG variable ω can be sampled once from the pretrained logits and held fixed during prompt optimization.
Cite this review
Pith. "Pith review of Bayesian Principles Improve Prompt Learning In Vision-Language Models." pith.science (2026). https://pith.science/paper/GU6O5DU6
@misc{pith2026250414123,
author = {Pith},
title = {Pith review of: Bayesian Principles Improve Prompt Learning In Vision-Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/GU6O5DU6}},
note = {Machine review of arXiv:2504.14123}
}
read the original abstract
Prompt learning is a popular fine-tuning method for vision-language models due to its efficiency. It requires a small number of additional learnable parameters while significantly enhancing performance on target tasks. However, most existing methods suffer from overfitting to fine-tuning data, yielding poor generalizability. To address this, we propose a new training objective function based on a Bayesian learning principle to balance adaptability and generalizability. We derive a prior over the logits, where the mean function is parameterized by the pre-trained model, while the posterior corresponds to the fine-tuned model. This objective establishes a balance by allowing the fine-tuned model to adapt to downstream tasks while remaining close to the pre-trained model.
Figures
Reference graph
Works this paper leans on
-
[1]
Bossard, L., Guillaumin, M., and Gool, L. V. (2014a). Food-101 - mining discriminative components with random forests. In European Conference on Computer Vision
work page 2014
-
[2]
Bossard, L., Guillaumin, M., and Gool, L. V. (2014b). Food-101--mining discriminative components with random forests. In European Conference on Computer Vision
work page 2014
-
[3]
Chen, G., Yao, W., Song, X., Li, X., Rao, Y., and Zhang, K. (2023). PLOT : Prompt learning with optimal transport for vision-language models. In The Eleventh International Conference on Learning Representations
work page 2023
-
[4]
Cho, Y., Bae, H., Shin, S., Youn, Y. D., Joo, W., and Moon, I.-C. (2024). Make prompts adaptable: Bayesian modeling for vision-language prompt learning with data-dependent prior. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 11552--11560
work page 2024
-
[5]
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. (2014). Describing textures in the wild. In IEEE Conference on Computer Vision and Pattern Recognition
work page 2014
-
[6]
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). ImageNet : A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition
work page 2009
-
[7]
M., Sanchez, E., Bulat, A., da Costa, V
Derakhshani, M. M., Sanchez, E., Bulat, A., da Costa, V. G. T., Snoek, C. G., Tzimiropoulos, G., and Martinez, B. (2023a). Bayesian prompt learning for image-language model generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15237--15246
work page 2023
-
[8]
M., Sanchez, E., Bulat, A., da Costa, V
Derakhshani, M. M., Sanchez, E., Bulat, A., da Costa, V. G. T., Snoek, C. G., Tzimiropoulos, G., and Martinez, B. (2023b). Bayesian prompt learning for image-language model generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 15237--15246
work page 2023
Show all 42 references
-
[9]
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In International Confe...
2021
-
[10]
Fei-Fei, L., Fergus, R., and Perona, P. (2004a). Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. 2004 Conference on Computer Vision and Pattern Recognition Workshop , pages 178--178
2004
-
[11]
Fei-Fei, L., Fergus, R., and Perona, P. (2004b). Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In IEEE Conference on Computer Vision and Pattern Recognition - Workshops
2004
-
[12]
Helber, P., Bischke, B., Dengel, A., and Borth, D. (2019). Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
2019
-
[13]
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T. (2021). Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning , pages 4904--4916. PMLR
2021
-
[14]
U., Rasheed, H., Maaz, M., Khan, S., and Khan, F
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S. (2023a). Maple: Multi-modal prompt learning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition
2023
-
[15]
U., Wasim, S
Khattak, M. U., Wasim, S. T., Naseer, M., Khan, S., Yang, M.-H., and Khan, F. S. (2023b). Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15190--15200
2023
-
[16]
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. (2013). 3D object representations for fine-grained categorization. In IEEE Conference on Computer Vision and Pattern Recognition - Workshops
2013
-
[17]
Lester, B., Al-Rfou, R., and Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. In Conference on Empirical Methods in Natural Language Processing
2021
-
[18]
J., and Adams, R
Linderman, S., Johnson, M. J., and Adams, R. P. (2015). Dependent multinomial models made easy: Stick-breaking with the polya-gamma augmentation. In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R., editors, Advances in Neural Information Processing Systems , v...
2015
-
[19]
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., and Tang, J. (2023). Gpt understands, too. AI Open
2023
-
[20]
Lu, Y., Liu, J., Zhang, Y., Liu, Y., and Tian, X. (2022). Prompt distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5206--5215
2022
-
[21]
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. (2013). Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151
2013 arXiv
-
[22]
Polson, J
Nicholas G. Polson, J. G. S. and Windle, J. (2013). Bayesian inference for logistic models using pólya–gamma latent variables. Journal of the American Statistical Association , 108(504):1339--1349
2013
-
[23]
and Zisserman, A
Nilsback, M.-E. and Zisserman, A. (2008). Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing
2008
-
[24]
Oh, C., Kim, M., Lim, H., Park, J., Jeong, E., Cheng, Z.-Q., and Song, K. (2023). Towards calibrated robust fine-tuning of vision-language models. arXiv preprint arXiv:2311.01723
2023 arXiv
-
[25]
M., Vedaldi, A., Zisserman, A., and Jawahar, C
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. (2012). Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition
2012
-
[26]
W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748--8763. PMLR
2021
-
[27]
and Zemel, R
Snell, J. and Zemel, R. (2021). Bayesian few-shot classification with one-vs-each p \'o lya-gamma augmented gaussian processes. In International Conference on Learning Representations
2021
-
[28]
R., and Shah, M
Soomro, K., Zamir, A. R., and Shah, M. (2012a). A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision
2012
-
[29]
R., and Shah, M
Soomro, K., Zamir, A. R., and Shah, M. (2012b). Ucf101: A dataset of 101 human actions classes from videos in the wild. ArXiv , abs/1212.0402
2012 arXiv
-
[30]
Sun, Q., Fang, Y., Wu, L., Wang, X., and Cao, Y. (2023). Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389
2023 arXiv
-
[31]
Titsias RC AUEB, M. (2016). One-vs-each approximation to softmax for scalable estimation of probabilities. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc
2016
-
[32]
Veličković, P., Perivolaropoulos, C., Barbero, F., and Pascanu, R. (2024). softmax is not enough (for sharp out-of-distribution)
2024
-
[33]
H., and Yang, L
Wu, C.-E., Tian, Y., Yu, H., Wang, H., Morgado, P., Hu, Y. H., and Yang, L. (2023). Why is prompt tuning for vision-language models robust to noisy labels? In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15488--15497
2023
-
[34]
A., Oliva, A., and Torralba, A
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. (2010a). Sun database: Large-scale scene recognition from abbey to zoo. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 3485--3492
2010
-
[35]
A., Oliva, A., and Torralba, A
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. (2010b). Sun database: Large-scale scene recognition from abbey to zoo. In IEEE Conference on Computer Vision and Pattern Recognition
2010
-
[36]
Yang, Y., Ko, J., and Yun, S.-Y. (2024). Towards difficulty-agnostic efficient transfer learning for vision-language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N., editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages ...
2024
-
[37]
Yao, H., Zhang, R., and Xu, C. (2023). Visual-language prompt tuning with knowledge-guided context optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6757--6767
2023
-
[38]
Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y. (2022). Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917
2022 arXiv
-
[39]
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. (2023). Sigmoid loss for language image pre-training. arXiv preprint arXiv:2303.15343
2023 arXiv
-
[40]
C., and Liu, Z
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. (2022a). Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16816--16825
2022
-
[41]
C., and Liu, Z
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. (2022b). Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337--2348
2022
-
[42]
Zhu, B., Niu, Y., Han, Y., Wu, Y., and Zhang, H. (2022). Prompt-aligned gradient for prompt tuning. arXiv preprint arXiv:2205.14865
2022 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.