{"id":"4f5a36fd-01c0-4ddc-93be-f349cdf326ac","arxiv_id":"2504.14123","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Replacing softmax with a one-vs-each Polya-Gamma augmented objective plus a logit-matching penalty improves unseen-class and cross-dataset accuracy across CoOp, CoCoOp, MaPLe, and APEX.","lead":"The authors test a Bayesian tweak to prompt learning in CLIP-style vision-language models: replace the softmax with a one-vs-each sigmoid objective and add a penalty that keeps the tuned model's logits close to the pretrained model's. On seen-to-unseen and cross-dataset benchmarks, this consistently outperforms ordinary prompt learning while using only a handful of extra training steps.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The A≈I step in §3.4/C.3 decouples ψ=Af, but the exact covariance of ψ is dense (e.g., Cov(ψ_12,ψ_13)=1/α), so the claimed posterior is not the posterior and the Bayesian derivation is unsupported.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the diagonal A≈I approximation in §3.4/C.3. I agree this is load-bearing because it is not a small numerical error; it removes correlations that are structurally present in pairwise differences, and the paper's own Appendix C.3 contradicts the 'uncorrelated' claim by doubling the variance. Appendix C.4.2 further concedes that the construction is not the usual Bayesian formulation. Therefore the theoretical justification for OVE-PG is currently unsupported. The empirical results could still hold as an engineered stochastic regularizer with a knowledge-distillation penalty, so the correct verdict remains conditional rather than reject: the authors should either supply a correct derivation or explicitly reframe the method as a heuristic, and release code that disambiguates Algorithm 1's 'f = sum_C ψ' step from the product formula in Appendix C.2. Since the reader's verdict is already CONDITIONAL and this pass reinforces that condition rather than moving it, no verdict change is needed.","tokens_in":27743,"tokens_out":6430,"duration_ms":61743,"concrete_test":"For C=3 and any fixed Ω=diag(ω), compare the exact posterior covariance of ψ, A(αI + A^TΩA)^−1 A^T, with the diagonal (α/2 I + Ω)^−1 used in Eq. C.44; compute the Frobenius norm of the off-diagonal part. Then run the Figure 1 synthetic 1D task (and, if code is available, the CoOp EuroSAT setup) with the exact posterior sampler in place of the diagonal sampler. If the training trajectories or final classification results differ materially, the approximation is load-bearing and the method should be reframed as a heuristic rather than as Bayesian inference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 and Appendix C.3 replace the posterior over ψ=Af with a diagonal distribution, justified by 'A is a sparse matrix ... we approximate A≈I.' This is the step that makes Gibbs sampling feasible and collapses the KL term to ||μθ−μ||^2. But the exact covariance of ψ is Cov(ψ_{nij}, ψ_{nkl}) = (1/α)(δ_{ik} − δ_{il} − δ_{jk} + δ_{jl}); for example Cov(ψ_12, ψ_13) = 1/α while Var(ψ_12) = 2/α. These off-diagonals are not small: pairwise differences sharing a class are strongly correlated. Appendix C.3 itself states ψθ ∼ N(Aμθ, 2/α I), conceding the variance is doubled, but the covariance matrix used in Eq. C.44, (α/2 I + Ω)^−1, is valid only if A A^T = 2I, which is false. Consequently, the PG samples in Algorithm 1 are drawn from a different distribution than the stated posterior, and the 'KL' term is an ad hoc logit-matching penalty rather than a Bayesian regularizer. Appendix C.4.2 openly acknowledges that the construction deviates from the usual Bayesian formulation. The empirical gains may survive, but the central claim that Bayesian principles drive them is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OVE-PG, a prompt-learning method for vision-language models that replaces the standard softmax likelihood with a one-vs-each (OVE) softmax approximation and uses Pólya-Gamma augmentation to make the resulting sigmoid factors conjugate with Gaussian priors over logits. The prior mean is set to the logits of the frozen pretrained CLIP model, and the posterior mean to the logits of the trainable prompt model; a KL-type term between prior and posterior is added as a regularizer. The method is evaluated on seen-to-unseen generalization across ten datasets and on cross-dataset transfer, integrated into CoOp, CoCoOp, MaPLe, and APEX. The reported results show consistent improvements over the corresponding softmax baselines, with the largest gains on EuroSAT and FGVC Aircraft for CoOp.","tokens_in":27963,"tokens_out":5148,"duration_ms":49171,"significance":"If the empirical results hold, the contribution is practically useful: it is a simple plug-in modification that improves unseen-class and cross-dataset accuracy across several prompt-learning frameworks without adding network parameters, and the code is publicly available. The reported improvements are large on some datasets (e.g., EuroSAT from 55.20 to 71.23 for CoOp) and consistent across baselines and seeds, with standard deviations reported in the appendix. However, the paper's central theoretical claim, that the objective is derived from Bayesian principles, is not currently established: the derivation relies on an acknowledged diagonal-covariance approximation and on collapsing a KL divergence to an L2 term, and several admitted deviations place the method closer to a heuristic regularized logit-matching procedure than to a posterior-based method. The significance of the paper therefore depends on whether the authors can either repair the derivation or honestly reframe the contribution as an empirical regularizer with PG noise.","major_comments":[{"comment":"","section":"Section 3.4, Eq. (17) and Appendix C.4.3"},{"comment":"","section":"Section 3.4 and Appendix C.3, Eq. (C.44)"},{"comment":"","section":"Algorithm 1, line 'fθ^(m) ← ∑_C ψθ^(m)'"},{"comment":"","section":"Appendix C.4.2"}],"minor_comments":[{"comment":"","section":"Section 1"},{"comment":"","section":"Section 5.1"},{"comment":"","section":"References"},{"comment":"","section":"Appendix F, Table F.6"},{"comment":"","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The manuscript header states that it was published in the AISTATS 2025 proceedings (PMLR Volume 258). If this submission is intended for a different venue, the editor should verify that prior publication does not conflict with the journal's policies. The main reviewable issue is the gap between the advertised Bayesian derivation and the implemented objective: the derivation is supported only by approximations that the authors themselves acknowledge, and the value of the paper rests on whether the empirical claims remain meaningful when the method is described as a heuristic regularizer."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, the short version: the recipe in this paper looks empirically useful, but the Bayesian story is mostly decoration. The PG noise plus an L2 pull toward frozen CLIP logits does improve unseen-class accuracy across several baselines, and that is worth knowing. But the derivation in Section 3 and the appendix does not support the paper's claims, and the implementation description has inconsistencies a referee will need cleared up.\n\nWhat is new: OVE-PG applies Snell and Zemel's one-vs-each PG softmax to prompt learning, with the frozen CLIP logits as the prior mean. That combination is not in the cited work, and the empirical evidence is broad: ten datasets, four prompt-learning baselines, consistent average gains on unseen classes, largest on weak baselines like EuroSAT and FGVC Aircraft. Appendix tables include standard deviations for six datasets, which is good-faith reporting. The synthetic 1D and ViT-on-EMNIST studies give useful intuition, and the code link appears in the abstract.\n\nThe soft spots are real. The load-bearing step is the A≈I approximation in Section 3.4 and Appendix C.3. A is the pairwise-difference operator, not identity: the exact covariance of ψ=Af has off-diagonal entries of the same magnitude as the diagonal, so sampling from the stated diagonal posterior is not sampling from the posterior of ψ. The paper's own Appendix C.4.2 says the construction \"deviates from the usual Bayesian construction,\" and C.4.3 admits the KL term reduces to an L2 penalty. That is honest, but it undercuts the title and the claim that Bayesian principles drive the gains. The method is effectively a stochastic regularizer: PG noise on pairwise logits plus β||μθ−μ||^2.\n\nAlso, Algorithm 1's line fθ←∑C ψθ does not match the product formula in Appendices C.2/C.3; one of them is wrong. Main Table 1 lacks error bars, even though the appendix shows the authors computed seeds for six datasets. The novelty beyond Snell-Zemel plus KgCoOp-style logit matching is modest.\n\nNet: the empirical recipe is plausible and deserves independent verification. If the implementation is correct, this could be a useful drop-in addition to CoOp-style training. But the Bayesian framing should be either fixed or dropped. I would send it to a serious referee, with expectation of heavy revision: correct the algorithm, verify with released code, report error bars, and reframe the contribution as an empirical regularizer.","headline":"The empirical OVE-PG recipe shows consistent gains on unseen classes, but the Bayesian derivation does not survive contact with the A≈I approximation, and the paper's own appendix admits the KL term is an L2 penalty.","tokens_in":28617,"tokens_out":3642,"would_cite":false,"duration_ms":34363,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Treating classifier logits as Bayesian random variables, with one-vs-each softmax and Pólya-Gamma augmentation, improves prompt-learning generalization in CLIP.","keywords":["prompt learning","vision-language models","CLIP","Pólya-Gamma augmentation","one-vs-each softmax","Bayesian inference","generalization","overfitting"],"falsifier":"Re-run the CoOp experiments with the exact posterior covariance $(\\alpha I + A^\\top \\Omega A)^{-1}$ in place of the diagonal approximation $(\\alpha I + \\Omega)^{-1}$: if unseen-class accuracy is unchanged, the diagonal approximation is not producing the gain; if training becomes unstable or accuracy drops, the method's stated posterior is not what it samples from.","tokens_in":27423,"feed_emoji":"📈","tokens_out":9333,"duration_ms":76461,"temperature":0.7,"pith_summary":"The paper claims that prompt learning in vision-language models can be made to generalize better by treating the classifier logits as Bayesian random variables rather than fixed outputs. Specifically, it replaces the standard softmax likelihood with a one-vs-each approximation—each class probability is built from sigmoids over pairwise logit differences—and augments those sigmoids with Pólya-Gamma latent variables. The fine-tuned model's logits are pulled toward the pretrained model's logits through a KL-divergence penalty, which acts as a knowledge-distillation regularizer. On unseen classes, this raises CoOp's average accuracy from 71.05 to 75.04, with particularly large gains on EuroSAT and FGVC Aircraft, while keeping the number of trainable parameters unchanged. The paper concludes that the combination is a simpler and more effective anti-overfitting device than extra regularization terms or extra networks.","feed_headline":"Bayesian softmax replacement lifts CLIP unseen-class accuracy","feed_subtitle":"On CoOp, unseen-class accuracy rises from 71.05 to 75.04 percent with no extra parameters.","key_machinery":"The central mechanism is a closed-form posterior sampler for logits built on the one-vs-each softmax bound and Pólya-Gamma augmentation. The pairwise-difference matrix $A \\in \\mathbb{R}^{C\\times C\\times C}$ maps logits $f$ to comparisons $\\psi = Af$, with entries $f_{ni} - f_{nj}$; each sigmoid $\\sigma(\\psi_{nij})$ in the one-vs-each likelihood is augmented by a Pólya-Gamma variable $\\omega$, which makes the conditional likelihood Gaussian in $\\psi$. The paper approximates $A \\approx I$ so the posterior covariance becomes diagonal, $(\\alpha I + \\Omega)^{-1}$, and samples fine-tuned logits from $\\mathcal{N}(\\mu_\\theta, (\\alpha I + \\Omega)^{-1})$ while drawing $\\omega$ from $\\mathrm{PG}(1, \\psi)$ with $\\psi$ built from the pretrained mean. These samples feed a negative log-likelihood loss, and the KL term $\\|\\mu_\\theta - \\mu\\|_2^2$ anchors the fine-tuned logits to the pretrained logits. The diagonal approximation is what makes the method computationally cheap, and it is also the step that carries the claim's Bayesian interpretation.","core_discovery":"The paper's central claim is that a Bayesian treatment of the logit function itself—rather than of the prompt parameters—reduces overfitting in prompt learning for vision-language models. The paper puts a Gaussian prior on the logits whose mean is the pretrained model's logit output $\\mu = I(x)^\\top T(p^c)$, and models the fine-tuned logits as a Gaussian centered at $\\mu_\\theta = I(x)^\\top T(p^c_\\theta + r^c_\\theta(x))$. It replaces the softmax likelihood with the one-vs-each approximation, a product of sigmoids over pairwise logit differences, and adds Pólya-Gamma auxiliary variables so the conditional likelihood is Gaussian and posterior sampling is closed form. A KL-divergence penalty $\\beta\\|\\mu_\\theta - \\mu\\|_2^2$ keeps the fine-tuned logits close to the pretrained ones. The combination, OVE-PG, is claimed to improve unseen-class accuracy and cross-dataset transfer across CoOp, CoCoOp, MaPLe, and APEX while adding no extra parameters or separate regularization terms.","pith_inferences":["This recipe could generalize beyond prompt tokens: any fine-tuning scheme that puts a pretrained linear classifier or logit head on a frozen encoder could anchor its logits to the pretrained output and use OVE-PG sampling, separating the benefit of the Bayesian logit prior from prompt parameterization.","The large gain on EuroSAT, a domain far from the pretraining distribution, suggests the regularizer may be most valuable under distribution shift; a controlled sweep over shift magnitude would test whether the advantage scales with domain distance.","The posterior logits are softened by Pólya-Gamma noise, so OVE-PG may also improve calibration or out-of-distribution detection on unseen classes; the paper does not report those metrics."],"forward_implications":["Unseen-class accuracy improves for every prompt-learning method tested, with CoOp rising from 71.05 to 75.04 on average and EuroSAT jumping from 55.20 to 71.23.","Cross-dataset transfer improves on average for both CoOp and CoCoOp, with the largest target gains on EuroSAT and DTD.","The Pólya-Gamma augmentation, not the OVE approximation by itself, is what produces the generalization gain; plain OVE can even hurt on the hardest datasets.","The KL-anchoring strength $\\beta$ is robust across 0.2–0.5, so the method does not depend on a finely tuned hyperparameter.","Because the change is confined to the likelihood and sampling step, it can be added to existing prompt-learning methods without extra parameters or architectural changes."],"supporting_citations":[{"why":"Supplies the pretrained CLIP model whose frozen logits define the prior mean $\\mu$ and which remains fixed during prompt tuning.","marker":"Radford et al., 2021"},{"why":"Introduces CoOp, the prompt-learning baseline OVE-PG is plugged into and whose seen/unseen class split defines the benchmark.","marker":"Zhou et al., 2022b"},{"why":"Introduces CoCoOp, the conditional-prompt baseline used for cross-dataset evaluation.","marker":"Zhou et al., 2022a"},{"why":"Provides the one-vs-each composite-likelihood approximation to softmax that replaces the standard likelihood.","marker":"Titsias RC AUEB, 2016"},{"why":"Gives the Pólya-Gamma augmentation identity that turns each sigmoid likelihood into a Gaussian conditional on $\\omega$.","marker":"Nicholas G. Polson and Windle, 2013"},{"why":"Supplies the one-vs-each Pólya-Gamma augmented Gaussian-process formulation that the paper adapts to CLIP logits.","marker":"Snell and Zemel, 2021"},{"why":"Introduces MaPLe, one of the four prompt-learning baselines compared in the experiments.","marker":"Khattak et al., 2023a"},{"why":"Introduces APEX, the difficulty-agnostic transfer baseline compared in the experiments.","marker":"Yang et al., 2024"},{"why":"Provides the Bayesian prompt-learning baseline whose code and hyperparameter setup the experiments build on.","marker":"Derakhshani et al., 2023a"}],"fun_headline_variants":["Bayesian logit prior lifts CLIP unseen accuracy, no extra params","Bayesian logit prior improves prompt learning generalization","OVE-PG: Bayesian logit prior reduces prompt overfitting without extra parameters","Bayesian logit prior balances CLIP adaptation and generalization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the matrix $A$ that forms every pairwise class comparison can be treated as the identity, so the sampled logits are uncorrelated and the posterior covariance is diagonal; if this approximation is unreliable, the sampling procedure no longer draws from the stated Bayesian posterior.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian logit prior lifts CLIP unseen accuracy, no extra params","Bayesian logit prior improves prompt learning generalization","OVE-PG: Bayesian logit prior reduces prompt overfitting without extra parameters","Bayesian logit prior balances CLIP adaptation and generalization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001164,"raw_usage":{"total_tokens":4789,"prompt_tokens":886,"completion_tokens":3903,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":3831}},"tokens_in":502,"tokens_out":3903,"duration_ms":23653,"temperature":1.0,"reasoning_tokens":3831,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:56:46.570620+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the CoOp experiments with the exact posterior covariance $(\\alpha I + A^\\top \\Omega A)^{-1}$ in place of the diagonal approximation $(\\alpha I + \\Omega)^{-1}$: if unseen-class accuracy is unchanged, the diagonal approximation is not producing the gain; if training becomes unstable or accuracy drops, the method's stated posterior is not what it samples from.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the one-vs-each composite-likelihood approximation to softmax that replaces the standard likelihood."},{"cited_title":"Polson, J","cited_arxiv_id":null,"evidence_quote":"Gives the Pólya-Gamma augmentation identity that turns each sigmoid likelihood into a Gaussian conditional on $\\omega$."},{"cited_title":"and Zemel, R","cited_arxiv_id":null,"evidence_quote":"Supplies the one-vs-each Pólya-Gamma augmented Gaussian-process formulation that the paper adapts to CLIP logits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces APEX, the difficulty-agnostic transfer baseline compared in the experiments."}],"review_version":1}