Pith. sign in

REVIEW 2 major objections 5 minor 51 references

A plug-in optimizer balances flatness and low loss so that learnable prompts keep both seen-class accuracy and unseen-class generalization.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 02:58 UTC pith:7X4ZMEQE

load-bearing objection Solid plug-in SAM optimizer for VLM prompts with real HM gains across five methods; the dual-constraint story is a clean increment, but the EMA that underpins the geometry is never checked. the 2 major comments →

arxiv 2607.05727 v1 pith:7X4ZMEQE submitted 2026-07-07 cs.CV

SAMPLe: SAM-based Optimizer for Prompt Learning in VLMs

Hossein Rajoli , Fatemeh Lotfi , Niloufar Alipour Talemi , Hossein Kashiani , Xiaolong Ma , Fatemeh Afghah This is my paper
classification cs.CV
keywords prompt learningsharpness-aware minimizationvision-language modelsgeneralizationCLIPbase-to-newoptimizer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Prompt learning lets a frozen vision-language model adapt to a new task by updating only a few token vectors. Those few parameters easily overfit the training distribution and land in sharp minima that fail on new classes. This paper claims that a carefully designed sharpness-aware optimizer, SAMPLe, can force every update to satisfy two simultaneous constraints: stay aligned with ordinary empirical-risk minimization (so training loss stays low) and stay orthogonal to the full-batch gradient direction (so the optimizer keeps exploring flatter regions). The dual objective is realized by a single first-order update that subtracts a scaled batch-specific component from the usual SAM perturbation. When the same optimizer is dropped into five existing prompt learners, base-to-new harmonic mean, cross-dataset transfer, and domain-shift accuracy all rise without any change to the model architecture.

Core claim

SAMPLe improves generalization of prompt learning by solving, at every step, a dual-objective loss that jointly minimizes empirical risk and enforces orthogonality of the perturbed gradient to the full-batch direction; the resulting flatter, still-low-loss minima transfer better to unseen classes and domains than ordinary SAM, F-SAM or SAGM.

What carries the argument

The dual-objective update of Eq. 10–14: SAM-style ascent to a nearby point, followed by a correction that subtracts the projection onto the full-batch gradient so the final step is forced toward the batch-specific (orthogonal) direction while remaining aligned with the current mini-batch gradient.

Load-bearing premise

The method assumes that a cheap exponential moving average of mini-batch gradients is a faithful enough stand-in for the true full-batch gradient that the orthogonality constraint actually steers the optimizer into flatter regions rather than merely adding noise.

What would settle it

Replace the EMA full-gradient estimate with the true full-batch gradient (or measure its cosine error on the prompt parameters) and check whether the reported gains in harmonic mean and domain-shift accuracy disappear or reverse.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes SAMPLe, a plug-in sharpness-aware optimizer for prompt learning in VLMs. It addresses the performance–generalization trade-off of learnable prompts by a dual-objective loss (Eq. 10) that simultaneously minimizes empirical risk and a perturbed loss while enforcing (i) alignment of the perturbed gradient with the mini-batch gradient (exploitation) and (ii) orthogonality of that gradient to an EMA approximation of the full-batch gradient (exploration). The method is integrated into CoOp, CoCoOp, MaPLe, CoPrompt and TCP and evaluated on base-to-new, cross-dataset and cross-domain protocols over 11 datasets. A convergence rate of O(log T / √T) is proved under standard bounded-gradient and K-Lipschitz assumptions (Thm. 1 / Appendix 8.1). Empirically, SAMPLe raises harmonic-mean accuracy relative to SAM, F-SAM and SAGM for every backbone (Table 1) and improves transfer averages (Tables 2–3).

Significance. If the dual-constraint geometry is the operative mechanism, SAMPLe supplies a clean, model-agnostic optimizer that consistently improves five distinct prompt learners without architectural changes. The breadth of the experimental suite (five methods × eleven datasets × three protocols), the term-wise ablation (Appendix Table 6), the sensitivity plots for ρ and λ (Fig. 3), and the standard-rate convergence proof are genuine strengths that make the work useful to the prompt-learning community even if the precise geometric story requires further verification.

major comments (2)
  1. [Sec. 3.1, 4.2 (Eqs. 9, 13–14)] Sec. 3.1 and 4.2 (Eqs. 9, 13–14): the entire dual-objective argument treats the EMA mt as a faithful proxy for the true full-batch gradient ∇F L. No cosine-similarity, norm-error or other fidelity metric between mt and the exact full-batch gradient on the prompt parameters is reported, nor is an exact-full-batch ablation provided. Because the prompt space is tiny, modest EMA lag can turn the claimed orthogonal-exploration term into unstructured noise; the observed HM gains over F-SAM/SAGM could therefore arise from a different effect of the α-term. A short diagnostic (or exact-full-batch control) is needed to substantiate the mechanistic claim.
  2. [Table 1, Fig. 3] Table 1 and Fig. 3: while average HM improvements are consistent, several individual base/new cells show non-monotonic or mixed behaviour (e.g., CoOp+SAMPLe base accuracy drops on StanfordCars relative to CoOp+FSAM; MaPLe+SAMPLe is essentially tied with MaPLe+SAGM on Food101). The paper asserts that SAMPLe “does not sacrifice one [base/new] in favour of the other,” yet the per-dataset tables contain counter-examples. A brief discussion of when the dual constraints fail to preserve base accuracy would strengthen the central claim.
minor comments (5)
  1. [Fig. 1] Fig. 1 caption and surrounding text claim that SAMPLe reaches both flatter minima and lower empirical risk; the visualisation is qualitative only. A quantitative sharpness measure (e.g., largest Hessian eigenvalue or average loss in a ρ-ball) would make the landscape claim more rigorous.
  2. [Algorithm 1] Algorithm 1 line 10 writes θ t ← heta t − ηt abla L( heta t;D) after computing the dual objective; it is unclear whether the gradient of the full dual loss or only the ERM term is used for the parameter update. Clarifying the exact gradient that is back-propagated would aid reproducibility.
  3. [Sec. 3.1–4.2] Notation for the full-batch gradient alternates between abla F L, mt and abla LF; a single consistent symbol would improve readability.
  4. [Table 3] Table 3 contains a typographical error (“71,03” instead of “71.03”) for CoCoOp+SAMPLe on ImageNet.
  5. [Appendix 8.6] The staged-training protocol used for MaPLe and CoPrompt (Appendix 8.6) is described only briefly; stating whether the same schedule is applied to the SAM/F-SAM/SAGM baselines would remove a possible confound.

Circularity Check

0 steps flagged

No circularity: dual-objective is an explicit design choice verified by algebra, not a prediction forced by its own inputs or self-citation.

full rationale

The paper proposes an optimizer (Eq. 10) whose form is deliberately chosen to encode two gradient constraints (ERM alignment of the perturbed gradient plus orthogonality to the full-batch direction). The subsequent Taylor expansion (Eqs. 11–14) and the cancellation that isolates the batch-specific component are pure algebraic consequences of that definition; they do not claim to derive an independent physical or statistical quantity. Convergence (Theorem 1) follows standard first-order arguments under Lipschitz and bounded-gradient assumptions that do not embed the reported accuracy numbers. Empirical gains (Tables 1–3) are measured on held-out base/new splits and external ImageNet variants never used to set ρ, α or λ. Self-citations are limited to ordinary SAM literature and do not supply a uniqueness theorem or load-bearing premise. The EMA approximation mt is an unvalidated modeling assumption, but that is a correctness risk, not circularity. Nothing reduces by construction to a fitted performance metric.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central claim rests on three free hyperparameters (ρ, α, λ), standard first-order optimization assumptions (bounded gradients, K-Lipschitz), and the modeling choice that an EMA of mini-batch gradients adequately approximates the full-batch direction. No new physical entities are postulated; the only invented object is the SAMPLe objective itself, whose independent evidence is the reported accuracy tables.

free parameters (3)
  • perturbation radius ρ = 0.05–0.10
    Chosen per prompt learner (0.05 or 0.10) and held fixed across all 11 datasets; directly controls the size of the SAM neighborhood and therefore the claimed flatness.
  • alignment coefficient α = 0.0015
    Set to 0.0015 for SAMPLe (slightly larger than SAGM’s 0.0010); scales the batch-specific correction term that realizes the dual constraints.
  • EMA decay λ = 0.15
    Fixed at 0.15; controls how quickly the full-batch gradient estimate mt forgets past mini-batches and therefore the quality of the orthogonality constraint.
axioms (4)
  • standard math Loss gradient is bounded: ∥∇L(θt;D)∥ ≤ ∇Lmax for all t
    Assumption (i) of Theorem 1; required for the O(log T/√T) convergence rate.
  • standard math Stochastic gradient is K-Lipschitz
    Assumption (ii) of Theorem 1; used to bound the difference between gradients at the original and perturbed points.
  • domain assumption EMA mt of mini-batch gradients is a faithful proxy for the true full-batch gradient ∇F L
    Sec. 3.1 and Eq. 9; the entire orthogonality argument (Eq. 13–14) collapses if mt is a poor approximation.
  • domain assumption First-order Taylor expansion adequately approximates the inner max of SAM
    Standard SAM approximation (Eq. 5) inherited without further justification.
invented entities (1)
  • SAMPLe dual-objective loss (Eq. 10) no independent evidence
    purpose: Simultaneously minimize training loss and enforce gradient alignment plus full-batch orthogonality so that prompt parameters settle in flat, low-loss regions.
    The objective is newly postulated; its only external handle is the empirical accuracy improvement on held-out splits.

pith-pipeline@v1.1.0-grok45 · 30145 in / 3131 out tokens · 37293 ms · 2026-07-11T02:58:34.621815+00:00 · methodology

0 comments
read the original abstract

Pre-trained Vision-Language Models (VLMs) like CLIP have proven highly effective as foundation models for various downstream applications. However, prompt learning in VLMs encounters a performance-generalization dilemma: while prompts can be tuned to achieve high accuracy on seen distributions, this tuning process often undermines their generalizability to unseen data. The limited set of learnable prompts, which contextualize and condition the input to steer it toward the task within the pretrained VLM, tends to overfit the training data, leading to a trade-off between task-specific performance and preserving generalization. To address this dilemma, we introduce SAMPLe (Sharpness-Aware Minimization Prompt Learning), a plug-in sharpness-aware optimizer that enhances prompt generalizability by accounting for loss landscape sharpness. Unlike conventional methods, SAMPLe balances exploration and exploitation by satisfying objective function constraints at each step, dynamically adapting to the current optimization state based on the local curvature and gradient properties. This approach reduces overfitting on seen distributions and improves adaptability to unseen data, preserving the generalization potential of pre-trained VLM models. We integrate SAMPLe into multiple prompt learning frameworks, including CoOp, CoCoOp, MaPLe, TCP, and Co-Prompt, demonstrating its effectiveness across diverse methods. Experiments show that SAMPLe elevates prompt learning frameworks and consistently outperforms existing optimizers across diverse settings, establishing itself as a robust, model-agnostic solution for prompt learning.

Figures

Figures reproduced from arXiv: 2607.05727 by Fatemeh Afghah, Fatemeh Lotfi, Hossein Kashiani, Hossein Rajoli, Niloufar Alipour Talemi, Xiaolong Ma.

Figure 1
Figure 1. Figure 1: Normalized loss landscapes of CoOp [47] on ImageNet using F-SAM [25], SAGM [38], and the proposed SAMPLe, each scaled by the maximum absolute loss (among FSAM, SAGM, and SAMPLe) preserving relative depth and sharpness. The visualization demonstrates SAMPLe’s effectiveness in achieving both flatter minima and lower empirical risk. 33]. For downstream applications, prompt learning has emerged as an efficient… view at source ↗
Figure 2
Figure 2. Figure 2: SAMPLe vs. vanilla SAM: Unlike SAM, which applies uniform gradient updates, SAMPLe dynamically adjusts gradients across stages, promoting smoother optimization in early stages and robust convergence in later stages, resulting in improved generalization and performance. unseen classes. CoCoOp [47] improves zero-shot generalization by conditioning prompts on image features, while MaPLe [21] optimizes prompts… view at source ↗
Figure 3
Figure 3. Figure 3: Accuracy and coefficient of variation of SAM, F-SAM, SAGM, and SAMPLe on ImageNet across different values of ρ and λ for various prompt learning methods, including CoOp, Co-CoOp, and MaPLe. and ρt = √ ρ0 t , respectively. Theorem. 1 in Appendix. 8.1 proves that the objective function satisfies following inequality \frac {1}{T} \sum _{t=1}^{T} \left [ \|\nabla \mathcal {L}(\theta _t; \mathcal {D})\|^2 \righ… view at source ↗
Figure 4
Figure 4. Figure 4: Gradient decomposition . 8.3 Datasets Details We describe the 11 datasets and 4 variations of ImageNet used for evaluation, providing details on the number of classes, along with the training and testing sample sizes, as summarized in Table. 4. 8.4 Robustness to Perturbation Radius Unlike SAM and SAGM, which maximize the worst-case perturbation per mini￾batch, SAMPLe and F-SAM mitigate instability by const… view at source ↗
Figure 3
Figure 3. Figure 3: 8.5 Robustness to Whole-Batch Gradient Approximation [PITH_FULL_IMAGE:figures/full_fig_p025_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 51 canonical work pages · 9 internal anchors

  1. [1]

    Sharpness-Aware Minimization Improves Language Model Generalization

    Bahri, D., Mobahi, H., Tay, Y.: Sharpness-aware minimization improves language model generalization. arXiv preprint arXiv:2110.08529 (2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  2. [2]

    In: Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13

    Bossard, L., Guillaumin, M., Van Gool, L.: Food-101–mining discriminative com- ponents with random forests. In: Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13. pp. 446–461. Springer (2014)

    work page 2014

  3. [3]

    In: International conference on machine learning

    Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)

    work page 2020

  4. [4]

    ICLR (2022)

    Chen, X., Hsieh, C.J., Gong, B.: When vision transformers outperform resnets without pre-training or strong data augmentations. ICLR (2022)

    work page 2022

  5. [5]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., Vedaldi, A.: Describing textures in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3606–3613 (2014)

    work page 2014

  6. [6]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 248–255. Ieee (2009)

    work page 2009

  7. [7]

    LPT: Less-overfitting Prompt Tuning for Vision-Language Model

    Ding, C., Gao, X., Dong, S., He, Y., Wang, Q., Kot, A., Gong, Y.: Lobg: less overfitting for better generalization in vision-language model. arXiv preprint arXiv:2410.10247 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [8]

    Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data

    Dziugaite, G.K., Roy, D.M.: Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008 (2017)

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  9. [9]

    In: 2004 Conference on Computer Vision and Pattern Recognition Workshop

    Fei-Fei, L., Fergus, R., Perona, P.: Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In: 2004 Conference on Computer Vision and Pattern Recognition Workshop. pp. 178–178 (2004).https://doi.org/10.1109/CVPR.2004.383

    work page doi:10.1109/cvpr.2004.383 2004

  10. [10]

    Sharpness-Aware Minimization for Efficiently Improving Generalization

    Foret, P., Kleiner, A., Mobahi, H., Neyshabur, B.: Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412 (2020)

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  11. [11]

    In: International Conference on Learning Representations (2021)

    Foret, P., Kleiner, A., Mobahi, H., Neyshabur, B.: Sharpness-aware minimization for efficiently improving generalization. In: International Conference on Learning Representations (2021)

    work page 2021

  12. [12]

    IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing12(7), 2217– 2226 (2019)

    Helber, P., Bischke, B., Dengel, A., Borth, D.: Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing12(7), 2217– 2226 (2019)

    work page 2019

  13. [13]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al.: The many faces of robustness: A critical analysis of out-of-distribution generalization. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 8340–8349 (2021)

    work page 2021

  14. [14]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., Song, D.: Natural adversarial examples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15262–15271 (2021)

    work page 2021

  15. [15]

    In: ICLR (2025),https://openreview.net/forum?id=vf5aUZT0Fz

    Iacob, A., Sani, L., Kurmanji, M., Shen, W.F., Qiu, X., Cai, D., Gao, Y., Lane, N.D.: DEPT: Decoupled embeddings for pre-training language models. In: ICLR (2025),https://openreview.net/forum?id=vf5aUZT0Fz

    work page 2025

  16. [16]

    Ishida, T., Yamane, I., Sakai, T., Niu, G., Sugiyama, M.: Do we need zero training loss after achieving zero training error? ICML’20, JMLR.org (2020) SAMPLe: SAM-based Optimizer for Prompt Learning in VLMs 17

    work page 2020

  17. [17]

    In: Annual Conference of the European Association for Computer Graphics, Eurographics (2022)

    Jetly, V., Ibayashi, H.: Splash in a flash: Sharpness-aware minimization for efficient liquid splash simulation. In: Annual Conference of the European Association for Computer Graphics, Eurographics (2022)

    work page 2022

  18. [18]

    In: International conference on machine learning

    Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International conference on machine learning. pp. 4904–4916. PMLR (2021)

    work page 2021

  19. [19]

    On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

    Keskar, N.S., Mudigere, D., Nocedal, J., Smelyanskiy, M., Tang, P.T.P.: On large- batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836 (2016)

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  20. [20]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Khattak, M.U., Rasheed, H., Maaz, M., Khan, S., Khan, F.S.: Maple: Multi-modal prompt learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19113–19122 (2023)

    work page 2023

  21. [21]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Khattak, M.U., Wasim, S.T., Naseer, M., Khan, S., Yang, M.H., Khan, F.S.: Self-regulating prompts: Foundational model adaptation without forgetting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15190–15200 (2023)

    work page 2023

  22. [22]

    In: International Conference on Machine Learning

    Kim, M., Li, D., Hu, S.X., Hospedales, T.: Fisher sam: Information geometry and sharpness aware minimisation. In: International Conference on Machine Learning. pp. 11148–11161. PMLR (2022)

    work page 2022

  23. [23]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops

    Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3d object representations for fine-grained categorization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 554–561 (2013)

    work page 2013

  24. [24]

    In: International Conference on Machine Learning

    Kwon, J., Kim, J., Park, H., Choi, I.K.: Asam: Adaptive sharpness-aware min- imization for scale-invariant learning of deep neural networks. In: International Conference on Machine Learning. pp. 5905–5914. PMLR (2021)

    work page 2021

  25. [25]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, T., Zhou, P., He, Z., Cheng, X., Huang, X.: Friendly sharpness-aware mini- mization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5631–5640 (2024)

    work page 2024

  26. [26]

    IEEE Transactions on Multimedia pp

    Liu, L., Wang, N., Zhou, D., Liu, D., Yang, X., Gao, X., Liu, T.: Generalizable prompt learning via gradient constrained sharpness-aware minimization. IEEE Transactions on Multimedia pp. 1–14 (2024).https://doi.org/10.1109/TMM. 2024.3521702

    work page doi:10.1109/tmm 2024

  27. [27]

    IEEE Transactions on Machine Learning in Communications and Networking4, 98–114 (2025)

    Lotfi, F., Rajoli, H., Afghah, F.: Task-specific sharpness-aware o-ran resource man- agement using multi-agent reinforcement learning. IEEE Transactions on Machine Learning in Communications and Networking4, 98–114 (2025)

    work page 2025

  28. [28]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition

    Lu, Y., Liu, J., Zhang, Y., Liu, Y., Tian, X.: Prompt distribution learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition. pp. 5206–5215 (2022)

    work page 2022

  29. [29]

    Fine-Grained Visual Classification of Aircraft

    Maji, S., Rahtu, E., Kannala, J., Blaschko, M., Vedaldi, A.: Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 (2013)

    work page internal anchor Pith review Pith/arXiv arXiv 2013

  30. [30]

    In: 2008 Sixth Indian conference on computer vision, graphics & image processing

    Nilsback, M.E., Zisserman, A.: Automated flower classification over a large number of classes. In: 2008 Sixth Indian conference on computer vision, graphics & image processing. pp. 722–729. IEEE (2008)

    work page 2008

  31. [31]

    Sensitivity and Generalization in Neural Networks: an Empirical Study

    Novak, R., Bahri, Y., Abolafia, D.A., Pennington, J., Sohl-Dickstein, J.: Sensi- tivity and generalization in neural networks: an empirical study. arXiv preprint arXiv:1802.08760 (2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  32. [32]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Parkhi, O.M., Vedaldi, A., Zisserman, A., Jawahar, C.: Cats and dogs. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3498–3505. IEEE (2012) 18 H. Rajoli et al

    work page 2012

  33. [33]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)

    work page 2021

  34. [34]

    Advances in Neural Information Processing Systems38, 168772–168794 (2026)

    Rajoli Nowdeh, H., Ji, J., Ma, X., Afghah, F.: Modality-aware sam: Sharpness- aware-minimization driven gradient modulation for harmonized multimodal learning. Advances in Neural Information Processing Systems38, 168772–168794 (2026)

    work page 2026

  35. [35]

    Recht, B., Roelofs, R., Schmidt, L., Shankar, V.: Do imagenet classifiers generalize to imagenet? In: International Conference on Machine Learning. pp. 5389–5400. PMLR (2019)

    work page 2019

  36. [36]

    UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

    Soomro, K.: Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)

    work page internal anchor Pith review Pith/arXiv arXiv 2012

  37. [37]

    Advances in Neural Information Processing Systems32(2019)

    Wang, H., Ge, S., Lipton, Z., Xing, E.P.: Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems32(2019)

    work page 2019

  38. [38]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wang, P., Zhang, Z., Lei, Z., Zhang, L.: Sharpness-aware gradient matching for domain generalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3769–3778 (2023)

    work page 2023

  39. [39]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: Large-scale scene recognition from abbey to zoo. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3485–3492. IEEE (2010)

    work page 2010

  40. [40]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yao, H., Zhang, R., Xu, C.: Visual-language prompt tuning with knowledge-guided context optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6757–6767 (2023)

    work page 2023

  41. [41]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yao, H., Zhang, R., Xu, C.: Tcp: Textual-based class-aware prompt tuning for visual-language model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 23438–23448 (2024)

    work page 2024

  42. [42]

    FILIP: Fine-grained Interactive Language-Image Pre-Training

    Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., Xu, C.: Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783 (2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  43. [43]

    Medical image analysis 12(5), 603–615 (2008)

    Yeo, B.T., Sabuncu, M.R., Desikan, R., Fischl, B., Golland, P.: Effects of registration regularization and atlas sharpness on segmentation accuracy. Medical image analysis 12(5), 603–615 (2008)

    work page 2008

  44. [44]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., Beyer, L.: Lit: Zero-shot transfer with locked-image text tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18123– 18133 (2022)

    work page 2022

  45. [45]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    Zhang, J., Huang, J., Jin, S., Lu, S.: Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    work page 2024

  46. [46]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhang, X., Xu, R., Yu, H., Zou, H., Cui, P.: Gradient norm aware minimization seeks first-order flatness and improves generalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20247–20257 (2023)

    work page 2023

  47. [47]

    In: 2022 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR)

    Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Conditional prompt learning for vision- language models. In: 2022 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR). pp. 16795–16804 (2022).https://doi.org/10.1109/ CVPR52688.2022.01631

    work page arXiv 2022

  48. [48]

    International Journal of Computer Vision130(9), 2337–2348 (2022)

    Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. International Journal of Computer Vision130(9), 2337–2348 (2022)

    work page 2022

  49. [49]

    Pattern Recognition p

    Zhou, Z., Dong, S., Ding, C., Gao, X., He, Y., Gong, Y.: Diversity covariance-aware prompt learning for vision-language models. Pattern Recognition p. 112806 (2025) SAMPLe: SAM-based Optimizer for Prompt Learning in VLMs 19

    work page 2025

  50. [50]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhu, B., Niu, Y., Han, Y., Wu, Y., Zhang, H.: Prompt-aligned gradient for prompt tuning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15659–15669 (2023)

    work page 2023

  51. [51]

    Zhuang, J., Gong, B., Yuan, L., Cui, Y., Adam, H., Dvornek, N., Tatikonda, S., Duncan, J., Liu, T.: Surrogate gap minimization improves sharpness-aware training. ICLR (2021) SAMPLe: SAM-based Optimizer for Prompt Learning in VLMs 1 8 Appendix In this supplementary material, we begin with the proof of Theorem1, followed by a comprehensive comparison of var...

    work page 2021