Pith. sign in

REVIEW 2 major objections 5 minor 72 references

One Last Attention for Your Vision-Language Model

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adapting the final cross-modal matrix, not intermediate features, improves CLIP fine-tuning.

desk verdict RAda is a simple, broadly evaluated mask-attention add-on for CLIP that delivers credible gains in three fine-tuning settings, but the information-theoretic appendix is wrong and should not be relied on. read the letter →

arxiv 2507.15480 v2 pith:3EIDTBYI submitted 2025-07-21 cs.CV

Liang Chen , Ghazi Shazan Ahmad , Tianjun Yao , Lingqiao Liu , Zhiqiang Shen This is my paper
classification cs.CV
keywords vision-languagemodelsCLIPfine-tuningrationalmatrixattentionmasktest-timetrainingbase-to-novelgeneralizationparameter-efficientfull
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RAda is a fine-tuning method for vision-language models that does not touch the image or text encoders. It attaches one lightweight attention layer at the very end of CLIP, where the layer learns a mask over the rational matrix—the entry-wise product of image and text features whose column sums are the classification logits. The paper claims that calibrating this fused representation improves CLIP in all three regimes it tests: full fine-tuning, frozen-encoder few-shot base-to-novel transfer, and test-time training with unlabeled data, while adding only a few lines of code. A reader should care because most existing adaptation methods modify the two modalities separately, and this work makes the case that the final cross-modal interaction is itself a cheap and effective place to adapt.

What carries the argument

The central object is the rational matrix $R$, the entry-wise product of the normalized image feature and the normalized text features; summing $R$ over its feature dimension reproduces CLIP's class logits. The carrying mechanism is a single multi-query attention layer placed after this matrix: it uses the image feature, the text feature, and the rational matrix itself as queries, all attending to the rational matrix as key and value, and averages the resulting attention outputs to form a mask. The mask is offset by the all-ones matrix and the attention output projection is zero-initialized, so the first update leaves CLIP's initial prediction unchanged. Under frozen encoders, an $L_2$ penalty pulling the mask back toward the all-ones matrix is what the paper credits for preserving CLIP's zero-shot generalization while still allowing adaptation.

What would settle it

Train RAda under frozen encoders on a base set whose class names are deliberately misleading, such as swapped labels, and evaluate on semantically meaningful novel classes: the regularizer predicts novel accuracy should stay near CLIP's zero-shot level, while a failure to recover any novel accuracy—or an equally good novel accuracy without the regularizer—would undermine the claim that the mask regularization, not the text prior, preserves generalization.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the final fused representation in CLIP—the rational matrix $R$ with entries $R_{i,j}=f_j h_{i,j}$—can be adapted by a learned input-dependent mask produced by one attention layer, and that this is enough to match or beat much heavier adaptation schemes. In full fine-tuning, RAda-FT sequentially trains the mask and then all parameters, improving out-of-distribution accuracy over vanilla fine-tuning on ImageNet. In efficient fine-tuning with frozen encoders, RAda raises average base accuracy by roughly 13 points on the 11-dataset base-to-novel benchmark while essentially preserving novel-class accuracy. In test-time training, the same mask trained with entropy minimization and a pull-back regularizer improves CLIP on four distribution shifts. The paper interprets these gains through the rational matrix's role as a sufficient statistic for the prediction, with adapting $R$ dominating adaptations of $f$ or $h$ alone.

Load-bearing premise

The load-bearing premise is that CLIP's text-driven class descriptions remain a reliable starting point for every class, including ones never seen in training, because the method's regularizer pulls the learned mask back toward treating every element equally; if those text descriptions are poorly aligned with the target classes, the regularizer will limit adaptation.

Editorial extensions

If this is right

  • Under frozen encoders and 16-shot base classes, RAda raises average base accuracy from 69.34 to 82.16 on the 11-dataset benchmark while novel accuracy stays nearly unchanged, improving the harmonic mean from 71.70 to 77.94.
  • In full fine-tuning on ImageNet, RAda-FT beats plain FT on every reported out-of-distribution shift and lifts the average OOD accuracy from 53.8 to 61.3, while remaining competitive with existing full-tuning methods on in-distribution accuracy.
  • At test time with only unlabeled data, RAda improves CLIP on all four distribution shifts and its average of 63.50 surpasses the test-time prompt-tuning baseline's 60.81; with ImageNet-pretrained initialization it leads all compared methods at 64.26.
  • RAda is orthogonal to prompt and adapter tuning: combining RAda with adapters inside both encoders reaches an average harmonic mean of 80.08 on the base-to-new benchmark, above either ingredient alone.
  • The single update step of RAda uses roughly a tenth of the computation and about half the memory of the closest multi-modal prompt-based method, and it does not require a transformer-only image encoder.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the mask acts directly on logits, the same module could be inserted during pretraining rather than fine-tuning; the paper names pretraining as a future direction but does not test whether the mask would alter the learned embeddings themselves.
  • If the rational matrix is a sufficient statistic for the label, as the supplementary material argues, then any gain from tuning $f$ or $h$ alone should be reproducible by a suitably constrained mask on $R$; this predicts RAda will keep matching or beating modality-specific adapters as datasets and backbones change.
  • A practical extension the paper does not explore is continual learning: since the mask can be reset to identity, it could restore zero-shot behavior after adapting to a new task without touching the encoder weights.
  • On tasks where the final prediction is not a dot-product logit, such as captioning or visual question answering, the rational matrix would need a surrogate fusion object; the paper lists these tasks as future work without specifying what that surrogate would be.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes RAda (Rational Adaptation), a lightweight fine-tuning module for vision-language models such as CLIP. RAda attaches a single attention layer after the final similarity computation, which produces a learned mask applied elementwise to the rational matrix (the outer product of image and text features), thereby calibrating contributions of individual fused features to the logits. The method is evaluated in three settings: full fine-tuning on ImageNet with five OOD benchmarks, efficient fine-tuning in base-to-novel generalization across 11 datasets, and test-time training on four distribution shifts. In all three settings RAda improves over the CLIP baseline and is competitive with specialized methods, while adding minimal code and compute. The paper also claims theoretical support for adapting fused representations over isolated modalities, presented in the supplementary material.

Significance. If the empirical findings hold, RAda is a simple, broadly applicable adaptation idea that is orthogonal to existing prompt- and adapter-based methods, and it extends the concept of the rational matrix to the VLM decision stage. The evaluation is unusually broad for a method of this type: it covers three different fine-tuning paradigms, 11+ datasets, multiple backbones (ViT and ResNet), and multiple VLM backbones (CLIP, OpenCLIP, SigLIP, ALIGN), with efficiency comparisons. The implementation is transparent (Algorithm 1, released code), and the initialization M=1 exactly reproduces the baseline at step zero, which is an honest design. The principal weakness is the theoretical support in Supplementary Sec. A, which contains a genuine proof error that should be corrected or withdrawn. The empirical tables themselves are the primary support for the versatility claim and are not invalidated by this error.

major comments (2)
  1. [Supplementary Sec. A, Lemma A.2] The proof of Eq. (10) misuses the data processing inequality. For the mask M actually learned by the attention layer (Algorithm 1, Eq. (6)), R∘(M_f ⊗ 1_K^T) is not a deterministic function of R∘M unless M is constrained to be column-wise constant, and no such constraint is imposed. DPI therefore does not yield I(Y;R∘M) ≥ I(Y;R∘(M_f ⊗ 1_K^T)) for the learned M. At best, the argument shows that maximizing over all M gives at least as much mutual information as using the constrained mask, which is a triviality and does not establish the lemma. The same defect propagates to Lemma A.3. Since Sec. 1 explicitly cites these results as theoretical support, the claim of theoretical support is overstated. The empirical results in Tables 1–4 are not affected, but the theory should be corrected or removed.
  2. [Supplementary Sec. A] The information-theoretic statements are not well-defined because the text embedding h is fixed for a given dataset in the EFT setting, yet Lemma A.1–A.3 treat h as a random variable. No joint distribution over (Y, f, h) is specified, so the mutual informations I(Y;h), I(Y;f∘M_f, h), etc. lack a clear meaning. This is a separate technical flaw that must be addressed if the theoretical section is retained.
minor comments (5)
  1. [Abstract] The term "RationalAdaptaion" should be "Rational Adaptation."
  2. [Sec. 1] There is a missing space in "i.e.rational matrix" in the Introduction; please also check for similar spacing issues elsewhere.
  3. [Sec. 5.1, Table 4] The variant label "W/OLreg" should be "W/O L_reg" to clearly indicate removal of the regularization term.
  4. [Supplementary Sec. B, Figure 3] The t-SNE caption mentions "18 clusters" and "19 clusters" but the plots are not labeled; please add a legend or annotation so the reader can verify the cluster counts.
  5. [Algorithm 1] The comment "#BxD, KxD" for f, h is slightly ambiguous; it would be clearer to write "# f: BxD, h: KxD."

Circularity Check

1 steps flagged · score 2.0 of 10

Central empirical versatility claim is externally benchmarked and independent; only a cosmetic renaming of the standard logit decomposition as a 'rational matrix' plus one non-load-bearing self-citation.

  1. renaming known result [Sec. 1 (contribution 1) and Sec. 3.2, Eqs. (2)-(3)]
    "We suggest extending the concept of rational matrix [5] to CLIP. ... Similarly, the i-th logit in the CLIP result can be represented as P_D_{j=1} R_{i,j}. ... Note that reformulating Eq.(1) to Eq.(3) does not introduce any additional parameters."

    The paper's first stated contribution claims novelty in 'extends the concept of rational matrix from the classical classifying system to VLMs,' but by its own equations the rational matrix R_{i,j} = f_j h_{i,j} is exactly the integrand of the standard CLIP logit <f, h_i> = sum_j R_{i,j}, and Eq. (3) is admitted to be a parameter-free rewrite of Eq. (1). The 'fused-representation insight' is therefore equivalent, by construction, to the well-known per-element logit decomposition; the actual adaptation mechanism is the attention mask in Eq. (4), whose gains are validated against external baselines (FLYP, CoOp, MaPLe, TPT, PromptAlign) in Tables 1-3.

full rationale

The central claim - RAda consistently benefits the CLIP baseline and performs comparably to existing arts in FFT, EFT, and TTT - is established against external methods on standard benchmark datasets (ImageNet and its OOD variants, 11 base-to-new datasets, 4 TTT distribution shifts), so it does not reduce to a fit or to the authors' prior work. The M=1 initialization is an honest design: Eq. (4) with M=1 reduces exactly to the baseline Eq. (3) at step zero, and the regularizer L_reg = ||M-1||^2 (Eq. 7) is empirically ablated, with Table 4 showing novel accuracy dropping from 74.14 to 71.58 without it; zero-shot preservation is earned, not forced. Supp. A (Lemmas A.2-A.3) is a genuine rigor defect, not circularity: the data processing inequality is misapplied because R◦(M_f ⊗ 1_K) is not a deterministic function of a general learned R◦M, and h is treated as random when it is fixed per dataset; this undermines an advertised supporting proof but does not transfer to the externally benchmarked tables. The only circularity-adjacent element is conceptual: the 'rational matrix' extension to CLIP is, by the paper's own Eq. (3), a parameter-free reformulation of the standard logit decomposition, and the term is attributed to same-author citation [5]. That renaming is cosmetic and not load-bearing for the accuracy gains, which is consistent with a score of 2 rather than higher.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method's trained weights are learned from data and are not listed separately, since they are standard neural network parameters. The load-bearing hand-set choices are the regularizer weight alpha (1.5 in EFT), the TTT number of steps, the confidence percentile, and the number of augmented views. The axioms are the CLIP model definition, the sufficiency of the rational matrix, the information-bottleneck heuristic, the classifier-anchor assumption in FFT, and the regularizer's zero-shot preservation assumption. No new physical entities are introduced; the rational matrix is taken from the authors' prior work and the mask is a learned model component.

free parameters (4)
  • Regularization weight alpha for L_reg in EFT/TTT = 1.5 (EFT), 1.0 default
    The objective in Sec. 3.3 includes L_reg; supplementary Sec. E states that alpha = 1.5 'yields relatively better results' in EFT and is fixed across EFT experiments.
  • TTT update steps = 3
    Sec. 4.3 sets three gradient steps per test sample.
  • TTT confidence percentile = 10%
    Sec. 4.3 selects the top 10% confident predictions with lowest entropy for computing the test-time loss.
  • Number of augmented views in TTT = 63 + 1 original (batch of 64)
    Sec. 4.3: 'we obtain 63 of its augmented view ... to form a batch of 64 samples'.
assumptions (5)
  • domain assumption CLIP predictions are computed as the dot product of l2-normalized image and text features (Eq. 1).
    This is the standard CLIP model definition used throughout Sec. 3.
  • standard math The rational matrix R (elementwise product f_j h_ij) is a sufficient statistic for the label given the features, because logits are row sums of R (Lemma A.1).
    Follows by definition from Eq. (1)-(3); used to justify focusing on R.
  • domain assumption Higher mutual information between the adapted representation and the label generally correlates with better predictive performance (information bottleneck principle, cited [41]).
    Invoked in supplementary Sec. A to argue that adapting R is better; this is a heuristic from prior literature, not a theorem guaranteed for the specific loss and architecture.
  • domain assumption The text embeddings h can act as classifier weights in the rational matrix, so a linear classifier in FFT initialized by text features preserves the rational-matrix interpretation (Eq. 14).
    Used in the FFT setting to replace the text encoder; assumes the text-feature initialization is a reasonable anchor.
  • ad hoc to paper Regularizing M toward the all-one matrix preserves the zero-shot ability of CLIP when encoders are frozen.
    This is the design assumption behind L_reg in Eq. (7); the paper validates it empirically but does not prove it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of One Last Attention for Your Vision-Language Model." pith.science (2026). https://pith.science/paper/3EIDTBYI

@misc{pith2026250715480,
  author       = {Pith},
  title        = {Pith review of: One Last Attention for Your Vision-Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3EIDTBYI}},
  note         = {Machine review of arXiv:2507.15480}
}
read the original abstract

Pretrained vision-language models (VLMs), such as CLIP, achieve remarkable zero-shot performance, yet their downstream potential hinges on effective fine-tuning. Most adaptation methods typically focus on refining representation from separate modalities (text or vision) but neglect the critical role of their fused representations in the decision-making process, \emph{\ie} rational matrix that drives the final prediction. To bridge the gap, we propose a simple yet effective \textbf{R}ational \textbf{Ada}ptaion ({RAda}) to explicitly exploit the final fused representation during fine-tuning. RAda employs a learned mask, obtained from a lightweight attention layer attached at the end of a VLM, to dynamically calibrate the contribution of each element in the rational matrix, enabling targeted adjustments to the final cross-modal interactions without incurring costly modifications to intermediate features. Experiments in different settings (i.e., updating, or freezing pretrained encoders in adaptation, and test-time training that can only access the unlabeled test data) show that RAda serves as a versatile fine-tuning technique, improving the baseline with minimal code and performing comparably against current arts in most settings. Code is available at \href{https://github.com/khufia/RAda/tree/main}{github.com/khufia/RAda}.

Figures

Figures reproduced from arXiv: 2507.15480 by the authors.

Figure 1
Figure 1. Comparisons between fine-tuning ideas for VLMs that [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Rational matrix [5] in the CLIP decision-making process for a given image, where predictions (i.e. logits) are computed by summing each column. It fuses text and visual features and pro￾vides a fine-grained characterization of different predictions. For instance, a commonly-used p =“a photo of a” can be adopted as the prefix prompts, and each text representation hi ∈ R D is then obtained via hi = Ft([p, Yi ], θt), g… view at source ↗
Figure 4
Figure 4. Distributions of values in the learned mask [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Heatmaps of the rational adaptation process. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Evaluations of RAda with different weights ( [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 54 canonical work pages

  1. [1]

    Align your prompts: Test-time prompting with distribution align- ment for zero-shot generalization

    Jameel Abdul Samadh, Mohammad Hanan Gani, Noor Hus- sein, Muhammad Uzair Khattak, Muhammad Muzammal Naseer, Fahad Shahbaz Khan, and Salman H Khan. Align your prompts: Test-time prompting with distribution align- ment for zero-shot generalization. InNeurIPS, 2023. 3, 7

    work page 2023

  2. [2]

    Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

    Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurIPS, 2019. 5

    work page 2019

  3. [3]

    Food-101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. InECCV, 2014. 5

    2014

  4. [4]

    Improved test-time adaptation for domain gen- eralization

    Liang Chen, Yong Zhang, Yibing Song, Ying Shan, and Lingqiao Liu. Improved test-time adaptation for domain gen- eralization. InCVPR, 2023. 3

    work page 2023

  5. [5]

    Domain generalization via rationale invariance

    Liang Chen, Yong Zhang, Yibing Song, Anton Van Den Hen- gel, and Lingqiao Liu. Domain generalization via rationale invariance. InICCV, 2023. 1, 2, 3, 4

    work page 2023

  6. [6]

    Lfme: A simple framework for learning from multiple experts in domain generalization.NeurIPS, 2024

    Liang Chen, Yong Zhang, Yibing Song, Zhiqiang Shen, and Lingqiao Liu. Lfme: A simple framework for learning from multiple experts in domain generalization.NeurIPS, 2024. 3

    work page 2024

  7. [7]

    A causal inspired early-branching structure for domain generalization.IJCV, 132(9):4052–4072, 2024

    Liang Chen, Yong Zhang, Yibing Song, Zhen Zhang, and Lingqiao Liu. A causal inspired early-branching structure for domain generalization.IJCV, 132(9):4052–4072, 2024. 3

    work page 2024

  8. [8]

    Vision transformer adapter for dense predictions

    Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. InICLR, 2023. 1, 3

    work page 2023

Show all 72 references
  1. [9]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. InCVPR, 2014. 6

    2014

  2. [10]

    John Wiley & Sons, 1999

    Thomas M Cover.Elements of information theory. John Wiley & Sons, 1999. 1

    1999

  3. [11]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InCVPR, 2009. 5

    2009

  4. [12]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Trans- formers for image recognition at scale. InICLR, 2021. 2, 4

    2021

  5. [13]

    Magma–multimodal augmentation of generative models through adapter-based finetuning

    Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. Magma–multimodal augmentation of generative models through adapter-based finetuning. InEMNLP Findings, 2022. 1, 3

    2022

  6. [14]

    Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories

    Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories. In CVPRW, 2004. 5

    2004

  7. [15]

    Clip-adapter: Better vision-language models with feature adapters.IJCV, 132(2):581–595, 2024

    Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters.IJCV, 132(2):581–595, 2024. 1, 2, 3, 4, 5, 6

    2024

  8. [16]

    Finetune like you pretrain: Im- proved finetuning of zero-shot vision models

    Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Im- proved finetuning of zero-shot vision models. InCVPR,

  9. [17]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InCVPR,

  10. [18]

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. 6

    2019

  11. [19]

    The many faces of robustness: A critical analysis of out-of-distribution generalization.ICCV, 2021

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization.ICCV...

    2021

  12. [20]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Stein- hardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021. 5

    2021

  13. [21]

    Open- clip, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open- clip, 2021. If you use this software, please cite it as below. 4

    2021

  14. [22]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. InICML, 2021. 1, 2, 4

    2021

  15. [23]

    Vi- sual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. InECCV, 2022. 1, 2, 3, 4, 6, 8

    2022

  16. [24]

    Maple: Multi-modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. InCVPR, 2023. 2, 3, 5, 6, 7, 8

    2023

  17. [25]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCVW, 2013. 5

    2013

  18. [26]

    Fine-tuning can distort pretrained fea- tures and underperform out-of-distribution

    Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained fea- tures and underperform out-of-distribution. InICLR, 2022. 1, 2, 3, 5

    2022

  19. [27]

    Su- pervision exists everywhere: A data efficient contrastive language-image pre-training paradigm

    Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Su- pervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. InICLR, 2022. 2

    2022

  20. [28]

    Ttt++: When does self-supervised test-time training fail or thrive? InNeurIPS, 2021

    Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Ttt++: When does self-supervised test-time training fail or thrive? InNeurIPS, 2021. 3

    2021

  21. [29]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. In ICLR, 2019. 2

    2019

  22. [30]

    A simple long-tailed recognition baseline via vision-language model

    Teli Ma, Shijie Geng, Mengmeng Wang, Jing Shao, Jiasen Lu, Hongsheng Li, Peng Gao, and Yu Qiao. A simple long-tailed recognition baseline via vision-language model. arXiv:2111.14745, 2021. 1, 3, 4

    2021 arXiv

  23. [31]

    Fine-grained visual classi- fication of aircraft.arXiv:1306.5151, 2013

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classi- fication of aircraft.arXiv:1306.5151, 2013. 6

    2013 arXiv

  24. [32]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008. 5

    2008

  25. [33]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. InCVPR, 2012. 5

    2012

  26. [34]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InICML, 2021. 1, 2, 6, 4

    2021

  27. [35]

    Do imagenet classifiers generalize to im- agenet? InICML, 2019

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to im- agenet? InICML, 2019. 5

    2019

  28. [36]

    Towards parameter-efficient integration of pre- trained language models in temporal video grounding

    Erica K Shimomoto, Edison Marrese-Taylor, Hiroya Taka- mura, Ichiro Kobayashi, Hideki Nakayama, and Yusuke Miyao. Towards parameter-efficient integration of pre- trained language models in temporal video grounding. In ACL Findings, 2023. 3

    2023

  29. [37]

    Test- time prompt tuning for zero-shot generalization in vision- language models

    Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test- time prompt tuning for zero-shot generalization in vision- language models. InNeurIPS, 2022. 1, 2, 3, 7

    2022

  30. [38]

    Ucf101: A dataset of 101 human actions classes from videos in the wild.arXiv:1212.0402, 2012

    K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild.arXiv:1212.0402, 2012. 6

    2012 arXiv

  31. [39]

    Test-time training with self- supervision for generalization under distribution shifts

    Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self- supervision for generalization under distribution shifts. In ICML, 2020. 3, 7

    2020

  32. [40]

    Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks

    Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. InCVPR, 2022. 1, 3

    2022

  33. [41]

    The information bottleneck method.arXiv preprint physics/0004057, 2000

    Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method.arXiv preprint physics/0004057, 2000. 1

    2000 arXiv

  34. [42]

    Visualizing data using t-sne.JMLR, 9(11), 2008

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.JMLR, 9(11), 2008. 2

    2008

  35. [43]

    Attention is all you need

    A Vaswani. Attention is all you need. InNeurIPS, 2017. 4

    2017

  36. [44]

    Tent: Fully test-time adaptation by entropy minimization

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Ol- shausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. InICLR, 2021. 3, 4

    2021

  37. [45]

    Learning robust global representations by penalizing local predictive power

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. InNeurIPS, 2019. 5

    2019

  38. [46]

    Robust fine-tuning of zero-shot models

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gon- tijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. InCVPR, 2022. 3, 5, 2

    2022

  39. [47]

    Sun database: Large-scale scene recognition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. InCVPR, 2010. 6

    2010

  40. [48]

    Explicit inductive bias for transfer learning with convolutional net- works

    LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional net- works. InICML, 2018. 5

    2018

  41. [49]

    Mma: Multi-modal adapter for vision-language models

    Lingxiao Yang, Ru-Yuan Zhang, Yanchen Wang, and Xiao- hua Xie. Mma: Multi-modal adapter for vision-language models. InCVPR, 2024. 5, 6

    2024

  42. [50]

    Visual- language prompt tuning with knowledge-guided context op- timization

    Hantao Yao, Rui Zhang, and Changsheng Xu. Visual- language prompt tuning with knowledge-guided context op- timization. InCVPR, 2023. 5, 6

    2023

  43. [51]

    Filip: Fine-grained interactive language-image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. InICLR, 2022. 2

    2022

  44. [52]

    Florence: A new foundation model for computer vision.arXiv:2111.11432, 2021

    Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision.arXiv:2111.11432, 2021. 2

    2021 arXiv

  45. [53]

    On the test-time zero- shot generalization of vision-language models: Do we really need prompt learning? InCVPR, 2024

    Maxime Zanella and Ismail Ben Ayed. On the test-time zero- shot generalization of vision-language models: Do we really need prompt learning? InCVPR, 2024. 3

    2024

  46. [54]

    Unified vision and language prompt learning.arXiv:2210.07225, 2022

    Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning.arXiv:2210.07225, 2022. 1, 3

    2022 arXiv

  47. [55]

    Lit: Zero-shot transfer with locked-image text tuning

    Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, 2022. 2

    2022

  48. [56]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InICCV, 2023. 4

    2023

  49. [57]

    Dept: Decoupled prompt tuning

    Ji Zhang, Shihan Wu, Lianli Gao, Heng Tao Shen, and Jingkuan Song. Dept: Decoupled prompt tuning. InCVPR,

  50. [58]

    Tip-adapter: Training-free clip-adapter for better vision- language modeling

    Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision- language modeling. InECCV, 2022. 1, 3

    2022

  51. [59]

    Contrastive learning of medical visual representations from paired images and text

    Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. InMachine Learning for Healthcare Conference, 2022. 2

    2022

  52. [60]

    Test-time adaptation with CLIP reward for zero-shot gener- alization in vision-language models

    Shuai Zhao, Xiaohan Wang, Linchao Zhu, and Yi Yang. Test-time adaptation with CLIP reward for zero-shot gener- alization in vision-language models. InICLR, 2024. 3

    2024

  53. [61]

    Conditional prompt learning for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Zi- wei Liu. Conditional prompt learning for vision-language models. InCVPR, 2022. 1, 2, 3, 4, 5, 6, 7

    2022

  54. [62]

    Learning to prompt for vision-language models.IJCV, 130(9):2337–2348, 2022

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models.IJCV, 130(9):2337–2348, 2022. 1, 2, 3, 5, 6, 7, 4

    2022

  55. [63]

    Prompt-aligned gradient for prompt tuning

    Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. InICCV,

  56. [65]

    Theoretical support for using fused information in Sec. A

  57. [66]

    Visualizations for RAda in Sec. B

  58. [67]

    Detailed settings of RAda in the FFT setting in Sec. C

  59. [68]

    Extending other fine-tuning ideas in FFT in Sec. D

  60. [69]

    Ablation studies onL reg in Sec. E

  61. [70]

    Experiments with different backbones in Sec. F

  62. [71]

    Experiments with different VLMs in Sec. G

  63. [72]

    Experiments for using more attention layers in Sec. H. A. Theoretical Support for Utilizing Fused In- formation Over Isolated Representation Our empirical observations indicate that adapting the ratio- nal matrix yields better performance than adapting the dif- ferent modaliti...

  64. [2023]

    5, 6 One Last Attention for Your Vision-Language Model Supplementary Material In this supplementary material, we provide,

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.