Pith. sign in

REVIEW 3 major objections 5 minor 77 references

LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Tuning only low-rank adapters in CLIP's image encoder at test time is enough to beat test-time prompt tuning in vision-language models.

desk verdict Plausible and efficient image-side LoRA TTT for VLMs, but the headline numbers rest on hyperparameters tuned on the same benchmarks and the calibration claim is contradicted by the paper's own table. read the letter →

arxiv 2502.02069 v1 pith:N3FM7D3N submitted 2025-02-04 cs.CV

Yuto Kojima , Jiarui Xu , Xueyan Zou , Xiaolong Wang This is my paper · ORCID
classification cs.CV
keywords test-timetraininglow-rankadaptationvision-languagemodelsCLIPzero-shotclassificationmarginalentropyminimizationmaskedautoencoderreconstructionparameter-efficientfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LoRA-TTT claims that adapting a vision-language model at test time can be done entirely on the vision side: instead of optimizing a text prompt, insert low-rank adapter matrices (LoRA) into the last transformer layers of CLIP's image encoder and update only those matrices on each test image. The update uses a combination of marginal entropy minimization over augmented views and a masked-reconstruction loss that asks the encoder to keep the class token of a heavily masked crop close to the class token of the original. On CLIP-ViT-B/16 this raises zero-shot top-1 accuracy by an average of 5.79 percentage points on four out-of-distribution ImageNet-derived datasets and 1.36 points across ten fine-grained datasets, surpassing test-time prompt tuning while using less memory and runtime. Because text features are precomputed, the text encoder is not needed during adaptation, and the method works with any prompt or prompt ensemble. The paper positions this as a practical path to per-instance adaptation on memory-constrained devices without external models or caches.

What carries the argument

The load-bearing mechanism is a low-rank bottleneck on the vision side: LoRA adapters (two small matrices A and B per projection, rank 16, initialised with A random and B zero) confine the test-time update to a low-intrinsic-rank subspace of the last two transformer layers, which prevents the catastrophic forgetting that the authors observe when the full image encoder is tuned. Around that bottleneck sits a decoder-free reconstruction objective: the mean squared error between the CLS tokens of masked and unmasked augmented views, a form of masked autoencoding that needs only the frozen image encoder and can be combined with marginal-entropy minimization at no extra memory cost. The masking strategy, confidence cutoff, rank-scale pairing, and choice of attention matrices (value projections work best) are all tuned to keep this update stable on a single image; the method resets the LoRA weights after each instance.

What would settle it

On a labeled held-out set from the same benchmark domains, compare the gradient of the reconstruction loss with the gradient of the supervised cross-entropy loss for the same images: if their cosine similarity is near zero, or if pairing each masked image with a different image's summary feature produces the same accuracy gain, then the reconstruction term is not supplying class-relevant signal.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the image encoder, not the prompt, is the right place to spend test-time adaptation budget in CLIP. LoRA-TTT applies LoRA to the query, key, value, and output projections of the last two transformer blocks of the ViT image encoder, keeps all pretrained weights frozen, and performs a single AdamW step per test instance on the resulting 0.2M trainable parameters. The gradient comes from two label-free losses computed on the top 10% highest-confidence augmented views: the marginal-entropy loss inherited from test-time prompt tuning, and a reconstruction loss $L_{MAE} = MSE(g(X)_{cls}, g(mask(X))_{cls})$ that matches the CLS token of a 50%-masked version of the image to the CLS token of the unmasked image, without any decoder. The authors report that this combination outperforms test-time prompt tuning (TPT) on 15 datasets and, unlike entropy-only adaptation, keeps expected calibration error close to the frozen baseline. They also show the reconstruction loss alone is competitive on fine-grained datasets, and that the gains hold when the method is combined with prompt ensembles or CoOp-style prompts.

Load-bearing premise

The load-bearing premise is that matching a half-masked crop's summary feature to the original crop's summary feature on one test image steers the model toward features that help distinguish classes; if that gradient is class-orthogonal, the reconstruction term contributes nothing beyond the entropy loss.

Editorial extensions

If this is right

  • Test-time adaptation for VLMs can run entirely in the image encoder, so the text encoder is unnecessary once text features are precomputed; this removes the main memory and runtime bottleneck of prompt tuning.
  • The decoder-free reconstruction loss provides a label-free signal that improves calibration compared with entropy-minimization-only TTT, making the adapted model safer for high-stakes use.
  • Because the adapted parameters are image-side, the same adapted encoder works with any text prompt, including ensembles or few-shot-learned prompts, and gains from those prompts add independently.
  • The method's per-instance, reset-after-use protocol and 0.2M trainable parameters fit the constraints of streaming data and memory-limited edge devices.
  • Among TTT methods that do not use external models, caches, or domain knowledge, LoRA-TTT reports the best average accuracy on both the OOD and fine-grained benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not shown in the paper, the success of last-two-layer LoRA suggests a general rule for test-time adaptation of ViT backbones: only the deepest semantic layers need to move, so other lightweight adapters or selective layer freezing should produce similar gains.
  • The paper leaves implicit that the class-token reconstruction objective is not tied to CLIP's contrastive training; if the mechanism is really semantic self-consistency, the same loss should improve other ViT-based classifiers at test time.
  • A testable extension of the calibration finding is that feature-consistency losses generally counteract entropy-minimization overconfidence, which would make the design principle portable beyond vision-language models.
  • The ablation logic implies a concrete prediction the authors do not test: the confidence cutoff (10%) should matter more than the mask ratio (50%), so an adaptive per-image cutoff could squeeze out further gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes LoRA-TTT, a test-time training method for vision-language models that freezes CLIP and optimizes only LoRA adapters injected into the later transformer layers of the image encoder. Adaptation is performed per test instance with a single optimizer step on a marginal entropy minimization loss and a masked-image consistency loss computed from the class token of a 50%-masked augmented view. The authors report average top-1 gains of 5.79% on an OOD benchmark and 1.36% on a fine-grained benchmark over CLIP-ViT-B/16, claim better calibration and lower memory/runtime than prompt-based TTT, and show prompt-agnostic behavior across several text-prompt initializations. The paper includes extensive ablations on LoRA rank, scale, layer placement, attention matrices, and masking strategy.

Significance. If the reported numbers are reproducible, the contribution is meaningful: it is, to my knowledge, the first demonstration that image-side LoRA adaptation at test time can be competitive with or superior to prompt-side test-time tuning for CLIP, without external models or cache, and with lower memory and runtime. The prompt-agnostic property and the ablation study on LoRA design are useful, and the efficiency measurements are informative. However, the central evidence is weakened by hyperparameters that appear to be selected on the same evaluation benchmarks, by the absence of error bars, repeated runs, and released code, and by a calibration claim that is contradicted by the paper's own Table S4. The significance of the result as currently established is therefore moderate.

major comments (3)
  1. [§4.1, Tables S1–S2, Fig. S5a] The headline accuracy gains are produced under hyperparameters that appear to be selected on the same 15 datasets used for evaluation: LoRA scale γ=12 for the OOD benchmark and γ=2 for the fine-grained benchmark, rank 16, layers 11–12, QKV+O matrices, mask ratio 0.5, cutoff 0.1, and λ2=1/16. No standard deviations, multiple seeds, or a validation split are reported, and Fig. S5a shows that the fine-grained average ranges from about 60 to 65 across rank/scale choices. Without a clearly separated hyperparameter-selection protocol, released code, or seed-level variance, a reader cannot distinguish a robust phenomenon from a favorable configuration.
  2. [§4.2, Table S4, Contribution bullet 2] The paper's claim that the combined loss "demonstrates excellent calibration performance" is contradicted by Table S4: LoRA-TTT has ECE 14.04 on ImageNet, 16.49 on the OOD average, and 12.75 on the fine-grained average, which are far worse than CLIP-ViT-B/16 (1.93, 4.80, 4.53) and C-TPT (3.11, 5.38, 5.29). Only LoRA-TTT-A is well calibrated. The text should restrict the calibration claim to LoRA-TTT-A and report ECE for the combined method honestly.
  3. [§3.2, Eq. (4), Table S6] The load-bearing premise is that minimizing L_MAE = MSE(g(X)_cls, g(mask(X))_cls) on one test image and its augmented views produces gradients useful for classification. Table S6 shows that reconstructing the class token outperforms visual tokens or pixels, but it does not show that the L_MAE gradient is correlated with class information; this matters because Table S2 shows per-dataset inconsistencies where LoRA-TTT-M or LoRA-TTT-A individually beat the combined model (e.g., DTD: 46.04 vs 45.86; EuroSAT: 44.52 vs 43.23), and the gap between LoRA-TTT-A and LoRA-TTT on the fine-grained average is only 0.28. An analysis of gradient alignment with class information, or an experiment with a non-informative reconstruction target, would strengthen the claim.
minor comments (5)
  1. [§3.1, Eq. (1)] The summation in the denominator uses i both as the free index and as the class index; it should be Σ_{j=1}^K.
  2. [§3.1] "trainablep" is missing a space and should read "trainable p".
  3. [Table S3] Entries such as "LoRA-TTT+ Ensemble(Ours)70.67" need a space and consistent formatting.
  4. [§4.2] "as inLoRA-TTT" should read "as in LoRA-TTT".
  5. [§5.3] The initialization study uses CC3M, an external dataset; the paper's claims of "no external resources" should be scoped to the standard zero-shot setup, with a clarification that the main benchmark numbers use Kaiming-initialized LoRA.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: LoRA-TTT's losses are self-supervised targets computed from the test image, and the reported gains are external benchmark accuracies not defined by the method's inputs.

full rationale

The paper's central claim is empirical: updating LoRA adapters in CLIP's image encoder with marginal-entropy minimization (Eq. 2) and class-token masked reconstruction (Eq. 4) improves zero-shot accuracy on external benchmarks. Neither loss is defined in terms of the benchmark accuracy or the reported evaluation metric. L_MAE is a self-supervised consistency target computed from the test image itself, MSE(g(X)_cls, g(mask(X))_cls), not from labels or dataset-level statistics, so the reported gains are not forced by construction. The loss weights, LoRA scale (12 for OOD, 2 for fine-grained), rank, layers, and mask ratio are hyperparameters selected on the benchmarks; this is a selection-bias or soundness concern, not definitional circularity, because the reported zero-shot top-1 accuracy is not a fitted parameter equal to the input. The only author-overlapping citation is [57] (Xiaolong Wang) supporting MAE-based TTT; it is background motivation and is accompanied by non-self citations [15, 36], and no uniqueness theorem is imported from it. No equation in the paper reduces to its own inputs, so there is no significant circularity.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim is empirical and rests on several untested domain assumptions about test-time objectives and on hyperparameters selected via ablations on the same benchmarks. No new physical, mathematical, or architectural entities are introduced; LoRA and masked autoencoding are existing tools.

free parameters (7)
  • LoRA scale gamma = 12 for OOD, 2 for fine-grained
    Controls the contribution of LoRA; selected separately for each benchmark via grid search in Figure S5a, directly affecting the reported gains.
  • Loss weight lambda2 = Not clearly reported; text prints 16, possibly intended as 1/6
    Balances MEM and MAE losses; set by hand with no sensitivity analysis shown.
  • LoRA rank r = 16
    Chosen via ablation in Figure S5a-b; not derived from theory.
  • LoRA layer range = Layers 11 and 12
    Selected via ablation in Table S5; deeper layers work better, but all layers perform worse.
  • LoRA attention matrices = Wk, Wq, Wv, Wo combined
    Selected via ablation in Figure S5b; Wv alone is best, but the paper uses all four.
  • Confidence selection cutoff = Top 10% of 64 augmented views
    Chosen from cutoff values 0.1, 0.5, and 1 in Table S6; inherited from TPT but re-tuned on the benchmarks.
  • Masking ratio = 50%
    Chosen from 25%, 50%, and 75% in Table S6 as a balance between performance and efficiency.
assumptions (5)
  • domain assumption Entropy minimization on high-confidence augmented views of a single test image provides a useful label-free adaptation signal for CLIP zero-shot classification.
    Adopted from TPT and MEM and used in Equation 2 and Section 3.2. If this signal is weak, LoRA-TTT-M gains would not transfer.
  • domain assumption Minimizing MSE between class tokens of masked and unmasked augmented views improves the image encoder's representation for downstream classification without labels.
    This is the core of the proposed MAE-style loss in Equation 4. The paper offers no formal justification; only the empirical ablation in Table S6 supports it.
  • domain assumption A single AdamW step on LoRA parameters in layers 11 and 12 suffices to adapt while preserving CLIP's zero-shot generalization.
    Core design choice supported only by experiments, not by analysis of optimization or forgetting.
  • domain assumption Precomputed text embeddings remain a valid classifier after the image encoder is adapted.
    LoRA-TTT freezes the text encoder and uses fixed prompts; the method relies on the shared CLIP embedding space remaining aligned after image-side updates.
  • domain assumption Random resized crops of one test image form a pseudo-batch whose top-confidence views estimate the test instance's marginal entropy reliably.
    Inherited from TPT/MEM and used to compute both losses; no evidence is given that 64 crops from one image are a stable distribution estimate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Models." pith.science (2026). https://pith.science/paper/N3FM7D3N

@misc{pith2026250202069,
  author       = {Pith},
  title        = {Pith review of: LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N3FM7D3N}},
  note         = {Machine review of arXiv:2502.02069}
}
read the original abstract

The rapid advancements in vision-language models (VLMs), such as CLIP, have intensified the need to address distribution shifts between training and testing datasets. Although prior Test-Time Training (TTT) techniques for VLMs have demonstrated robust performance, they predominantly rely on tuning text prompts, a process that demands substantial computational resources and is heavily dependent on entropy-based loss. In this paper, we propose LoRA-TTT, a novel TTT method that leverages Low-Rank Adaptation (LoRA), applied exclusively to the image encoder of VLMs. By introducing LoRA and updating only its parameters during test time, our method offers a simple yet effective TTT approach, retaining the model's initial generalization capability while achieving substantial performance gains with minimal memory and runtime overhead. Additionally, we introduce a highly efficient reconstruction loss tailored for TTT. Our method can adapt to diverse domains by combining these two losses, without increasing memory consumption or runtime. Extensive experiments on two benchmarks, covering 15 datasets, demonstrate that our method improves the zero-shot top-1 accuracy of CLIP-ViT-B/16 by an average of 5.79% on the OOD benchmark and 1.36% on the fine-grained benchmark, efficiently surpassing test-time prompt tuning, without relying on any external models or cache.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 28 canonical work pages

  1. [1]

    In- trinsic dimensionality explains the effectiveness of language model fine-tuning

    Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. In- trinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255, 2020. 3

    arXiv 2012

  2. [2]

    Self-supervised 8 test-time adaptation on video data

    Fatemeh Azimi, Sebastian Palacio, Federico Raue, J ¨orn Hees, Luca Bertinetto, and Andreas Dengel. Self-supervised 8 test-time adaptation on video data. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3439–3448, 2022. 2, 7

    work page 2022

  3. [3]

    Food-101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part VI 13, pages 446–461. Springer, 2014. 5

    2014

  4. [4]

    Parameter-free online test-time adaptation

    Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 8344–8353, 2022. 1

    work page 2022

  5. [5]

    Lasp: Text-to- text optimization for language-aware soft prompting of vi- sion & language models

    Adrian Bulat and Georgios Tzimiropoulos. Lasp: Text-to- text optimization for language-aware soft prompting of vi- sion & language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23232–23241, 2023. 1

    work page 2023

  6. [6]

    Tinytl: Reduce memory, not parameters for efficient on-device learning

    Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Sys- tems, 33:11285–11297, 2020. 2, 7

    work page 2020

  7. [7]

    Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558–3568, 2021. 1

    2021

  8. [8]

    Contrastive test-time adaptation

    Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 295–305, 2022. 1, 2

    work page 2022

Show all 77 references
  1. [9]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014. 4

    2014

  2. [10]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 4

    2009

  3. [11]

    Parameter-efficient fine-tuning of large-scale pre-trained language models.Nature Machine In- telligence, 5(3):220–235, 2023

    Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models.Nature Machine In- telligence, 5(3):220–235, 2023. 2

    2023

  4. [12]

    Clip- nav: Using clip for zero-shot vision-and-language naviga- tion

    Vishnu Sashank Dorbala, Gunnar Sigurdsson, Robinson Pi- ramuthu, Jesse Thomason, and Gaurav S Sukhatme. Clip- nav: Using clip for zero-shot vision-and-language naviga- tion. arXiv preprint arXiv:2211.16649, 2022. 2, 6, 7

    2022 arXiv

  5. [13]

    Frustratingly easy test- time adaptation of vision-language models

    Matteo Farina, Gianni Franchi, Giovanni Iacca, Massim- iliano Mancini, and Elisa Ricci. Frustratingly easy test- time adaptation of vision-language models. arXiv preprint arXiv:2405.18330, 2024. 3

    2024 arXiv

  6. [14]

    Diverse data augmentation with diffusions for effective test-time prompt tuning

    Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2704–2714, 2023. 2, 4

    2023

  7. [15]

    Test-time training with masked autoencoders

    Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. Advances in Neural Information Processing Systems , 35:29374–29385,

  8. [16]

    Mimic before reconstruct: Enhancing masked autoencoders with feature mimicking

    Peng Gao, Ziyi Lin, Renrui Zhang, Rongyao Fang, Hongyang Li, Hongsheng Li, and Yu Qiao. Mimic before reconstruct: Enhancing masked autoencoders with feature mimicking. International Journal of Computer Vision , 132 (5):1546–1556, 2024. 8

    2024

  9. [17]

    A systematic survey of prompt engineer- ing on vision-language foundation models

    Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, V olker Tresp, and Philip Torr. A systematic survey of prompt engineer- ing on vision-language foundation models. arXiv preprint arXiv:2307.12980, 2023. 1, 6

    2023 arXiv

  10. [18]

    On calibration of modern neural networks

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR,

  11. [19]

    Parameter-efficient fine-tuning for large models: A compre- hensive survey

    Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficient fine-tuning for large models: A compre- hensive survey. arXiv preprint arXiv:2403.14608, 2024. 2, 3

    2024 arXiv

  12. [20]

    Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026–1034, 2015. 5

    2015

  13. [21]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000– 16009, 2022. 4

    2022

  14. [22]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. 5

    2019

  15. [23]

    The many faces of robust- ness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robust- ness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF internation...

    2021

  16. [24]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Stein- hardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 15262–15271, 2021. 4

    2021

  17. [25]

    Masked image mod- eling: A survey

    Vlad Hondru, Florinel Alin Croitoru, Shervin Minaee, Radu Tudor Ionescu, and Nicu Sebe. Masked image mod- eling: A survey. arXiv preprint arXiv:2408.06687, 2024. 8

    2024 arXiv

  18. [26]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 2, 3

    2021 arXiv

  19. [27]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR,

  20. [28]

    Efficient test-time adaptation of vision-language models

    Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14162–14171, 2024. 2, 4

    2024

  21. [29]

    Simple but effective: Clip embed- dings for embodied ai

    Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: Clip embed- dings for embodied ai. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 14829–14838, 2022. 2, 6, 7

    2022

  22. [30]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on com- puter vision workshops, pages 554–561, 2013. 5

    2013

  23. [31]

    Fine-tuning can distort pretrained fea- tures and underperform out-of-distribution

    Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained fea- tures and underperform out-of-distribution. arXiv preprint arXiv:2202.10054, 2022. 2, 3

    2022 arXiv

  24. [32]

    Language-driven semantic seg- mentation

    Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Ren ´e Ranftl. Language-driven semantic seg- mentation. arXiv preprint arXiv:2201.03546, 2022. 3

    2022 arXiv

  25. [33]

    Caltech 101

    Fei-Fei Li, Marco Andreeto, M Ranzato, and Pietro Perona. Caltech 101. CaltechDATA: Pasadena, CA, USA, 2022. 5

    2022

  26. [34]

    A comprehensive sur- vey on test-time adaptation under distribution shifts

    Jian Liang, Ran He, and Tieniu Tan. A comprehensive sur- vey on test-time adaptation under distribution shifts. Inter- national Journal of Computer Vision, pages 1–34, 2024. 2

    2024

  27. [35]

    Clip-driven universal model for organ segmentation and tumor detection

    Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. Clip-driven universal model for organ segmentation and tumor detection. In Proceedings of the IEEE/CVF International Conference on Compute...

    2023

  28. [36]

    Continual-mae: Adaptive distribution masked autoencoders for continual test-time adaptation

    Jiaming Liu, Ran Xu, Senqiao Yang, Renrui Zhang, Qizhe Zhang, Zehui Chen, Yandong Guo, and Shanghang Zhang. Continual-mae: Adaptive distribution masked autoencoders for continual test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2024

  29. [37]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 5

    2017 arXiv

  30. [38]

    Fine-grained visual classi- fication of aircraft

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classi- fication of aircraft. arXiv preprint arXiv:1306.5151 , 2013. 5

    2013 arXiv

  31. [39]

    Autoclip: Auto-tuning zero-shot classifiers for vision-language models

    Jan Hendrik Metzen, Piyapat Saranrittichai, and Chaithanya Kumar Mummadi. Autoclip: Auto-tuning zero-shot classifiers for vision-language models. arXiv preprint arXiv:2309.16414, 2023. 1

    2023 arXiv

  32. [40]

    Simple open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. In European Conference on Computer Vision, pages 728–755. S...

    2022

  33. [41]

    Obtaining well calibrated probabilities using bayesian binning

    Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, 2015. 6

    2015

  34. [42]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & im- age processing, pages 722–729. IEEE, 2008. 4

    2008

  35. [43]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012. 4

    2012

  36. [44]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  37. [45]

    Do imagenet classifiers generalize to im- agenet? In International conference on machine learning , pages 5389–5400

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to im- agenet? In International conference on machine learning , pages 5389–5400. PMLR, 2019. 4

    2019

  38. [46]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2018

  39. [47]

    Test- time prompt tuning for zero-shot generalization in vision- language models

    Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test- time prompt tuning for zero-shot generalization in vision- language models. Advances in Neural Information Process- ing Systems, 35:14274–14289, 2022. 1, 2, 3, 4, 5, 6

    2022

  40. [48]

    Clipood: Generalizing clip to out-of-distributions

    Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jian- min Wang, and Mingsheng Long. Clipood: Generalizing clip to out-of-distributions. In International Conference on Machine Learning, pages 31716–31731. PMLR, 2023. 1

    2023

  41. [49]

    Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization

    Junha Song, Jungsoo Lee, In So Kweon, and Sungha Choi. Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11920–11929, 2023. 2, 7

    2023

  42. [50]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 5

    2012 arXiv

  43. [51]

    Just shift it: Test-time prototype shifting for zero-shot gen- eralization with vision-language models

    Elaine Sui, Xiaohan Wang, and Serena Yeung-Levy. Just shift it: Test-time prototype shifting for zero-shot gen- eralization with vision-language models. arXiv preprint arXiv:2403.12952, 2024. 1

    2024 arXiv

  44. [52]

    Test-time training with self- supervision for generalization under distribution shifts

    Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self- supervision for generalization under distribution shifts. In International conference on machine learning, pages 9229–

  45. [53]

    Yfcc100m: The new data in multimedia research

    Bart Thomee, David A Shamma, Gerald Friedland, Ben- jamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016. 1

    2016

  46. [54]

    Tent: Fully test-time adaptation by entropy minimization

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Ol- shausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726,

    2006 arXiv

  47. [55]

    Learning robust global representations by penalizing local predictive power.Advances in Neural Information Pro- cessing Systems, 32, 2019

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power.Advances in Neural Information Pro- cessing Systems, 32, 2019. 4

    2019

  48. [56]

    Actionclip: A new paradigm for video action recognition

    Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472, 2021. 3

    2021 arXiv

  49. [57]

    Test-time training on video streams

    Renhao Wang, Yu Sun, Yossi Gandelsman, Xinlei Chen, Alexei A Efros, and Xiaolong Wang. Test-time training on video streams. arXiv preprint arXiv:2307.05014, 2023. 2, 4, 7, 8

    2023 arXiv

  50. [58]

    Medclip: Contrastive learning from unpaired medical images and text

    Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. arXiv preprint arXiv:2210.10163, 2022. 2, 6, 7

    2022 arXiv

  51. [59]

    In search of lost online test-time adap- tation: A survey

    Zixin Wang, Yadan Luo, Liang Zheng, Zhuoxiao Chen, Sen Wang, and Zi Huang. In search of lost online test-time adap- tation: A survey. International Journal of Computer Vision, pages 1–34, 2024. 2

    2024

  52. [60]

    Robust fine-tuning of zero-shot models

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gon- tijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vi- ...

    2022

  53. [61]

    Sun database: Large-scale scene recognition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer so- ciety conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010. 4

    2010

  54. [62]

    Any-shift prompt- ing for generalization over distributions

    Zehao Xiao, Jiayi Shen, Mohammad Mahdi Derakhshani, Shengcai Liao, and Cees GM Snoek. Any-shift prompt- ing for generalization over distributions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13849–13860, 2024. 1

    2024

  55. [63]

    Parameter-efficient fine-tuning for pre-trained vision models: A survey

    Yi Xin, Siqi Luo, Haodi Zhou, Junlong Du, Xiaohong Liu, Yue Fan, Qing Li, and Yuntao Du. Parameter-efficient fine-tuning for pre-trained vision models: A survey. arXiv preprint arXiv:2402.02242, 2024. 3

    2024

  56. [64]

    Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assess- ment

    Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assess- ment. arXiv preprint arXiv:2312.12148, 2023. 2

    2023 arXiv

  57. [65]

    Filip: Fine-grained interactive language-image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783, 2021. 3

    2021 arXiv

  58. [66]

    C-tpt: Calibrated test-time prompt tuning for vision- language models via text feature dispersion

    Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark Hasegawa-Johnson, Yingzhen Li, and Chang D Yoo. C-tpt: Calibrated test-time prompt tuning for vision- language models via text feature dispersion. arXiv preprint arXiv:2403.14119, 2024. 1, 2, 5, 6

    2024 arXiv

  59. [67]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022. 3

    2022 arXiv

  60. [68]

    Low-rank few-shot adaptation of vision-language models

    Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1593–1603, 2024. 3, 8

    2024

  61. [69]

    Maxime Zanella and Ismail Ben Ayed. On the test-time zero- shot generalization of vision-language models: Do we really need prompt learning? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23783–23793, 2024. 2, 5, 6

    2024

  62. [70]

    Vision-language models for vision tasks: A survey

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence,

  63. [71]

    Memo: Test time robustness via adaptation and augmentation

    Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Ad- vances in neural information processing systems, 35:38629– 38642, 2022. 2, 3

    2022

  64. [72]

    Adalora: Adaptive budget alloca- tion for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget alloca- tion for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023. 7, 8

    2023 arXiv

  65. [73]

    Test-time adaptation with clip reward for zero-shot gen- eralization in vision-language models

    Shuai Zhao, Xiaohan Wang, Linchao Zhu, and Yi Yang. Test-time adaptation with clip reward for zero-shot gen- eralization in vision-language models. arXiv preprint arXiv:2305.18010, 2023. 2, 4

    2023 arXiv

  66. [74]

    Conditional prompt learning for vision-language mod- els

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language mod- els. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 16816–16825,

  67. [75]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. In- ternational Journal of Computer Vision, 130(9):2337–2348,

  68. [76]

    Prompt-aligned gradient for prompt tuning

    Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Han- wang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15659–15669, 2023. 1

    2023

  69. [77]

    Melo: Low-rank adaptation is better than fine-tuning for medical image diagnosis

    Yitao Zhu, Zhenrong Shen, Zihao Zhao, Sheng Wang, Xin Wang, Xiangyu Zhao, Dinggang Shen, and Qian Wang. Melo: Low-rank adaptation is better than fine-tuning for medical image diagnosis. In 2024 IEEE International Sym- posium on Biomedical Imaging (ISBI) , pages 1–5. IEEE,

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.