Pith. sign in

REVIEW 3 major objections 5 minor 58 references

StarFT: Robust Fine-tuning of Zero-shot Models via Spuriosity Alignment

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read StarFT claims that aligning fine-tuned CLIP's masked-logit distribution over LM-generated spurious captions to the frozen zero-shot model prevents spurious feature learning and preserves robustness.

desk verdict A practically effective robust fine-tuning regularizer whose headline mechanism—spurious semantics driving the gains—is weakly supported, since the random-suffix gap is tiny and no error bars or Lipsum-FT comparison are provided. read the letter →

arxiv 2505.13232 v3 pith:HKIVOLZD submitted 2025-05-19 cs.AI cs.CV

Younghyun Kim , Jongheon Jeong , Sangkyung Kwak , Kyungmin Lee , Juho Lee , Jinwoo Shin This is my paper · ORCID
classification cs.AIcs.CV
keywords robustfine-tuningzero-shotmodelsCLIPspuriouscorrelationshortcutlearningtextualalignmentregularizationgrouprobustnessdistributionshift
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

StarFT claims that the drop in out-of-distribution robustness seen when fine-tuning zero-shot models such as CLIP can be traced to spurious feature learning, and that this can be countered by a single textual regularizer. The regularizer builds captions that append language-model-generated spurious descriptors (for example, "in the mountains" or "with a rough texture") to the class prompt, then forces the fine-tuned model's masked-logit distribution over those captions to match the frozen zero-shot model's distribution. If the claim is right, robustness is not a property that must be preserved wholesale; it can be protected by telling the model which confounds not to rely on. On Waterbirds, StarFT improves worst-group accuracy by 14.30 percentage points and average accuracy by 3.02 points over the FLYP baseline, where competing robust fine-tuning methods degrade.

What carries the argument

The load-bearing object is the spurious textual alignment loss $\mathcal{L}_{\mathrm{Star}}$, computed on a batch of spuriosity-augmented captions $S_i$ formed by appending LM-generated descriptors to each class template. For each image, the logits of both the fine-tuned and frozen zero-shot models are masked to remove the true-class column, so the softmax $q_i$ (fine-tuned) and $\tilde{q}_i$ (zero-shot) encode only relative likelihoods among spurious descriptions; the loss is $\frac{1}{N}\sum_i D_{\mathrm{KL}}(\tilde{q}_i \| q_i)$. This masked-KL form is what prevents the fine-tuned model from learning to lean on the spurious descriptions, while the true-class masking keeps an imperfect zero-shot teacher from injecting erroneous confidence.

What would settle it

Take a dataset whose known spurious cue is deliberately absent from the LM descriptor bank, for example Waterbirds with only texture and resolution descriptors and no background descriptors, and fine-tune with StarFT; if worst-group accuracy on the water/land subgroups still rises substantially, the background descriptors are not the active mechanism, and if it does not rise, the mechanism is descriptor coverage rather than generic regularization.

Watch

Extended reading notes

Core claim

The paper's central claim is that minimizing the spurious textual alignment loss $\mathcal{L}_{\mathrm{Star}}$ (Eq. 3), a KL divergence between the fine-tuned and frozen zero-shot softmax distributions over spuriosity-augmented captions with true-class logits masked, prevents the fine-tuned model from further extracting the spurious features named by those captions. Masking the true class avoids distilling the zero-shot model's confident but wrong positive-pair predictions when it is weak; decaying the regularization weight balances in-distribution accuracy against out-of-distribution robustness. The authors argue that this mechanism, rather than prompt or context preservation, explains why StarFT improves OOD averages on ImageNet variants, worst-group accuracy on Waterbirds, PACS, and CIFAR-10.02, zero-shot accuracy on four natural-image benchmarks, and transfer average rank, all without extra data or task-specific prompts.

Load-bearing premise

The method only suppresses the spurious cues written in the language-model-generated descriptor bank, so the bank must actually cover the confounds a dataset exploits; if it misses the real bias or overlaps with features that genuinely define the class, the promised robustness gain will not appear or in-distribution learning will suffer.

Editorial extensions

If this is right

  • If the central claim holds, robust fine-tuning can be achieved by specifying what not to learn, rather than only by preserving the pretrained model's weights or contexts.
  • A single generic descriptor bank transfers across benchmarks: fine-tuning on ImageNet with StarFT gives zero-shot worst-group gains on Waterbirds, PACS, and CIFAR-10.02 without those datasets appearing in training.
  • StarFT improves worst-group and average accuracy together, which goes against the usual expectation that removing spurious reliance must sacrifice average performance.
  • Because the frozen zero-shot model acts as teacher, StarFT does not inherit the EMA-based failure mode of CaRot on datasets where the zero-shot model is weak, such as WILDS-iWILDCam.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If descriptor coverage drives the gains, an oracle bank that names the exact dataset confound should make StarFT nearly solve the group-shift benchmark, while a bank that omits it should erase the improvement; this comparison would isolate the mechanism from generic regularization.
  • The masked-logit distribution could double as an audit tool: tracking per-descriptor KL movement during fine-tuning would reveal which confounds the model is actually learning to suppress.
  • For datasets whose confounds are not generic natural-image cues (medical imaging, satellite imagery), the method would need automatic per-task spuriosity discovery; the paper leaves that extension open.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes StarFT, an auxiliary regularizer for contrastive fine-tuning of CLIP. Spurious descriptors are generated by querying a language model, appended to class captions, and the regularizer computes a KL divergence between the fine-tuned model and the frozen zero-shot model over spuriosity-augmented captions with true-class logits masked. Experiments on ImageNet OOD, group-shift, zero-shot transfer, and WILDS benchmarks report consistent improvements over FLYP, CAR-FT, and CaRot, with the largest gain being a 14.30% worst-group improvement on Waterbirds. Ablations attribute the gain to adding spurious suffixes, masking positive pairs, and decaying the regularization weight.

Significance. If the mechanism is as claimed, the paper would show that LLM-generated spurious text is a generic and cheap robustness regularizer, and it would provide a practical method that improves OOD, group-shift, zero-shot, and transfer performance simultaneously. The method is clearly specified, the code is released, and the evaluation spans many benchmarks. The authors also include a random-suffix ablation and a positive-pair-masking ablation, which are the right controls. However, the central mechanistic claim rests on a single ablation whose effect size is comparable to run-to-run noise, and the paper reports no error bars or multiple seeds; these gaps need to be closed before the significance can be assessed.

major comments (3)
  1. [Section 4.5, Table 6] The only experiment that isolates the semantic content of the spurious descriptor bank is the comparison between the 'Random' and 'Spurious' suffix rows, which differ by 0.4% on ImageNet OOD average (63.8 vs 64.2) and by 0.4% on ImageNet (82.3 vs 82.7), on single runs with no error bars. Since the jump from clean-caption regularization to any suffix is 0.6% (63.2 to 63.8), the spurious-vs-random difference is too small to support the central claim that spuriosity alignment, rather than generic KL-to-zero-shot regularization over augmented captions, drives the reported gains. Please add multiple seeds with confidence intervals, and run Lipsum-FT (cited in Section 2 but never evaluated) as a direct baseline; if the gap is within noise, the mechanistic claim needs to be revised.
  2. [Section 3.2, Eq. (3)] L_Star is a KL divergence between the fine-tuned model and the frozen zero-shot model over masked non-class logits, so the only place where spuriosity enters is the choice of the caption set S. The zero-shot model itself exhibits strong spurious bias (Table 3: Waterbirds zero-shot WG 25.9 vs Avg 87.1); aligning to its distribution is therefore not by itself a penalty on spurious features. To support the proposed mechanism, please provide a direct test that the regularizer specifically suppresses use of the spurious descriptors, e.g., an ablation with semantically plausible but non-spurious suffixes, or a quantitative measure of reliance on background versus foreground features.
  3. [Section 4.5 and Appendix C.2] The default descriptor bank contains only the 'background' concept (Table 7), yet StarFT improves performance on style/texture shifts (ImageNet-Sketch, PACS) and data-source shift (CIFAR-10.02). The paper does not analyze whether the background bank covers the actual spurious cues in these benchmarks, so the observed benchmark pattern is at least as consistent with a content-agnostic regularizer as with a spuriousness-specific mechanism. Please add descriptor-coverage analysis or run a task-matched descriptor bank to show that matching the spurious cues matters.
minor comments (5)
  1. [Sections 3.1 and 3.2] Equation (1) is referred to as '(3.1)' in several places, e.g., 'the contrastive loss (3.1)' and 'Eq. (3.1)'; please fix the equation references.
  2. [Section 3.2 and Appendix C] Please specify how the spurious descriptor is sampled for each caption (uniformly from the bank? once per image per batch?) and report the number of descriptors per concept; Appendix C.2 lists examples but not the sampling distribution.
  3. [Appendix D] The statement that the method is 'inherently robust' to LM hallucination is supported only by an intuition; either add a targeted experiment with intentionally misleading descriptors or soften the claim.
  4. [Appendix A.2, Table 10] The lambda sweep reports ID/OOD values for three settings with no variance; since lambda=1.0 gives higher OOD than the default lambda=0.5, please state the explicit selection criterion for the default value.
  5. [Section 4.2] The sentence 'narrows the gap between the worst group and the average accuracy to 49.88%, which is 61.19% in zero-shot models' is confusing; the second number is the zero-shot gap (87.1 - 25.9), not a percentage of the model, and should be rephrased.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: StarFT's regularization is a self-distillation loss against the frozen zero-shot model, but the central robustness claims are evaluated on external OOD and group-shift benchmarks, and the spurious descriptor bank is generated without using target labels or fitted benchmark values.

full rationale

The derivation chain is not circular. L_Star (Eq. 3) is a KL-divergence training objective between the fine-tuned model's masked softmax distribution and the frozen zero-shot model's distribution over spuriosity-augmented captions; this is a regularizer used during training, not a prediction derived from its own inputs. The reported gains are measured on held-out external benchmarks (ImageNet-R/A/Sketch/V2, Waterbirds, PACS, CIFAR-10.02, and transfer datasets) that are not used to construct the spurious descriptors, and the descriptors themselves are obtained by prompting GPT-3.5 or Llama with generic instructions such as 'List possible spurious correlations while classifying natural images.' No equation in the paper reduces a claimed result to a fitted parameter or to a self-citation chain. The ablation in Table 6 shows only a 0.4% OOD-average difference between random and spurious suffixes with no error bars; this weakens the mechanistic attribution to spuriosity, but it is an experimental-control concern, not a definitional circularity. Regularizing toward the frozen zero-shot model plausibly explains some preservation of zero-shot behavior, yet the zero-shot classification results in Table 4 are external benchmark numbers rather than consequences forced by construction. The paper also cites prior work by overlapping authors (e.g., Kim et al. 2024 for spurious-correlation discovery and Lipsum-FT as a baseline), but these citations are not load-bearing: they support qualitative analysis or serve as baselines and correspond to independently published methods. The appended limitations section explicitly acknowledges the need for further study of spurious-concept construction and possible LLM hallucination, which further confirms that the paper does not hide a self-referential justification. Overall, I find no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

StarFT introduces no new physical entities; it is a regularization method. Its empirical results depend on a bank of text prompts generated by an external LM, on the choice of spurious concepts, and on two hyperparameters (lambda and its decay schedule). The evaluation benchmarks are external.

free parameters (4)
  • lambda_Star = 0.5 for ImageNet, 0.1 for other datasets, linearly decayed
    Regularization weight in Eq. (4); chosen by the authors and validated on ImageNet via a sweep in Appendix A.2; the exact decay schedule is not specified.
  • spurious concept set = background, texture, resolution
    The choice of which spurious concepts to query from the LM is a hand-picked set; Table 1 and Appendix C.2. This set defines the corruption space.
  • spurious descriptor bank = not fully disclosed (examples only)
    The specific list of textual suffices used in the experiments is only partly shown in Table 15, so the exact set is a hidden input that affects results.
  • prompt templates for evaluation = e.g., 'This is a picture of a [class]' for group shift
    Zero-shot evaluation prompts follow Zhang and Re 2022; choice of template can affect accuracy.
assumptions (4)
  • standard math The contrastive loss (Eq. 1) is the correct fine-tuning objective and is used exactly as in FLYP.
    The paper builds on the CLIP contrastive loss as the base objective.
  • domain assumption Aligning the fine-tuned model's distribution over spuriosity-augmented captions with the zero-shot model reduces spurious reliance.
    This is the core mechanism of L_Star; no proof is given, only empirical validation.
  • domain assumption LLM-generated spurious descriptors accurately reflect the features that cause shortcut learning in fine-tuned CLIP.
    The method hinges on the quality of the LM-generated descriptor bank; hallucinations or missing concepts would weaken the regularization.
  • ad hoc to paper Masking positive pairs in Eq. (2) prevents distillation of unreliable true-class confidence from a weak zero-shot model.
    The masking is introduced to handle cases where zero-shot model performs poorly (Section 4.5 and Appendix A.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of StarFT: Robust Fine-tuning of Zero-shot Models via Spuriosity Alignment." pith.science (2026). https://pith.science/paper/HKIVOLZD

@misc{pith2026250513232,
  author       = {Pith},
  title        = {Pith review of: StarFT: Robust Fine-tuning of Zero-shot Models via Spuriosity Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HKIVOLZD}},
  note         = {Machine review of arXiv:2505.13232}
}
read the original abstract

Learning robust representations from data often requires scale, which has led to the success of recent zero-shot models such as CLIP. However, the obtained robustness can easily be deteriorated when these models are fine-tuned on other downstream tasks (e.g., of smaller scales). Previous works often interpret this phenomenon in the context of domain shift, developing fine-tuning methods that aim to preserve the original domain as much as possible. However, in a different context, fine-tuned models with limited data are also prone to learning features that are spurious to humans, such as background or texture. In this paper, we propose StarFT (Spurious Textual Alignment Regularization), a novel framework for fine-tuning zero-shot models to enhance robustness by preventing them from learning spuriosity. We introduce a regularization that aligns the output distribution for spuriosity-injected labels with the original zero-shot model, ensuring that the model is not induced to extract irrelevant features further from these descriptions. We leverage recent language models to get such spuriosity-injected labels by generating alternative textual descriptions that highlight potentially confounding features. Extensive experiments validate the robust generalization of StarFT and its emerging properties: zero-shot group robustness and improved zero-shot classification. Notably, StarFT boosts both worst-group and average accuracy by 14.30% and 3.02%, respectively, in the Waterbirds group shift scenario, where other robust fine-tuning baselines show even degraded performance.

Figures

Figures reproduced from arXiv: 2505.13232 by the authors.

Figure 1
Figure 1. Overview of StarFT. Aside from the base contrastive objective Lc, we propose a novel spuriosity textual alignment regularization LStar. We first extract spurious textual descriptions from language model, and corrupt the label textual descriptions. We then prevent fine￾tuned models from learning spuriosity by minimizing the KL divergence of negative pairs’ corrupted textual descriptions. making, which results in impr… view at source ↗
Figure 2
Figure 2. Subgroup accuracies in group shift benchmarks. Dif￾ferences of subgroup accuracies (%) between zero-shot and FLYP fine-tuned models. schemes [Kumar et al., 2022, Goyal et al., 2023, Choi et al., 2024], and regularization schemes [Mao et al., 2022a, Nam et al., 2024, Oh et al., 2024] to better preserve the zero-shot model as prior knowledge. For instances, WiSE-FT [Worts￾man et al., 2022] considers a weight ensemblin… view at source ↗
Figure 3
Figure 3. Mitigation of spuriosity in ImageNet. We display the GradCAM [Selvaraju et al., 2017] of fine-tuned models for compar￾ison. Each class has the spurious correlations with background such as “snow” in “husky” and “sand” in “rattle snake.” Rather than fo￾cusing on mostly background like FLYP, StarFT focuses on object itself to make decisions. Choice of different spurious concept. By prompting lan￾guage models, we attai… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 46 canonical work pages

  1. [1]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, et al. Learning transferable visual models from natural language supervision. In ICML, 2021

    work page 2021

  2. [2]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, et al. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, 2021

    work page 2021

  3. [3]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975--11986, 2023

    2023

  4. [4]

    Measuring robustness to natural distribution shifts in image classification

    Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. In Neural Information Processing Systems, 2020

    work page 2020

  5. [5]

    Accuracy on the line: On the strong correlation between out-of-distribution and in-distribution generalization

    John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, et al. Accuracy on the line: On the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning, 2021

    work page 2021

  6. [6]

    Hudson, Ehsan Adeli, et al

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, et al. On the opportunities and risks of foundation models, 2022

    work page 2022

  7. [7]

    Robust fine-tuning of zero-shot models

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, et al. Robust fine-tuning of zero-shot models. In Conference on Computer Vision and Pattern Recognition, 2022

    work page 2022

  8. [8]

    Finetune like you pretrain: Improved finetuning of zero-shot vision models

    Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Conference on Computer Vision and Pattern Recognition, 2023

    work page 2023

Show all 58 references
  1. [9]

    Context-aware robust fine-tuning

    Xiaofeng Mao, Yuefeng Chen, Xiaojun Jia, Rong Zhang, Hui Xue, and Zhao Li. Context-aware robust fine-tuning. International Journal of Computer Vision, 2022 a

    2022

  2. [10]

    Towards accountable ai: Hybrid human-machine analyses for characterizing system failure

    Besmira Nushi, Ece Kamar, and Eric Horvitz. Towards accountable ai: Hybrid human-machine analyses for characterizing system failure. In AAAI Conference on Human Computation and Crowdsourcing, 2018

    2018

  3. [11]

    Shortcut learning in deep neural networks

    Robert Geirhos, J \"o rn-Henrik Jacobsen, Claudio Michaelis, et al. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2 0 (11): 0 665--673, 2020

    2020

  4. [12]

    Intriguing properties of generative classifiers

    Priyank Jaini, Kevin Clark, and Robert Geirhos. Intriguing properties of generative classifiers. In International Conference on Learning Representations, 2024

    2024

  5. [13]

    Wichmann and Robert Geirhos

    Felix A. Wichmann and Robert Geirhos. Are deep neural networks adequate behavioral models of human visual perception? Annual Review of Vision Science, pages 501--524, 2023

    2023

  6. [14]

    Noise or signal: The role of image backgrounds in object recognition

    Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. In International Conference on Learning Representations, 2021

    2021

  7. [15]

    Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness

    Robert Geirhos, Patricia Rubisch, Claudio Michaelis, et al. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019

    2019

  8. [16]

    Fixing the train-test resolution discrepancy

    Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. In Neural Information Processing Systems, 2019

    2019

  9. [17]

    A fine-grained analysis on distribution shift

    Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvise Rebuffi, Ira Ktena, Krishnamurthy Dj Dvijotham, and Ali Taylan Cemgil. A fine-grained analysis on distribution shift. In International Conference on Machine Learning, 2022

    2022

  10. [18]

    Antonio Torralba and Alexei A. Efros. Unbiased look at dataset bias. In Conference on Computer Vision and Pattern Recognition, 2011

    2011

  11. [19]

    Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, 2019

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, 2019

    2019

  12. [20]

    Cubuk, et al

    Dan Hendrycks, Norman Mu, Ekin D. Cubuk, et al. Augmix: A simple data processing method to improve robustness and uncertainty. In International Conference on Learning Representations, 2020

    2020

  13. [21]

    Evaluating machine accuracy on I mage N et

    Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang, Benjamin Recht, and Ludwig Schmidt. Evaluating machine accuracy on I mage N et. In International Conference on Machine Learning, 2020

    2020

  14. [22]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In International Conference on Computer Vision, 2021 a

    2021

  15. [23]

    Adversarial training and robustness for multiple perturbations

    Florian Tramèr and Dan Boneh. Adversarial training and robustness for multiple perturbations. In NeurIPS, 2019

    2019

  16. [24]

    Vision transformers are robust learners

    Sayak Paul and Pin-Yu Chen. Vision transformers are robust learners. In AAAI, 2021

    2021

  17. [25]

    Towards robust vision transformer

    Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer. In Neural Information Processing Systems, 2022 b

    2022

  18. [26]

    Tent: Fully test-time adaptation by entropy minimization

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021

    2021

  19. [27]

    The evolution of out-of-distribution robustness throughout fine-tuning, 2021

    Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs. The evolution of out-of-distribution robustness throughout fine-tuning, 2021

    2021

  20. [28]

    Fine-tuning can distort pretrained features and underperform out-of-distribution

    Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations, 2022

    2022

  21. [29]

    Explicit inductive bias for transfer learning with convolutional networks

    Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In ICML, 2018

    2018

  22. [30]

    Trainable projected gradient method for robust fine-tuning

    Junjiao Tian, Xiaoliang Dai, Chih-Yao Ma, Zecheng He, Yen-Cheng Liu, and Zsolt Kira. Trainable projected gradient method for robust fine-tuning. In Conference on Computer Vision and Pattern Recognition, 2023

    2023

  23. [31]

    Lipsum-ft: Robust fine-tuning of zero-shot models using random text guidance

    Giung Nam, Byeongho Heo, and Juho Lee. Lipsum-ft: Robust fine-tuning of zero-shot models using random text guidance. In International Conference on Learning Representations, 2024

    2024

  24. [32]

    Towards calibrated robust fine-tuning of vision-language models, 2024

    Changdae Oh, Hyesu Lim, Mijoo Kim, Jaegul Choo, Alexander Hauptmann, Zhi-Qi Cheng, and Kyungwoo Song. Towards calibrated robust fine-tuning of vision-language models, 2024

    2024

  25. [33]

    Autoft: Learning an objective for robust fine-tuning, 2024

    Caroline Choi, Yoonho Lee, Annie Chen, Allan Zhou, Aditi Raghunathan, and Chelsea Finn. Autoft: Learning an objective for robust fine-tuning, 2024

    2024

  26. [34]

    Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization

    Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In ICLR, 2020

    2020

  27. [35]

    Hospedales

    Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In ICCV, 2017

    2017

  28. [36]

    Visual classification via description from large language models

    Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In International Conference on Learning Representations, 2023

    2023

  29. [37]

    Nguyen, and Tsui-Wei Weng

    Tuomas Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In International Conference on Learning Representations, 2023

    2023

  30. [38]

    Zero-shot robustification of zero-shot models

    Dyah Adila, Changho Shin, Linrong Cai, and Frederic Sala. Zero-shot robustification of zero-shot models. In International Conference on Learning Representations, 2024

    2024

  31. [39]

    L AION -400m: Open dataset of clip-filtered 400 million image-text pairs, 2021

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, et al. L AION -400m: Open dataset of clip-filtered 400 million image-text pairs, 2021

    2021

  32. [40]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017

    2017

  33. [41]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015

    2015

  34. [42]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021 b

    2021

  35. [43]

    Xing, and Zachary C

    Haohan Wang, Songwei Ge, Eric P. Xing, and Zachary C. Lipton. Learning robust global representations by penalizing local predictive power. In Neural Information Processing Systems, 2019

    2019

  36. [44]

    Contrastive adapters for foundation model group robustness

    Michael Zhang and Christopher R \'e . Contrastive adapters for foundation model group robustness. In Neural Information Processing Systems, 2022

    2022

  37. [45]

    Learning multiple layers of features from tiny images, 2009

    Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009

    2009

  38. [46]

    Harder or different? a closer look at distribution shift in dataset reproduction

    Shangyun Lu, Bradley Nott, Aaron Olson, et al. Harder or different? a closer look at distribution shift in dataset reproduction. In ICML Workshop on Uncertainty and Robustness in Deep Learning, 2020

    2020

  39. [47]

    Caltech 101, 2022

    Fei-Fei Li, Marco Andreeto, Marc'Aurelio Ranzato, and Pietro Perona. Caltech 101, 2022

    2022

  40. [48]

    An analysis of single-layer networks in unsupervised feature learning

    Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Machine Learning, 2011

    2011

  41. [49]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In IEEE International Conference on Computer Vision Workshops, pages 554--561, 2013

    2013

  42. [50]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722--729. IEEE, 2008

    2008

  43. [51]

    The i WildCam 2020 competition dataset, 2020

    Sara Beery, Elijah Cole, and Arvi Gjoka. The i WildCam 2020 competition dataset, 2020

    2020

  44. [52]

    Wilds: A benchmark of in-the-wild distribution shifts

    Pang Wei Koh, Shiori Sagawa, Henrik Marklund, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, 2021

    2021

  45. [53]

    Functional map of the world

    Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Conference on Computer Vision and Pattern Recognition, 2018

    2018

  46. [54]

    Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra

    Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad- CAM : Visual explanations from deep networks via gradient-based localization. In ICCV, 2017

    2017

  47. [55]

    Discovering and mitigating visual biases through keyword explanation

    Younghyun Kim, Sangwoo Mo, Minkyu Kim, Kyungmin Lee, Jaeho Lee, and Jinwoo Shin. Discovering and mitigating visual biases through keyword explanation. In Conference on Computer Vision and Pattern Recognition, 2024

    2024

  48. [56]

    The Llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  49. [57]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

    2023

  50. [58]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.