Pith. sign in

REVIEW 10 cited by

Robust fine-tuning of zero-shot models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2109.01903 v3 pith:MCI3KDH3 submitted 2021-09-04 cs.CV cs.LG

Robust fine-tuning of zero-shot models

Mitchell Wortsman , Gabriel Ilharco , Jong Wook Kim , Mike Li , Simon Kornblith , Rebecca Roelofs , Raphael Gontijo-Lopes , Hannaneh Hajishirzi
show 3 more authors
Ali Farhadi Hongseok Namkoong Ludwig Schmidt
This is my paper
classification cs.CV cs.LG
keywords accuracydistributionfine-tuningwise-ftlargemodelsrobustnessshifts
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Large pre-trained models such as CLIP or ALIGN offer consistent accuracy across a range of data distributions when performing zero-shot inference (i.e., without fine-tuning on a specific dataset). Although existing fine-tuning methods substantially improve accuracy on a given target distribution, they often reduce robustness to distribution shifts. We address this tension by introducing a simple and effective method for improving robustness while fine-tuning: ensembling the weights of the zero-shot and fine-tuned models (WiSE-FT). Compared to standard fine-tuning, WiSE-FT provides large accuracy improvements under distribution shift, while preserving high accuracy on the target distribution. On ImageNet and five derived distribution shifts, WiSE-FT improves accuracy under distribution shift by 4 to 6 percentage points (pp) over prior work while increasing ImageNet accuracy by 1.6 pp. WiSE-FT achieves similarly large robustness gains (2 to 23 pp) on a diverse set of six further distribution shifts, and accuracy gains of 0.8 to 3.3 pp compared to standard fine-tuning on seven commonly used transfer learning datasets. These improvements come at no additional computational cost during fine-tuning or inference.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Editing Models with Task Arithmetic

    cs.LG 2022-12 accept novelty 8.0

    Task vectors from weight differences allow arithmetic operations to edit pre-trained models, improving multiple tasks simultaneously and enabling analogical inference on unseen tasks.

  2. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

    cs.CL 2022-02 accept novelty 8.0

    Randomly replacing labels in in-context demonstrations barely hurts performance, showing that label space, input distribution, and sequence format drive in-context learning more than ground-truth labels.

  3. LAION-5B: An open large-scale dataset for training next generation image-text models

    cs.CV 2022-10 accept novelty 7.0

    LAION-5B is an openly released dataset of 5.85 billion CLIP-filtered image-text pairs that enables replication of foundational vision-language models.

  4. Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

    cs.CV 2022-04 unverdicted novelty 7.0

    Socratic Models compose zero-shot multimodal reasoning by prompting pretrained language and vision models to exchange information and enable new capabilities without finetuning.

  5. Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining

    cs.LG 2026-05 unverdicted novelty 6.0

    DG-Hard uses Donoho-Gavish hard thresholding on the fine-tuning weight delta to separate task-aligned signal from noise-like residual, recovering damaged capabilities while preserving target-task gains.

  6. AlphaWiSE: Adaptive Weight Interpolation for Continual Multimodal Representation Learning

    cs.CV 2026-07 conditional novelty 5.0

    Fitting one interpolation coefficient per parameter tensor on a small exemplar memory improves continual audio–image–text retrieval over individual continual-learning checkpoints.

  7. DeltaMerge-LowRes: Composing Language and Task Deltas for Low-Resource Adaptation

    cs.CL 2026-07 conditional novelty 5.0

    A TIES-style merge across language-delta and task-delta axes (cross-axis TIES) beats additive and task-only composition on low-resource summarisation and QA.

  8. Generalization Boundaries of Fine-Tuned Small Language Models for Graph Structural Inference

    cs.LG 2026-04 unverdicted novelty 5.0

    Fine-tuned small language models (3-4B parameters) preserve ordinal consistency in ranking graph structural properties for graphs larger than training data and from held-out families, showing architecture-specific deg...

  9. NTIRE 2026 Challenge on Robust AI-Generated Image Detection in the Wild

    cs.CV 2026-04 unverdicted novelty 4.0

    The NTIRE 2026 challenge provides a dataset of over 294,000 real and AI-generated images with 36 transformations to benchmark robust detection models.

  10. Matched-Learning-Rate Analysis of Attention Drift and Transfer Retention in Fine-Tuned CLIP

    cs.LG 2026-04 unverdicted novelty 4.0

    Matched learning-rate experiments show LoRA retains substantially higher zero-shot transfer (45% vs 11% on EuroSAT, 58% vs 9% on Pets) than Full FT in CLIP adaptation.