Self-Soupervision: Cooking Model Soups without Labels

Anthony Fuller; Evan Shelhamer; James R. Green

REVIEW 5 major objections 5 minor 3 references

Self-Soupervision: Cooking Model Soups without Labels

T0 review · 5 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Model soups, which average the weights of several models trained from a shared starting point, can be made without labels by mixing ingredients that differ in their self-supervised objectives, data, and hyperparameters — and the resulting s

desk verdict The cross-algorithm LMC result is the real contribution; the +3.5/+7 headline numbers don't separate souping from continued SSL on shifted data because per-ingredient corruption accuracy is never reported. read the letter →

arxiv 2602.02890 v2 pith:7VANNMS4 submitted 2026-02-02 cs.LG

Anthony Fuller , James R. Green , Evan Shelhamer This is my paper

classification cs.LG

keywords modelsoupsself-supervisedlearningweightaveraginglinearmodeconnectivityrobustnessdistributionshifttransferlabel-freeadaptation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Model soups — averaging the weights of several models trained from a shared starting point — have always required supervised fine-tuning on labeled data. This paper shows the mixing step can survive self-supervised inter-training: ingredients trained with different SSL algorithms (MAE, MoCoV3, MMCR), different SSL hyperparameters, or on different unlabeled datasets can be averaged into a soup that beats every single ingredient. The biggest wins come from shift-aware cooking: inter-training ingredients on corrupted, unlabeled test data and then fine-tuning back on clean training data improves corrupted-data accuracy by +3.5% on ImageNet-C and +7% on LAION-C. The paper also introduces Self-Seasoning, a fully label-free way to pick the mixing weights by minimizing the entropy of nearest-neighbor predictions, which is competitive with using labels for the same job.

What carries the argument

The load-bearing object is the Self-Soup: an average over parameters of ingredients that differ in their self-supervised inter-training stage (defined in the paper's Eq. 2), optionally followed by supervised fine-tuning. The property that makes the average useful is linear mode connectivity (Eq. 1): for every interpolation weight, the accuracy of the interpolated model is at least the interpolation of the accuracies. The paper's new mixing operation is Self-Seasoning, which selects mixture coefficients without labels by gradient descent on the entropy of k-nearest-neighbour predictions over unlabeled data.

What would settle it

Run the paper's protocol with two SSL objectives known to produce very different representations (e.g., MAE vs. MoCoV3) but inter-train for 50 epochs instead of 5, then fine-tune and average; if the 50/50 soup fails to at least match the average of the two ingredient accuracies on ImageNet-Val, linear mode connectivity is broken and the central claim no longer generalizes.

Watch

Extended reading notes

Core claim

The paper's central claim is that linear mode connectivity — the property that the accuracy of the weight-average of two models never falls below the average of their individual accuracies — holds between ingredients that differ in their self-supervised training, not just in their supervised fine-tuning. After continued self-supervised training from a shared stock using different losses, followed by supervised fine-tuning on the target task, averaging the ingredient weights yields a soup that is more accurate and more robust than any single ingredient on 8 ImageNet test sets and across 21 transfer tasks. The paper further shows that inter-training ingredients on unlabeled data from the shift

Load-bearing premise

The load-bearing premise is that linear mode connectivity holds even when ingredients differ in their self-supervised objectives, hyperparameters, or data; the paper verifies this for three SSL algorithms and one additional stock, but every soup gain depends on it — if LMC fails for other SSL families, longer inter-training, or very different data, the central generalization collapses.

Editorial extensions

If this is right

Shift-aware Self-Soups beat supervised soups and even supervised test-time adaptation on corrupted data, and the gains remain when test-time adaptation is applied on top (35.72 → 37.50 on ImageNet-C).
Because inter-training needs no labels, soups can now be cooked on unlabeled data sources — a task's own unlabeled data, shifted data, or web-scale data — that were previously inaccessible to soup recipes.
Different SSL algorithm families (reconstruction, instance-contrastive, dimension-contrastive) can be mixed into one soup, so practitioners no longer need to commit to a single self-supervised method to benefit from weight averaging.
Self-Seasoning gives a fully unsupervised route to transfer: mixture coefficients can be chosen by entropy minimization on unlabeled data, with accuracy competitive to supervised coefficient selection on 4 of 21 tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If linear mode connectivity holds across a wider sweep of SSL families and longer inter-training schedules than the five epochs tested here, the same recipe could enable label-free soup adaptation for language, audio, and multimodal models — where unlabeled in-domain data is abundant — by treating every domain or shift as just another inter-training flavor.
The gains from inter-training on corrupted data suggest a cheap alternative to distribution alignment: rather than learning an explicit domain-invariant representation, one can simply fine-tune a few ingredients on unlabeled target-like data and average. A natural test is whether the benefit scales with the size and diversity of the unlabeled pool, and whether it holds for natural (non-synthetic)
Self-Seasoning is reminiscent of entropy-minimization test-time adaptation, and the paper's own numbers show the two are complementary; a practical extension would be to jointly optimize mixture coefficients and batch-norm statistics at deployment time.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: this paper's solid core is the observation that linear mode connectivity holds across different SSL algorithms. The flashier robustness claims are real but under-attributed — as written, the numbers can't tell you how much of the gain is mixing versus continued SSL on shifted data.

What's genuinely new: Fig. 2 is the first clean demonstration I've seen that ingredients trained with different SSL objectives (MAE, MoCoV3, MMCR) can be averaged after fine-tuning, with 52 mixtures showing the mix beats the corners on eight ImageNet test sets. That's a real empirical finding for the merging community. Self-Seasoning — picking mixture weights by minimizing kNN entropy, no labels — is simple, reproducible, and honestly evaluated: it loses to supervised seasoning on average but wins 4/21 VTAB tasks, which is a fair trade for being label-free. The paper also gets the baselines right: the "continued SSL + supervised soup" comparison is the correct equal-search-cost control, and the Franca-stock check (Fig. 4) shows the effect isn't an artifact of the MAE stock. Code and data links are there, and the appendix includes working PyTorch for the entropy objective.

Soft spots, in proportion. The big one is the gap the stress test flags: Table 2's +3.5% (IN-C) and +7% (LAION-C) compare Self-Soup inter-trained on the shifted test set against Self-Soup inter-trained on ImageNet-train. The per-ingredient accuracies on the corruption sets are never reported — only the best ingredient as selected by clean IN-Val. Since all four ingredients are MAE trained at different learning rates on the corrupted data, it's entirely possible one of them already carries most of the IN-C gain; the uniform soup at 35.72 might not beat that ingredient. The headline claim therefore conflates continued SSL on shifted data with souping. This isn't a demonstrated error — Fig. 2 suggests mixing does help on corruptions — but it's the load-bearing evidence for the most-cited number, and it's absent. That's a missing table, not a redesign. The rest is minor: all results are single-run with no error bars, which matters when deltas are 0.5–3.5 points; the TTA learning-rate sweep in Table 3 appears to be selected on the same corrupted split used for evaluation (small bias, not circularity); and the abstract names LeJEPA ingredients that never appear in the body — a version-control slip.

Who it's for: anyone working on model merging, soups, or robustness via self-supervised pre-training. It deserves a serious referee. The referee's ask should be concrete: per-ingredient corruption accuracy in Table 2, multi-seed or error bars on the headline numbers, and an abstract-body consistency pass. I'd take the cross-algorithm LMC result seriously even if the robustness attribution never gets fully cleaned up.

Referee Report

5 major / 5 minor

Summary. The paper introduces Self-Soupervision, a framework that extends model soups to self-supervised learning (SSL). Instead of fine-tuning a stock only with supervised labels, the authors inter-train multiple SSL ingredients (differing in SSL algorithm, hyperparameters, or data) and then fine-tune and mix them. The core empirical claims are: (i) linear mode connectivity holds across ingredients trained with different SSL objectives (MAE, MoCoV3, MMCR); (ii) mixing SSL ingredients yields soups more accurate than any single ingredient; (iii) inter-training on unlabeled shifted/test data (ImageNet-C, LAION-C) then fine-tuning on ImageNet training data boosts robustness by +3.5% on ImageNet-C and +7% on LAION-C; (iv) Self-Soups improve transfer on VTAB; (v) a new unsupervised mixing method, Self-Seasoning, can select mixture coefficients without labels. The paper is written in a playful 'cooking' style and includes code and data links.

Significance. If the results hold, this is a worthwhile contribution: it substantially broadens the applicability of model soups beyond supervised fine-tuning, shows a new instance of linear mode connectivity (across SSL objectives), and provides a practical recipe for leveraging unlabeled data for robustness. The paper ships code and data, and the central idea is clearly novel. However, the headline robustness claims are currently not supported by controlled experiments, and the VTAB evaluation has a selection-on-test-set protocol that inflates the reported results. The ideas are promising, but the evidence needs strengthening.

major comments (5)

[§4.3, Table 2] The headline +3.5% (IN-C) and +7% (LAION-C) gains compare 'Self-Soup on IN-C/LA-C' to 'Self-Soup on IN-Train' from Table 1. These recipes differ not only in inter-training data (shifted vs clean) but also in number of ingredients (4 vs 3), SSL algorithms (MAE-only vs MAE/MoCoV3/MMCR), inter-training length (100K steps vs 5 epochs), and fine-tuning schedule. The reported gain is therefore a confounded difference, not an effect of inter-training on shifted data. Provide a matched control where only the inter-training data are changed (same MAE recipe, same learning rates, same 100K steps, same fine-tuning) and ideally also a 'Continued SSL + Supervised Soup' baseline on the same shifted inter-training data.
[§4.3, Table 2] The 'Best ingredient' rows in Table 2 report only the ingredient that maximizes IN-Val accuracy, not the per-ingredient accuracies on IN-C/LA-C. Thus the abstract's claim that the soup is 'more accurate than any single SSL ingredient' is not established for the shifted-data setting. It is possible that one of the four MAE inter-trainings (e.g., the best on IN-C) already reaches or exceeds the soup's 35.72 on IN-C. Report the accuracies of all four ingredients on each corruption set (or at least the max on the target metric) and for the matched control.
[§4.4, Table 4] The caption and text state that 'Best ingredient' and 'Greedy Search' are 'selected on clean test data'—the same clean test data on which accuracy is reported. This is test-set leakage; the reported mean clean accuracy of 67.5% for Self-Soup with greedy search is an oracle selection. The correct protocol is to select on a separate validation split or on the training set (as is done for the 'seasoning' baselines in Table 5). Recompute the VTAB results with a leakage-free selection protocol; the uniform-mix results (which are selection-free) may be the only currently valid numbers.
[Tables 1–5] All experiments are single-run with no error bars, standard deviations, or multiple seeds. Many of the reported gains are small (e.g., +0.4% on LAION-C in Table 1, roughly 1% on VTAB in Table 4) and could be within run-to-run noise. The absence of variance estimates is especially problematic for a paper whose central claims are quantitative. Report results over at least 3 seeds with error bars, or otherwise demonstrate statistical stability.
[§4.3, Table 3 and Appendix B] When computing TTA results with SAR, the base learning rate is reported as 'best' after sweeping {1e-5,...,3e-3}, but the validation set used to select it is not stated. If the selection is performed on the same ImageNet-C odd-index test set used for evaluation, the TTA comparison is biased. Specify the validation protocol; ideally use a held-out split of the corruption set for hyperparameter selection.

minor comments (5)

[Abstract vs. Full Text] The abstract mentions 'LeJEPA' as one of the SSL algorithms used in the soups, but the full text and experiments only use MAE, MoCoV3, and MMCR (Fig. 2, §4.1). Remove LeJEPA from the abstract or provide the corresponding experiments.
[Figure 2] The triangle plots would be easier to read if the corner (single-ingredient) accuracies were printed explicitly; the caption refers to circles at corners but the numerical values are not visible in the text version. Adding numeric labels or a table of the 3 ingredient accuracies per test set would support the LMC claim.
[§2.2, Eq. (1)] The formal definition of linear mode connectivity uses 'acc' but does not clarify which test set it is evaluated on. Since LMC is shown to hold on some test sets and not necessarily all, specify in the definition that LMC is a property with respect to a given evaluation metric and distribution.
[Table 2] The 'joint' setting uses even-indexed corrupted test samples both for inter-training and for evaluation. This is a transductive evaluation and should be clearly labeled in the text as not a measure of generalization to unseen corruptions. The abstract and Section 4.3 do label it, but the 'joint' numbers behind the +3.5% should not be confused with the disjoint results.
[Related Work / Additional References] The paper cites a wide range of literature, but the discussion of continued SSL could mention recent works on self-supervised continued pretraining beyond those listed (e.g., diet-CP is cited, but there are others). This is not essential, but would improve coverage.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation loop: core claims are held-out benchmark measurements on disjoint even/odd test splits with per-ingredient baselines; self-citations are non-load-bearing; TTA tuning and 49-vs-3 mixture selection are caveats, not circularity.

full rationale

The paper's central claims are empirical measurements rather than derivations. The headline result (Eq. 2, §4.3, Table 2) — Self-Souping on even-indexed corrupted test samples, then fine-tuning on uncorrupted train data, +3.5% ImageNet-C / +7% LAION-C — is evaluated on disjoint odd-indexed samples ('In the disjoint setting, we use even-indexed test samples for inter-training on corruptions and odd-indexed samples for testing on corruptions'), and Table 2 reports 'Best ingredient' rows (e.g., 31.41 vs 35.72 IN-C), so the soup's gain is not forced by construction: the uniform soup is a fixed average with no selection on the evaluation metric. The LMC claim (Eq. 1, §4.1, Fig. 2) is verified by plotting 52 mixtures against the one-hot ingredients, an empirical check, not a definitional consequence. The paper also isolates itself from the closest prior combination ('Continued SSL + supervised soup... is a combination of existing methods yet is not a Self-Soup by our definition') and measures an advantage over it (Table 1: +1% IN-C). Self-citations (Fuller et al. 2024 for ImageNet-HR; Shelhamer co-author on Seasoning/Croce et al. 2023 and Tent/Wang et al. 2021; Kyrollos et al. 2023) are background or baseline citations and are not load-bearing for the headline results, which rest on MAE (an external method) with i.i.d. held-out evaluation. Remaining caveats are benchmark-hygiene, not circularity: TTA hyperparameters for Table 3 are swept and 'A 5e-5 base learning rate is best' on the same corrupted evaluation split, and §4.1's 'best models are always a combination' is read from 49 sampled mixtures versus 3 one-hots, a multiple-comparisons caveat. Neither makes any prediction equivalent to its inputs by construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper is empirical, so the ledger mainly records hyperparameter choices and empirical priors. The central claim rests on LMC for SSL-trained ingredients and on the permissibility of using unlabeled test-distribution data for inter-training. No new physical or conceptual entities are postulated.

free parameters (5)

Fine-tuning learning rate grid = {6e-5, 8e-5, 1e-4, 1.5e-4}
Ingredients for ImageNet soups are swept over these learning rates; observed soup gains depend on having diverse fine-tuned ingredients.
Inter-training learning rate grid = {1e-5, 2e-5, 3e-5, 4e-5}
Used for test-distribution inter-training in §4.3; the diversity of this sweep drives the robustness gains.
MAE SSL hyperparameters for §4.2 = mask ratio 90%, 1 decoder layer
Authors state these were 'mostly chosen arbitrarily'; they affect ingredient diversity.
TTA base learning rate = 5e-5
Selected as best on the same ImageNet-C odd-index split used for evaluation (Appendix B, Table 3), adding favorable selection to the TTA comparison.
Self-Seasoning kNN hyperparameters = k=16, T=0.07, 100 epochs
Used for entropy-based mixing; chosen a priori (k=16 generally works), not swept.

assumptions (4)

domain assumption Ingredients that share the same pretrained stock are linearly mode connected after supervised fine-tuning; this extends to SSL inter-training.
Paper relies on LMC as precondition for soups; verified empirically in Fig. 2/4 but not proven (Eq. 1, §2.2).
domain assumption Algorithm-specific SSL heads can be discarded; mixing backbones/encoders preserves useful representations.
Appendix B: 'Before mixing or fine-tuning the ingredients, we discard all algorithm-specific heads.' If false, the soups would not work.
ad hoc to paper Unlabeled samples from the test distribution (even-indexed corrupted samples) may be used for inter-training while still calling the odd-indexed evaluation a fair robustness measure.
§4.3 sets this transductive setting; without permission to use test-distribution data, the headline +3.5/+7 gains do not apply.
domain assumption Minimizing kNN entropy on unlabeled training data selects good mixture coefficients.
Self-Seasoning (§4.5, Appendix D) assumes entropy minimization is a valid unsupervised proxy; shown competitive on 4/21 tasks but not universally.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Self-Soupervision: Cooking Model Soups without Labels." pith.science (2026). https://pith.science/paper/7VANNMS4

@misc{pith2026260202890,
  author       = {Pith},
  title        = {Pith review of: Self-Soupervision: Cooking Model Soups without Labels},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7VANNMS4}},
  note         = {Machine review of arXiv:2602.02890}
}

read the original abstract

Model soups are strange and strangely effective combinations of parameters. They take a model (the stock), fine-tune it into multiple models (the ingredients), and then mix their parameters back into one model (the soup) to improve predictions. While all known soups require supervised learning, and optimize the same loss on labeled data, our recipes for Self-Soupervision generalize soups to self-supervised learning (SSL). Our Self-Souping lets us flavor ingredients on new data sources, e.g. from unlabeled data from a task for transfer or from a shift for robustness. We show that Self-Souping on corrupted test data, then fine-tuning back on uncorrupted train data, boosts robustness by +3.5% (ImageNet-C) and +7% (LAION-C). Self-Soupervision also unlocks countless SSL algorithms to cook the diverse ingredients needed for more robust soups. We show for the first time that ingredients can differ in their SSL hyperparameters -- and more surprisingly, in their SSL algorithms. We cook soups of MAE, MoCoV3, MMCR, and LeJEPA ingredients that are more accurate than any single SSL ingredient.

Figures

Figures reproduced from arXiv: 2602.02890 by the authors.

**Figure 1.** Soups & Supervision. Soups fine-tune then mix models to improve predictions. The original Model Soup (left) fine-tunes across hyperparameters for the task, while Model Ratatouilles (center) first inter-trains on different labeled data then fine-tunes for the task. Both are supervised and cannot harness unlabeled data. Our Self-Soupervision (right) inter-trains across different losses and data to make more and better… view at source ↗

**Figure 2.** Souping across different SSL algorithms. We explore mixtures of 3 ingredient models that we inter-train with different selfsupervised losses: MAE, MoCoV3, and MMCR. MAE is a pixel-reconstruction algorithm, MoCoV3 is an instance-contrastive algorithm, and MMCR is a dimension-contrastive algorithm. Through this experiment, we are the first to show that linear-mode connectivity can hold between ingredients that differ… view at source ↗

**Figure 3.** Model Soup mixes ingredients from a fine-tuning hyperparameter search; likewise, our Self-Soupervision mixes ingredients from a continued SSL + fine-tuning hyperparameter search. Continued SSL enables learning from unlabeled data—e.g., from the fine-tuning domain or test-set shift—before fine-tuning. Self-Souping these diverse ingredients further improves accuracy. Symbols follow [PITH_FULL_IMAGE:figures/full_fig_p… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Self-Souping is possible and productive for another stock: Franca. (Venkataramanan et al., 2025) Mixing models independently trained using SSL (= ingredients) improves accuracy over ingredients alone. We initialize from Franca’s ViT-B, intertrain using SSL, fine-tune …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

3 extracted references · 3 linked inside Pith

[274]

Chen, M., Jiang, M., Zhang, X., Dou, Q., Wang, Z., and Li, X

Springer, 2024. Chen, M., Jiang, M., Zhang, X., Dou, Q., Wang, Z., and Li, X. Local superior soups: A catalyst for model merg- ing in cross-silo federated learning.Advances in Neural Information Processing Systems, 37:20858–20886, 2024. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentat...

arXiv 2024
[2022]

best ingredient

URL https://openreview.net/forum? id=UYneFzXSJWh. Kyrollos, D. G., Fuller, A., Greenwood, K., Harrold, J., and Green, J. R. Under the cover infant pose estimation using multimodal data.IEEE Transactions on Instrumentation and Measurement, 72:1–12, 2023. LeCun, Y ., Huang, F. J., and Bottou, L. Learning methods for generic object recognition with invarianc...

arXiv 2023
[2025]

Bachmann, R., Mizrahi, D., Atanov, A., and Zamir, A

URL https://openreview.net/forum? id=tiEsJtw3FH. Bachmann, R., Mizrahi, D., Atanov, A., and Zamir, A. Mul- tiMAE: Multi-modal multi-task masked autoencoders. In European Conference on Computer Vision, pp. 348–367. Springer, 2022. Balestriero, R., Ibrahim, M., Sobal, V ., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y ....

arXiv 2022

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [274]

Chen, M., Jiang, M., Zhang, X., Dou, Q., Wang, Z., and Li, X

Springer, 2024. Chen, M., Jiang, M., Zhang, X., Dou, Q., Wang, Z., and Li, X. Local superior soups: A catalyst for model merg- ing in cross-silo federated learning.Advances in Neural Information Processing Systems, 37:20858–20886, 2024. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentat...

arXiv 2024

[2] [2022]

best ingredient

URL https://openreview.net/forum? id=UYneFzXSJWh. Kyrollos, D. G., Fuller, A., Greenwood, K., Harrold, J., and Green, J. R. Under the cover infant pose estimation using multimodal data.IEEE Transactions on Instrumentation and Measurement, 72:1–12, 2023. LeCun, Y ., Huang, F. J., and Bottou, L. Learning methods for generic object recognition with invarianc...

arXiv 2023

[3] [2025]

Bachmann, R., Mizrahi, D., Atanov, A., and Zamir, A

URL https://openreview.net/forum? id=tiEsJtw3FH. Bachmann, R., Mizrahi, D., Atanov, A., and Zamir, A. Mul- tiMAE: Multi-modal multi-task masked autoencoders. In European Conference on Computer Vision, pp. 348–367. Springer, 2022. Balestriero, R., Ibrahim, M., Sobal, V ., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y ....

arXiv 2022