Pith. sign in

REVIEW 3 major objections 4 minor 22 references

The paper claims that a single fine-tuned checkpoint, edited by reweighting the high- and low-energy singular components of each layer's update, can match or exceed the ID–OOD balance of multi-checkpoint model soups.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-03 02:43 UTC pith:CHTPNHY5

load-bearing objection MonoSoup's single-checkpoint reweighting looks real and useful, but the 'hyperparameter-free' claim is the weak link—the automatic entropy rank consistently trails the tuned R=0.8. the 3 major comments →

arxiv 2602.09689 v2 pith:CHTPNHY5 submitted 2026-02-10 cs.LG

Model soups need only one ingredient

Alireza Abdollahpoorrostam , Nikolaos Dimitriadis , Adam Hazimeh , Pascal Frossard This is my paper
classification cs.LG
keywords MonoSoupmodel soupsweight-space mergingsingular value decompositioneffective rankout-of-distribution robustnessfine-tuningCLIP
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Fine-tuning a pretrained model on a target dataset typically improves in-distribution accuracy but erodes performance on shifted inputs. MonoSoup claims this trade-off can be fixed with the fine-tuned checkpoint alone: decompose each layer's weight update into a high-energy part that encodes task adaptation and a low-energy part that carries robustness-relevant residuals, then reweight the two parts using a rule that reads the layer's spectral decay and geometric alignment. The method needs no data, no extra checkpoints, and in its fully automated form no hyperparameters. On CLIP ViT-B/32 it raises average OOD accuracy from 50.67% to 51.60% on the strongest checkpoint and recovers up to 7.9 points on collapsed checkpoints, with similar gains on Qwen math benchmarks. If correct, this means the benefits of weight averaging across dozens of models can be obtained from a single stored model, changing the economics of robust deployment.

Core claim

The central discovery is that within a single fine-tuned checkpoint, the layer-wise update can be split by singular value decomposition into a dominant, task-specialized subspace and a low-energy complement that preserves pretrained features needed for OOD robustness; contrary to low-rank-truncation practice, the tail cannot be discarded—it must be damped, not deleted. MonoSoup reweights the two subspaces with lambda_Low = rho + (1 - rho) cos alpha, where rho is the squared ratio of the first excluded singular value to the leading one and cos alpha is the fraction of update energy in the low-energy subspace, and it sets the split point automatically with the entropy-based effective rank. The

What carries the argument

The engine is the per-layer SVD of the fine-tuning task vector W = W_FT - W_0. Each layer's update is partitioned into W_High (singular vectors above an automatic entropy-rank threshold) and W_Low (the orthogonal complement); the mixing coefficient lambda_Low = rho + (1 - rho) cos alpha, with rho the squared spectral decay ratio and cos alpha the fractional energy in the low-energy subspace, is the minimal bilinear function satisfying the suppression and retention boundary conditions. This delivers anisotropic, layer-wise scaling rather than a uniform interpolation, which is what lets the method preserve task-specific directions while damping the noisy tail.

Load-bearing premise

The fully automatic version rests on the assumption that the entropy-based effective rank selects a near-optimal split between high- and low-energy directions for every layer and architecture; the paper itself notes a residual reliance on a variance-retention threshold R, and the automated form loses about 1.4 OOD points against R=0.8 on the worst-OOD checkpoint.

What would settle it

Take a held-out model family not used in the paper (e.g., BERT-Large fine-tuned on GLUE, or a ViT-L/14 fine-tuned on a non-ImageNet distribution), sweep R across [0.2, 0.9], and compare against automated MonoSoup. If the entropy-selected k falls outside the optimal R range on a substantial fraction of layers, or if the automated version underperforms MonoSoup(R=0.8) by more than roughly 2 OOD points averaged over many checkpoints, the hyperparameter-free claim is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • With one checkpoint, MonoSoup raises average OOD accuracy of the best-OOD CLIP ViT-B/32 from 50.67% to 51.60%, surpassing Greedy Soup while using only a single model.
  • On collapsed checkpoints, it recovers up to roughly 7.5–7.9 OOD accuracy points while maintaining or improving ID accuracy.
  • The automated entropy-based form trails the tuned R=0.8 version by about 1.4 OOD points on the worst-OOD checkpoint but still captures most of the benefit with no hyperparameter.
  • On Qwen3-0.6B, MonoSoup improves math reasoning benchmarks across all three fine-tuning settings and matches or exceeds pairwise ModelStock with a single checkpoint.
  • MonoSoup is complementary to Wise-FT: applying it first gives a stronger endpoint, and the resulting Pareto fronts dominate Wise-FT alone and LiNeS+Wise-FT.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is whether entropy-rank partitioning stays near-optimal on non-ImageNet domains, such as medical imaging or multilingual text; if it does, MonoSoup becomes a drop-in safety layer for any deployed fine-tune.
  • The same decomposition could serve as a diagnostic: layers with large rho/cos-alpha gaps identify where fine-tuning most distorts pretrained features and where OOD degradation originates.
  • Because MonoSoup is data-free and one-shot, it could be composed with task arithmetic pipelines as a pre-processing step to reduce interference before merging or editing.
  • The coefficient's 1-Lipschitz dependence on rho and cos alpha suggests a continuous family of edited checkpoints, which could be used for lightweight ID-OOD trade-off control at inference time.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes MonoSoup, a post-hoc, single-checkpoint method that reweights the high- and low-energy singular components of each layer's fine-tuning update. The decomposition is defined in Eqs. (2)–(8): a layer update W^(l) is split by SVD into W_High and W_Low using either a variance-retention threshold R or an entropy-based effective rank, and recombined with coefficients λ_Low = ρ + (1−ρ)cosα and λ_High = 1−λ_Low. The authors claim this 'data-free, hyperparameter-free' edit improves OOD robustness while maintaining ID accuracy and 'matches or exceeds' multi-model methods such as Model Soups and Model Stock. Experiments are reported on CLIP ViT-B/32 (four representative checkpoints plus all 70 in an appendix), ConvNeXt, and Qwen3-0.6B, and on integration with Wise-FT.

Significance. If the automatic-rank component were sound, MonoSoup would be a practically useful post-hoc robustness tool: it is data-free, operates on a single checkpoint, and the coefficient rule in Eq. (8) is derived from clear boundary conditions. The paper also contains a valuable empirical observation—low-energy directions are not simply noise in the ImageNet-scale fine-tuning regime (Fig. 3)—and the CKA analysis (Appendix G) connects the weight-space edit to feature-space behavior. The all-70-checkpoint plot (Fig. 12) is a strength, as is the cross-domain evidence on CLIP, ConvNeXt, and Qwen. However, the central 'hyperparameter-free' claim is not supported, and the headline comparison to multi-model methods is overstated. These issues are load-bearing rather than cosmetic.

major comments (3)
  1. [§4, Eqs. (2)–(4), Table 1] The automatic variance-free rank k^(ℓ)=exp(−Σ p_i ln p_i), with p_i=σ_i/Σ_j σ_j, is never calibrated against the energy-based threshold R of Eq. (2). The two criteria use different normalizations (L1 over singular values vs. L2 energy), and no argument or experiment shows that the entropy rank selects a near-optimal R per layer or architecture. Table 1 quantifies the cost: automatic MonoSoup loses 0.69, 1.43, 1.01, and 1.32 OOD points on OOD+, OOD−, ID+, and ID− relative to MonoSoup(R=0.8), and Table 5 shows a 0.63-point loss on ConvNeXt. Section 7's admission that 'a current limitation is the reliance on a variance-retention threshold R' directly contradicts the abstract's 'hyperparameter-free' claim. This is the load-bearing contribution, and it is not established.
  2. [§5.1, Table 1, Abstract] The headline 'matches or exceeds multi-model methods while using just one fine-tuned checkpoint' is not supported by the reported numbers. Automated MonoSoup on the OOD+ checkpoint reaches 50.91% Avg. OOD, below Uniform Model Soup's 51.4%. In the pairwise block, ModelStock has higher OOD on three of six pairs (OOD+,OOD−: 49.39 vs. 48.37; OOD+,ID−: 51.02 vs. 50.48; ID−,OOD−: 47.81 vs. 47.79). Figure 12 shows improvement over the fine-tuned endpoint across 70 checkpoints, but does not compare against soup baselines per checkpoint. The claim should be restricted to 'competitive with' or supported by per-pair error bars and a full 70-checkpoint comparison against Uniform/Greedy Soup.
  3. [§5.4, Appendix H] The sensitivity analysis is not connected to the automatic rank. Figure 5 and Appendix H show that OOD performance is non-monotonic in R, with the best R varying by initialization and architecture (roughly 0.4–0.85). This makes the missing calibration more consequential: a proxy that truly removes R needs to stay within a small margin of the optimal R across this range, but Tables 1 and 5 show it does not. Please provide a calibration study (e.g., per-layer correlation between entropy rank and the optimal R, or a grid showing the entropy rank's induced R per layer) or change the abstract to recommend a fixed default R instead of claiming the method is hyperparameter-free.
minor comments (4)
  1. [§4, Eq. (8)] Typo 'coeffiecient' should be 'coefficient'. The four boundary conditions are described in prose; a small table would make the derivation of Eq. (8) easier to follow.
  2. [Table 2] The header contains 'MONOSOUP' and 'MONOSOUPR=0.8' with missing spaces; also 'MonoSoupand' appears in the table caption. Please fix formatting.
  3. [Table 3] The naming 'MonoSoup +FT model' is inconsistent with Eq. (5), where MonoSoup is applied to the update of a single fine-tuned model. Clarify that the plus sign denotes application to the checkpoint, not an ensemble.
  4. [Figure 13] The axis label contains literal ' and ' formatting that appears to be a rendering artifact; the labels for λ_High and λ_Low are not legible. Also state whether the coefficients shown are from the automatic entropy-rank variant or from a fixed R.

Circularity Check

0 steps flagged

No circular derivation: MonoSoup's reweighting is constructed from the checkpoint's own SVD spectrum and evaluated on held-out OOD shifts; self-citations are baselines/data sources, not load-bearing. Section 7's admitted reliance on R undercuts the 'hyperparameter-free' label but is not a circular step.

full rationale

The central derivation (Eqs. 2-8) is self-contained: W_High and W_Low are orthogonal SVD components of the layer update; k is selected by spectral energy (Eq. 2) or entropy effective rank (Eq. 3); and lambda_Low = rho + (1-rho) cos(alpha) is fixed from the spectral decay ratio and the low-energy energy fraction (Eqs. 6-8). No parameter is fitted to ID/OOD labels, and the resulting single edited checkpoint is evaluated on unseen natural distribution shifts and held-out language benchmarks, so the reported gains are external measurements rather than identities. The same-group citations (Wang et al. 2024a for the 20-task benchmark; LiNeS as a comparison baseline; Dimitriadis et al. in related work) are used as data or comparisons, not to force the MonoSoup equations; there is no imported uniqueness theorem or ansatz. I flag one non-circular validity concern: Section 7 admits 'a current limitation is the reliance on a variance-retention threshold R', and Table 1 shows the automated entropy-rank variant trails MonoSoup(R=0.8) by 0.4-1.4 OOD points. This makes the abstract's 'hyperparameter-free' claim overstated, but it is an uncalibrated-proxy/validity issue, not a reduction of the prediction to the method's inputs by construction. Overall, no significant circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The automated method introduces no fitted scalar constants, but it assumes that the entropy-based spectral partition and the bilinear reweighting rule are universally valid. The fixed-R variant introduces R=0.8, chosen from sweeps. No new physical entities are postulated.

free parameters (2)
  • variance-retention threshold R = 0.8
    The fixed-threshold variant MonoSoup(R=0.8) is reported as a main result; R is selected from sensitivity sweeps (Fig. 5 on ViT-L/14; Appendix H on CLIP/ConvNeXt), and the conclusion admits reliance on R. The hyperparameter-free version replaces R with the entropy effective rank, but the R-based variant remains central to the paper's headline numbers.
  • Similarity-Filtered Greedy Soup alignment threshold delta = unspecified
    SFGS uses a cosine-similarity threshold delta to accept checkpoints, but the paper never reports its value. This parameter only supports the motivation and is not part of MonoSoup itself.
axioms (5)
  • domain assumption Linear mode connectivity: fine-tuned checkpoints from the same pretrained initialization lie in a low-loss basin where weight averaging is meaningful.
    Relied on in Sections 2-3 to justify treating the fine-tuning update as the object to reweight; cited to Frankle et al. (2020) and Garipov et al. (2018).
  • ad hoc to paper SVD of each layer's update separates task-specific (high-energy) from OOD-preserving (low-energy) subspaces.
    Core hypothesis of Section 4; supported empirically by Figure 3 and Appendix G, but not proven. If false, the reweighting may amplify noise or discard useful signal.
  • ad hoc to paper Entropy-based effective rank k=exp(-sum p_i ln p_i) is a valid automatic proxy for the variance-retention threshold R.
    Eq. 3; the hyperparameter-free version depends on this. Table 1 shows it underperforms R=0.8 on the worst-OOD checkpoint, and Section 7 admits R remains a limitation.
  • ad hoc to paper The four boundary conditions in Eq. 8 encode the correct behavior of low-energy weighting, and the bilinear interpolation satisfying them transfers across architectures.
    The rule is unique given the stated boundary conditions, but the conditions themselves are chosen heuristically; transferability is an empirical claim validated only on the tested benchmarks.
  • ad hoc to paper Low-energy directions in the update carry OOD-relevant information on ImageNet-scale fine-tuning, unlike small-task settings.
    Supported by Figure 3b, Figure 6, and the CKA analysis in Appendix G; this is a key empirical claim and is only demonstrated in the tested regimes.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Model soups need only one ingredient." pith.science (2026). https://pith.science/paper/CHTPNHY5

@misc{pith2026260209689,
  author       = {Pith},
  title        = {Pith review of: Model soups need only one ingredient},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CHTPNHY5}},
  note         = {Machine review of arXiv:2602.09689}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Fine-tuning large pre-trained models on a target distribution often improves in-distribution (ID) accuracy, but at the cost of out-of-distribution (OOD) robustness as representations specialize to the fine-tuning data. Weight-space ensembling methods, such as Model Soups, mitigate this effect by averaging multiple checkpoints, but they are computationally prohibitive, requiring the training and storage of dozens of fine-tuned models. In this paper, we introduce MonoSoup, a simple, data-free, hyperparameter-free, post-hoc method that achieves a strong ID-OOD balance using only a single checkpoint. Our method applies Singular Value Decomposition (SVD) to each layer's update and decomposes it into high-energy directions that capture task-specific adaptation and low-energy directions that introduce noise but may still encode residual signals useful for robustness. MonoSoup then uses entropy-based effective rank to automatically re-weigh these components with layer-wise coefficients that account for the spectral and geometric structure of the model. Experiments on CLIP models fine-tuned on ImageNet and evaluated under natural distribution shifts, as well as on Qwen language models tested on mathematical reasoning and multiple-choice benchmarks, show that this plug-and-play approach is a practical and effective alternative to multi-checkpoint methods, retaining much of their benefits without their computational overhead.

Figures

Figures reproduced from arXiv: 2602.09689 by Adam Hazimeh, Alireza Abdollahpoorrostam, Nikolaos Dimitriadis, Pascal Frossard.

Figure 1
Figure 1. Figure 1: Performance and alignment analysis of Model Stock on 2,409 pairwise combinations of CLIP ViT-B/32 models fine-tuned on ImageNet. (a) Scatter plot of ID vs. OOD performance relative to the better constituent model. (b) and (c): Layer-wise cosine similarity for low-performing and high-performing, respectively. Stronger alignment coincides with consistent gains, highlighting that alignment can serve as a key … view at source ↗
Figure 2
Figure 2. Figure 2: Performance of Similarity-Filtered Greedy Soup (SFGS). Evaluated on CLIP ViT-B/32 checkpoints, SFGS achieves competitive ID and OOD performance relative to validation-based greedy soup. This supports the finding that geometric alignment is a key indicator of merging effectiveness. Soup, achieves performance comparable to validation-based Greedy Soup (Wortsman et al., 2022a), showing that geomet￾ric alignme… view at source ↗
Figure 3
Figure 3. Figure 3: Effect of truncating low-energy components on different benchmarks. (a) On the 20-task vision benchmark, performance saturates after retaining only a small number of singular values, consistent with prior reports that low-rank updates suffice. (b) On ImageNet with natural OOD shifts, truncation substantially reduces both ID and OOD accuracy, even when preserving 95% of spectral energy. This highlights that… view at source ↗
Figure 4
Figure 4. Figure 4: MonoSoup integrated with Wise-FT on CLIP ViT-B/32. MonoSoup improves ID and OOD accuracy across individual checkpoints. When combined with Wise-FT, the Pareto fronts consistently dominate those of Wise-FT and LiNeS, showing that MonoSoup provides a stronger endpoint for interpolation-based robustness. 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Variance Threshold (R) 80.0 82.0 84.0 86.0 ImageNet Accuracy (%) 0.2 0.3 0… view at source ↗
Figure 5
Figure 5. Figure 5: Component Analysis. Effect of varying the variance threshold R and the contributions of each term in the coefficient λ ℓ = λ ℓ low on CLIP ViT-L/14. Results are stable across a wide range of R values, and both the spectral decay and cosine overlap components contribute meaningfully to the final balance between ID and OOD performance. 2023; Lee et al., 2023; Goyal et al., 2023; Wortsman et al., 2022b; Mao e… view at source ↗
Figure 6
Figure 6. Figure 6: The task vector rank consistently enhances performance on both ID and OOD benchmarks. The x-axis represents the rank of the task vector, with blue curves indicating ID accuracy and red curves depicting average OOD accuracy. Since cos α = ∥WLow-Energy∥F/∥W∥F , we obtain cos2 α = ∥WLow-Energy∥ 2 F ∥W∥ 2 F = ∥W∥ 2 F − ∥WHigh-Energy∥ 2 F ∥W∥ 2 F = 1 − Pk. Theorem 1. The angle parameter α satisfies the followin… view at source ↗
Figure 7
Figure 7. Figure 7: Mean absolute accuracy of the CLIP ViT-{B/32, B/16, L/14} models across increasing fractions of retained singular components, averaged over 20 tasks released by (Wang et al., 2024a). The yellow line represents the average accuracy of the original fine-tuned models with full-rank task matrices, while the blue line shows the accuracies using low-rank approximations. E. Qwen fine-tuning In this section, we de… view at source ↗
Figure 8
Figure 8. Figure 8: Feature-space alignment maps (CKA). (a) CKA of the model M-14 (worst-OOD) on the ImageNet-A (OOD) dataset (b) CKA of the model M-61 (best-OOD) on the ImageNet-A (OOD) dataset For each transformer block ℓ, we compare the hidden features of: (a) Pre-trained, (b) fine-tuned, (c) MonoSoup, (d) High-only (WHigh ), (e) Low-only (WLow). We compute linear CKA (Kornblith et al., 2019) to the pre-trained features on… view at source ↗
Figure 9
Figure 9. Figure 9: Feature-space alignment maps (CKA). (a) CKA of the model M-14 (worst-OOD) on the ImageNet (ID) dataset (b) CKA of the model M-61 (best-OOD) on the ImageNet (ID) dataset increase of 0.04–0.05), which corresponds to the observed OOD performance gains. Conversely, on stronger backbones where fine-tuning already maintains a high similarity to pretrained features (CKA ≥ 0.9), MonoSoup’s adjustments and accuracy… view at source ↗
Figure 10
Figure 10. Figure 10: Ablation on variance threshold R. Zero-Shot initialization. To assess the generalizability of our method, we evaluate its performance on two CLIP ViT-B/32 models that were fine-tuned from zero-shot initialization (models utilized in Appendix B), rather than from linear probing initialization [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Ablation study on the variance threshold R according to the ConvNeXt model. ConvNeXt model. In this part, we assess the generalizability of our method on ConvNeXt1 (Liu et al., 2022) model pretrained on ImageNet-22k and fine-tuned on ImageNet-1k (See [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Application of MonoSoup on 70 CLIP checkpoints. To demonstrate robustness across a variety of models, we further assess our method using all 70 CLIP checkpoints provided by (Wortsman et al., 2022a). We present a vector plot in which the x- and y-axes represent ID and OOD performance, respectively. Each vector begins at the performance level of a fine-tuned checkpoint and extends to the performance achieve… view at source ↗
Figure 13
Figure 13. Figure 13: Evidence of Anisotropic Scaling. Distribution of the computed mixing coefficients λ (ℓ) High and λ (ℓ) Low across different layers of the ConvNeXt model. J. Anisotropic Scaling A potential critique of single-model editing is that it might collapse to simple scalar interpolation, which means effectively just finding an optimal mixing coefficient α for the entire weight matrix. We empirically verify that Mo… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 1 canonical work pages

  1. [3]

    Finetune like you pretrain: Improved finetuning of zero-shot vision models

    Goyal, S., Kumar, A., Garg, S., Kolter, Z., and Raghunathan, A. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19338–19347, Vancouver, BC, Canada, June

    2023

  2. [9]

    Mao, X., Chen, Y ., Jia, X., Zhang, R., Xue, H., and Li, Z

    URL https://arxiv.org/abs/ 1711.05101. Mao, X., Chen, Y ., Jia, X., Zhang, R., Xue, H., and Li, Z. Context-aware robust fine-tuning. Int. J. Comput. Vision, 132(5):1685–1700, December

    Pith/arXiv arXiv

  3. [12]

    Oh, C., Lim, H., Kim, M., Han, D., Yun, S., Choo, J., Haupt- mann, A., Cheng, Z.-Q., and Song, K

    URLhttps://arxiv.org/abs/2404.00860. Oh, C., Lim, H., Kim, M., Han, D., Yun, S., Choo, J., Haupt- mann, A., Cheng, Z.-Q., and Song, K. Towards calibrated robust fine-tuning of vision-language models,

    Pith/arXiv arXiv

  4. [13]

    Radford, A., Kim, J

    URL https://arxiv.org/abs/2311.01723. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research,

    Pith/arXiv arXiv

  5. [14]

    High-Resolution Image Synthesis with Latent Diffusion Models

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-Resolution Image Synthesis with Latent Diffusion Models. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10674–10685, New Orleans, LA, USA, June

    2022

  6. [15]

    ISBN 978-1-6654-6946-3

    IEEE. ISBN 978-1-6654-6946-3. doi: 10.1109/CVPR52688. 2022.01042. Roy, O. and Vetterli, M. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pp. 606–610. IEEE,

    arXiv 2022

  7. [16]

    Tang, A., Yang, E., Shen, L., Luo, Y ., Hu, H., Du, B., and Tao, D

    URLhttps://arxiv.org/abs/2410.19735. Tang, A., Yang, E., Shen, L., Luo, Y ., Hu, H., Du, B., and Tao, D. Merging models on the fly without retraining: A sequential approach to scalable continual model merg- ing,

    Pith/arXiv arXiv

  8. [17]

    Wang, H., Ge, S., Lipton, Z., and Xing, E

    URL https://arxiv.org/abs/2502.03461. Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning Robust Global Representations by Pe- nalizing Local Predictive Power. In Advances in Neural Information Processing Systems, volume

    Pith/arXiv arXiv

  9. [18]

    Mmlu-pro: A more robust and challenging multi- task language understanding benchmark, 2024b

    Wang, Y ., Ma, X., Zhang, G., Ni, Y ., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark, 2024b. URL https://arxiv.org/abs/2406.01574. Welbl, J., Liu, N. F., and Gardner, M. Crowdsourc- ing ...

    Pith/arXiv arXiv

  10. [19]

    Yang, E., Wang, Z., Shen, L., Liu, S., Guo, G., Wang, X., and Tao, D

    URL https: //arxiv.org/abs/2505.09388. Yang, E., Wang, Z., Shen, L., Liu, S., Guo, G., Wang, X., and Tao, D. AdaMerging: Adaptive model merging for multi-task learning. In International Conference on Learning Representations (ICLR),

    Pith/arXiv arXiv

  11. [20]

    T., Li, Z., Weller, A., and Liu, W

    Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y ., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284,

    Pith/arXiv arXiv

  12. [21]

    Performance metrics are presented for ImageNet and the average of five OOD datasets across fine-tuned models utilizingLP initializationreleased by Wortsman et al

    11 Model soups need only one ingredient Table 3.A comprehensive comparison of single-model merging methods on CLIP ViT-B/32. Performance metrics are presented for ImageNet and the average of five OOD datasets across fine-tuned models utilizingLP initializationreleased by Wortsman et al. (2022a). For Wise-FT, we sweep the interpolation coefficientαand repo...

    2024

  13. [32]

    To mitigate train–test context mismatch, we fix the context length at 2,048 tokens. We evaluate three specific model variants: M-1, which utilizes a linear learning rate schedule; M-2, which employs a cosine schedule; and M-3, which uses a cosine schedule but is trained for two additional epochs compared to the others. F. Derivation and Interpretability o...

    2019

  14. [1989]

    doi: 10.1016/S0079-7421(08)60536-8

    ISBN 978-0-12- 543324-2. doi: 10.1016/S0079-7421(08)60536-8. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381–2391,

    doi:10.1016/s0079-7421(08)60536-8 2018

  15. [2013]

    ISBN 978-1-4799-3022-7

    IEEE. ISBN 978-1-4799-3022-7. doi: 10.1109/ICCVW.2013.77. Kumar, A., Raghunathan, A., Jones, R., Ma, T., and Liang, P. Fine-Tuning can Distort Pretrained Features and Un- derperform Out-of-Distribution, February

    doi:10.1109/iccvw.2013.77 2013

  16. [2019]

    3D Object Representations for Fine-Grained Catego- rization

    Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3D Object Representations for Fine-Grained Catego- rization. In 2013 IEEE International Conference on Computer Vision Workshops, pp. 554–561, Sydney, Aus- tralia, December

    2013

  17. [2021]

    org/abs/2110.14168

    URL https://arxiv. org/abs/2110.14168. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei- Fei, L. ImageNet: A Large-Scale Hierarchical Image Database

    Pith/arXiv arXiv

  18. [2022]

    Loshchilov, I

    URL https://arxiv.org/abs/2201.03545. Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization,

    Pith/arXiv arXiv

  19. [2023]

    ISBN 979-8-3503-0129-8

    IEEE. ISBN 979-8-3503-0129-8. doi: 10.1109/ CVPR52729.2023.01853. Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Para- juli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generaliza- tion. In 2021 IEEE/CVF International Conference ...

    arXiv 2023

  20. [2024]

    Ling, W., Yogatama, D., Dyer, C., and Blunsom, P

    URL https://arxiv.org/abs/2402.19255. Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. Program induction by rationale generation: Learning to solve and explain algebraic word problems,

    Pith/arXiv arXiv

  21. [2025]

    Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D

    URLhttps://arxiv.org/abs/2412.00081. Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D. P., and Wilson, A. G. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems (NeurIPS),

    Pith/arXiv arXiv

  22. [5691]

    URL https: //doi.org/10.1007/s11263-023-01951-2

    doi: 10.1007/s11263-023-01951-2. URL https: //doi.org/10.1007/s11263-023-01951-2. McCloskey, M. and Cohen, N. J. Catastrophic Interfer- ence in Connectionist Networks: The Sequential Learn- ing Problem. In Psychology of Learning and Motivation, volume 24, pp. 109–165. Elsevier,

    work page doi:10.1007/s11263-023-01951-2

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.