REVIEW 17 cited by
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder. In this paper, we revisit the second step of this procedure in the context of fine-tuning large pre-trained models, where fine-tuned models often appear to lie in a single low error basin. We show that averaging the weights of multiple models fine-tuned with different hyperparameter configurations often improves accuracy and robustness. Unlike a conventional ensemble, we may average many models without incurring any additional inference or memory costs -- we call the results "model soups." When fine-tuning large pre-trained models such as CLIP, ALIGN, and a ViT-G pre-trained on JFT, our soup recipe provides significant improvements over the best model in a hyperparameter sweep on ImageNet. The resulting ViT-G model, which attains 90.94% top-1 accuracy on ImageNet, achieved a new state of the art. Furthermore, we show that the model soup approach extends to multiple image classification and natural language processing tasks, improves out-of-distribution performance, and improves zero-shot performance on new downstream tasks. Finally, we analytically relate the performance similarity of weight-averaging and logit-ensembling to flatness of the loss and confidence of the predictions, and validate this relation empirically. Code is available at https://github.com/mlfoundations/model-soups.
Forward citations
Cited by 17 Pith papers
-
Unlearning as Distribution Restoration: A Controlled Counterfactual Study, a Validated Selective Screen, and the Limits of Oracle-Free Certification
Matching a retrained oracle on trained probes can certify models that still retain held-out forget knowledge, and oracle-free unlearning certification is only possible for counterfactual, non-inferable facts.
-
Making Open-Source Text LLM Watermarks Durable Against Merging
A new training method, Merge-Adversarial Training, makes watermarks embedded in open-source LLMs survive model merging, boosting post-merge detection by up to 51 percentage points.
-
Asymmetric Collapse in Model Merging: When Refusal Over- writes Recognition
Across four standard merging methods, refusal behavior from a large task vector overwrites fine-grained harm classification, leaving at most 12.9% accuracy.
-
FORGE-plus: Force-Budgeted Recovery for Contact-Rich Assembly with a Frozen LLM Supervisor
With a hidden per-episode breaking force, an LLM-set force ceiling plus force-signature recovery achieves 256/256 clean insertions on fragile and robust parts and resolves 40–64% of injected jams in simulation.
-
TwistedMerge: Certified Higher-Order Diagnostics and Abstention for Model Merging
Cycle inconsistency in model merging is not automatically a cohomological obstruction: TwistedMerge certifies a class only after frozen-complex, centrality, closure, and statistical gates, and finds no natural central class.
-
Persona Cartography: Charting Language Model Personality Traits in Weight Space
Composable LoRA adapters can amplify or suppress OCEAN traits in LLMs, combine approximately additively, preserve moderate-scale capability, and move safety-relevant behaviours.
-
DivMerge: A divergence-based model merging method for multi-tasking
DivMerge learns task-arithmetic merging weights by minimizing Jensen-Shannon divergence between each specialist model and the merged model, improving multi-task performance and scalability.
-
Forget the Data and Fine-Tuning! Just Fold the Network to Compress
Model folding compresses a network by k-means clustering similar neurons across adjacent layers and repairing activation statistics without data (Fold-AR, Fold-DIR), surpassing prior data-free methods at high sparsity.
-
Model Fusion via Neuron Transplantation
A new fusion method, Neuron Transplantation, concatenates ensemble members and prunes back down to a single model's size, outperforming individual models after fine-tuning.
-
Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch
Streaming DiLoCo trains billion-parameter LLMs at data-parallel quality while cutting the inter-datacenter bandwidth by about two orders of magnitude via partial, overlapped, and 4-bit-quantized synchronization.
-
Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs
Iterative Puzzle plus KD, RL, quantization, and MTP compresses Nemotron-3-Super to 75B total / 9B active parameters with ~2× interactive throughput and 8× 1M-context concurrency while retaining most parent accuracy.
-
First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers
Single task-vector perturbations around a multitask LoRA point are first-order linear to 1e-2, but pairwise update-order sensitivity is set by a per-pair Lie bracket and can appear inside that window.
-
AlphaWiSE: Adaptive Weight Interpolation for Continual Multimodal Representation Learning
Fitting one interpolation coefficient per parameter tensor on a small exemplar memory improves continual audio–image–text retrieval over individual continual-learning checkpoints.
-
Clinical Semantic Intelligence (CSI): Emulating the Cognitive Framework of the Expert Clinician for Comprehensive Oral Disease Diagnosis
A CLIP+ChatGLM system with a six-level decision tree claims 89.5% internal and 85.2% external accuracy for 118 oral diseases, but the hierarchical-reasoning benefit is confounded with extra user input.
-
Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts
Sparse adapters trained with max connection sensitivity outperform LoRA and full fine-tuning both alone and after merging 20 task experts, but still lag multitask training on unseen tasks.
-
Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques
A survey that maps LLM applications, vulnerabilities, and defenses across eight cybersecurity domains, but with significant citation and rigor problems.
-
Domain Expansion: Parameter-Efficient Modules as Building Blocks for Composite Domains
Weight-space arithmetic over LoRA/IA3 trait adapters produces composite MBTI personality models that mostly match their target type on an online quiz, but the combining weights are tuned on the same quiz.
Discussion (0). Continue with ORCID to comment.