Navigating the Accuracy-Size Trade-Off with Flexible Model Merging
Pith reviewed 2026-05-19 13:17 UTC · model grok-4.3
The pith
Merged models of just double the size deliver up to 13.5 percent accuracy gains and change which merging method works best.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
FlexMerge is a unified data-free framework that generates merged models of arbitrary sizes between a single merged model and the full set of individual fine-tuned models while supporting multiple existing merging algorithms inside one implementation. Systematic evaluation on up to 30 tasks reveals that accuracy improves steeply with modest size growth, reaching gains of 13.5 percent when size is merely doubled, and that algorithm rankings are inconsistent across the size spectrum with some methods overtaking others only after leaving the one-model regime.
What carries the argument
FlexMerge, a flexible merging framework that unifies multiple algorithms and produces models of any intermediate size in a data-free way.
If this is right
- Accuracy-size curves become the primary way to compare merging algorithms rather than single-point single-model scores.
- Deployers can choose an operating point on the curve that trades a small storage increase for a large accuracy increase.
- Existing merging methods need to be re-evaluated at multiple sizes because their relative strengths shift.
- Multi-task systems can be built without either full model retention or full accuracy loss.
Where Pith is reading between the lines
- Hybrid merging strategies that switch algorithms depending on the target size may outperform any single algorithm used uniformly.
- The same size-tuning principle could be applied to merging in other modalities or to models larger than those tested here.
- Storage-aware deployment pipelines could dynamically select model size at inference time based on accuracy budgets.
- Future merging research may benefit from reporting full accuracy-size curves as standard practice.
Load-bearing premise
A single framework can host several merging algorithms and produce models of any size without introducing new accuracy artifacts or needing per-task adjustments.
What would settle it
A controlled run on a standard vision or NLP benchmark where doubling the merged-model size from one to two produces less than a 5 percent accuracy lift or where the relative ordering of algorithms stays identical at every tested size.
read the original abstract
Model merging has emerged as an efficient method to combine multiple single-task fine-tuned models. The merged model can enjoy multi-task capabilities without expensive training. While promising, merging into a single model often suffers from an accuracy gap with respect to the fine-tuned models. On the other hand, deploying all individual fine-tuned models incurs high storage costs. We propose FlexMerge, a novel data-free model merging framework that: (a) flexibly generates merged models of varying sizes, spanning the full spectrum from a single merged model to retaining all fine-tuned models; and (b) supports multiple merging algorithms in a unified framework. Using FlexMerge, we systematically characterize the accuracy-size trade-off of different algorithms. Our study reveals two key findings: first, even modestly larger merged models can yield steep accuracy gains (up to 13.5% when just doubling the size); second, algorithm rankings are not consistent as size increases, with some methods overtaking others beyond the one-model regime. These results uncover a new design dimension for model merging: developing and comparing algorithms across the full spectrum of sizes rather than only at the single-model limit. Extensive experiments on vision and NLP benchmarks, with up to 30 tasks, confirm the generality and practicality of FlexMerge.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces FlexMerge, a data-free model merging framework that unifies multiple merging algorithms (such as TIES and DARE) and flexibly generates merged models of arbitrary sizes spanning from a single model to retaining all fine-tuned models. Using this framework, the authors characterize the accuracy-size trade-off on vision and NLP benchmarks with up to 30 tasks, reporting that modestly larger merged models yield steep accuracy gains (up to 13.5% when doubling size) and that algorithm rankings shift with increasing size, with some methods overtaking others beyond the single-model regime.
Significance. If the size-control mechanism proves neutral and unified, the work establishes model size as a new axis for evaluating and designing merging algorithms, moving beyond the conventional single-model limit. The empirical results on accuracy gains and ranking inconsistencies could inform practical multi-task deployments that balance performance and storage costs.
major comments (2)
- [§3.2] §3.2 (Size-control mechanism): The central claim that algorithm rankings are inconsistent as size increases depends on FlexMerge applying an identical, neutral procedure to vary model size across all supported methods while remaining data-free and tuning-free. The manuscript lacks an explicit equation, pseudocode, or unified size parameter (e.g., a single retention ratio or interpolation schedule) demonstrating that size variation is implemented without algorithm-specific adaptations; this makes the reported overtaking behavior potentially artifactual rather than intrinsic.
- [§4.1] §4.1 and Table 2 (Experimental results): The reported accuracy gains of up to 13.5% upon doubling size are load-bearing for the first key finding, yet the text does not specify the exact size metric (parameter count vs. effective model count) or confirm that the same size-control rule is applied uniformly across benchmarks and algorithms; without these details, it is difficult to rule out post-hoc choices affecting the gains and ranking shifts.
minor comments (2)
- [Figure 3] Figure 3: The x-axis labeling for model size is ambiguous (number of retained models vs. total parameters); adding a secondary axis or explicit legend would improve clarity.
- [§2] §2 (Related work): A brief comparison table summarizing how FlexMerge differs from prior merging frameworks in terms of size flexibility would help readers quickly grasp the novelty.
Simulated Author's Rebuttal
We thank the referee for the detailed and constructive comments on our manuscript. We address each of the major comments point by point below, providing clarifications and committing to revisions where appropriate to strengthen the presentation of FlexMerge.
read point-by-point responses
-
Referee: [§3.2] §3.2 (Size-control mechanism): The central claim that algorithm rankings are inconsistent as size increases depends on FlexMerge applying an identical, neutral procedure to vary model size across all supported methods while remaining data-free and tuning-free. The manuscript lacks an explicit equation, pseudocode, or unified size parameter (e.g., a single retention ratio or interpolation schedule) demonstrating that size variation is implemented without algorithm-specific adaptations; this makes the reported overtaking behavior potentially artifactual rather than intrinsic.
Authors: We appreciate the referee pointing out the need for greater explicitness in describing the size-control mechanism. Section 3.2 of the manuscript introduces a unified size parameter, which we term the 'retention ratio' (denoted as ρ), that controls the proportion of parameters retained from the fine-tuned models in the merged output. This parameter is applied in an identical manner to all merging algorithms supported by FlexMerge (e.g., TIES, DARE) by modulating the task vector contributions or parameter selection thresholds proportionally to ρ, without any algorithm-specific modifications. The procedure remains entirely data-free and tuning-free. To make this clearer and address the concern directly, we will add an explicit equation defining the size variation and pseudocode illustrating the unified procedure in the revised manuscript. revision: yes
-
Referee: [§4.1] §4.1 and Table 2 (Experimental results): The reported accuracy gains of up to 13.5% upon doubling size are load-bearing for the first key finding, yet the text does not specify the exact size metric (parameter count vs. effective model count) or confirm that the same size-control rule is applied uniformly across benchmarks and algorithms; without these details, it is difficult to rule out post-hoc choices affecting the gains and ranking shifts.
Authors: We agree that specifying the size metric and confirming uniformity is essential for reproducibility and to support the claims. Throughout the experiments, the model size is quantified using the effective model count, which is directly determined by the retention ratio ρ (e.g., doubling the size corresponds to increasing ρ such that the merged model effectively incorporates parameters equivalent to two full models). This size-control rule is applied uniformly across all vision and NLP benchmarks and all algorithms, as outlined in the experimental protocol in Section 4. We will revise Section 4.1 and the caption of Table 2 to explicitly define the size metric as the effective model count and reaffirm the uniform application of the size-control mechanism. revision: yes
Circularity Check
No circularity: empirical results from unified framework on benchmarks
full rationale
The paper introduces FlexMerge as a data-free framework supporting multiple merging algorithms and variable model sizes, then reports accuracy gains and ranking shifts as direct outcomes of experiments on vision and NLP benchmarks with up to 30 tasks. No derivation chain reduces a claimed prediction to a fitted parameter or self-citation by construction; the size-control and algorithm support are presented as implementation choices whose effects are measured externally rather than defined tautologically. The central findings (up to 13.5% gains on doubling size, inconsistent rankings) are falsifiable via the reported benchmarks and do not rely on load-bearing self-citations or ansatzes smuggled from prior work.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.