Pith. sign in

REVIEW 3 major objections 6 minor 17 references

The optimal training duration for LLM experts before merging is method-dependent: sparsification-based methods peak with overfitted experts while simple averaging degrades sharply.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:45 UTC pith:BRCTF5OO

load-bearing objection Sparsification-based merging may benefit from overtrained experts, but the single-seed results don't yet quantify the gains; worth a serious referee. the 3 major comments →

arxiv 2607.11997 v2 pith:BRCTF5OO submitted 2026-07-13 cs.LG cs.AIstat.ML

Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs

Nikita Kozodoi , Zainab Afolabi , Jack Butler This is my paper
classification cs.LG cs.AIstat.ML
keywords model mergingexpert training durationearly stoppingsparsificationTIESDAREbias-variance decompositionLoRA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether the common practice of fine-tuning each expert model to its individual validation loss minimum before merging is actually the right stopping point. Across three model sizes, five domains, and five merging methods, it finds the answer is no: the optimal training duration depends on the merging method. Simple weight averaging peaks with undertrained experts and degrades sharply if experts overfit, while sparsification-based methods such as TIES and DARE+TIES keep improving when experts are trained to 2–5 times their validation optimum. The paper explains the pattern through a bias-variance decomposition of merged-model error, arguing that sparsification removes the noise that overfitting adds while preserving the lower-bias task knowledge. If correct, the result changes a standard design choice in multi-task model merging: training duration and merging method should be selected together.

Core claim

The paper's central claim is that expert training duration and merging method are coupled, not independent choices. Empirically, overfitted experts (2–5× the per-expert validation-optimal step T*) yield the best merged models under sparsification-based methods, while simple averaging performs best in the undertrained regime (0.25–1× T*) and loses 14–23 points by 5× T*. The mechanism proposed is that overtraining lowers bias (experts capture more task-specific signal) but raises variance and inter-expert covariance; methods like TIES and DARE+TIES filter out exactly this added noise through trimming, sign resolution, and stochastic pruning, thereby converting high-variance experts into a bett

What carries the argument

The bias-variance-covariance decomposition of merged-model error, adapted from weight-averaging theory, is the central lens: as training passes T*, bias falls while variance and covariance rise. Sparsification-based merging methods (TIES, DARE+TIES) act as variance-reduction operators that remove small-magnitude or sign-conflicting parameter updates before averaging, analogous to how averaging in random forests tolerates deeper, higher-variance trees. This decomposition, together with linear mode connectivity measurements, explains why the optimal training duration shifts with the merging method.

Load-bearing premise

The central pattern is empirical, and every result comes from a single training run per configuration with no error bars, so the claim assumes the reported gaps are larger than run-to-run noise.

What would settle it

Run the same experiment with at least five seeds per configuration and compare TIES' merged accuracy at T* against its accuracy at 2–3× T*; if the mean difference is within one standard deviation of the seed spread, the method-dependent pattern would collapse.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If a practitioner uses TIES or DARE+TIES, they should deliberately train each expert past its validation optimum (2–5× T* for the tested family) instead of early stopping.
  • If using simple averaging, experts should be undertrained (0.25–1× T*) to avoid degrading the merged model.
  • The choice of training duration and merging method should be optimized jointly, not sequentially.
  • Sparsification's variance-reduction role gives a practical rule of thumb: the stronger the merge's interference resolution, the more overfitting the experts can tolerate.
  • The observed rightward shift of the optimum with model size suggests larger backbones reward even more aggressive overtraining for sparsification methods.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the single-seed limitation is addressed and the pattern holds, this would reconcile conflicting reports in the merging literature about whether overtraining helps: the answer may be method-dependent rather than universal.
  • The random-forest analogy suggests a testable prediction: merging methods with stronger variance reduction (e.g., lower density k) should shift the optimal training duration even further past T*.
  • The results imply that benchmark comparisons of merging methods may be unfair if all experts are at their individual T*; each method should be evaluated at its own optimal duration.
  • The reliance on a constant learning rate may amplify the overtraining regime; under learning-rate decay, the optimal overtraining multiple could shrink, a question the paper itself leaves open.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies how the training duration of domain experts (relative to their individual validation optimum T*) affects the quality of merged multi-task LLMs. The authors fine-tune Qwen 3.5 models at three scales (0.8B, 2B, 4B) on five domains using QLoRA, save checkpoints from 0.25× to 5×T*, and evaluate five merging methods (Simple Averaging, Task Arithmetic, TIES, DARE+TIES, Greedy Soup). They report a method-dependent pattern: Simple Averaging peaks with undertrained experts, Task Arithmetic/Greedy Soup are intermediate, and sparsification-based methods (TIES, DARE+TIES) peak with overtrained experts. They explain this via a bias-variance-covariance decomposition and linear mode connectivity, drawing an analogy to random forests, and recommend training experts 2–5×T* when using sparsification-based merging.

Significance. If the empirical pattern holds, the paper challenges a widespread convention — merging experts at their individual validation optima — and offers concrete practical guidance for choosing expert training duration together with the merging method. The study is systematic in covering multiple model sizes, domains, and merging methods, and it transparently acknowledges several limitations (single model family, QLoRA/rank r=16, constant LR, single seed for stochastic methods). The random-forest analogy is attractive and the bias-variance account is a plausible explanation. However, the central claim is entirely empirical and currently rests on single training runs without error bars; moreover, the specific quantitative recommendation (2–5×T*) is not consistently supported by the Appendix tables. The paper's significance will depend on whether the key differences are statistically robust and whether the practical guidance is stated accurately.

major comments (3)
  1. [§6, Tables 7–9] The central empirical claim — TIES/DARE+TIES peak after T* with gains of 2.5–8.7 points, while Simple Averaging degrades by 14–23 points — is based on a single training run per configuration; Greedy Soup and DARE's stochastic pruning use a single seed. Since the 'best duration' is selected as the maximum over 8 noisy checkpoints, the apparent peak is inflated by selection noise even if the true curve is flat. Section 6 acknowledges the single-seed limitation but does not quantify its impact. Please report mean±std over at least 3 seeds for a representative subset (e.g., 4B, all methods at 1.0/1.5/2.0/3.0/5.0×T*) and/or paired significance tests. Without this, the reported method-dependent pattern could be within run-to-run noise.
  2. [§5.1, Figure 2] The conclusion that 'sparsification-based methods (TIES, DARE+TIES) achieve peak performance with deliberately overfitted experts (2–5 × T*)' is contradicted by the Appendix's per-domain optimal-duration tables. For example, DARE+TIES achieves its best Instruction score at 0.25×T* for Qwen3.5-2B (Table 8), at 1.5×T* for Safety on 0.8B (Table 7), and TIES is best at 1.0×T* for 2B Safety (Table 8). The overfitted-expert pattern holds on average and for many accuracy domains, but the stated 2–5×T* range is too broad. Please qualify the practical guidance to the domains/scales where it is supported, or explain why these exceptions are consistent with the claimed mechanism.
  3. [Appendix A.1] The Abstract and Contributions state that the results are 'formalized through bias-variance decomposition,' but Section 5.1 gives no equations, no explicit estimators for bias/variance/covariance of a merged weight-space model, and no formal link between the decomposition and the merging methods. Section 6 later concedes this is 'an explanatory hypothesis rather than a proven mechanism.' Either provide the actual decomposition (definitions, assumptions, how it is computed from the fine-tuned models) or soften the 'formalize' claim. The random-forest analogy is useful as intuition, but as written the formalization is not delivered.
minor comments (6)
  1. [Table 5] The sentence 'For every method and size, the best training duration outperforms T*' is tautological, since T* is one of the evaluated checkpoints. Replace with a meaningful comparison, e.g., the gain of the best checkpoint over T* with a significance test or confidence interval.
  2. [Figure 1] The 'Base (no FT)' row for Qwen3.5-4B Code is reported as '–'. If the base model was not evaluated on HumanEval at that scale, state this explicitly; otherwise fill the missing value.
  3. [§4.1] The caption says 'Optimal T steps is shown as method-specific,' which is unclear. T* is defined per domain and model size in Section 3; clarify what 'method-specific' means here.
  4. [§5.1] The description of Greedy Soup as showing 'a more balanced selection at 4B' is vague. Quantify the selected subsets (e.g., average number of experts retained) if this is meant to support the interpretation.
  5. [References] Figure 2 would be more informative with axis labels/units and a brief description of how each component is computed; currently it is hard to verify the claimed decrease in bias and increase in variance/covariance.
  6. Several references are to works dated 2026 (Tran et al., Rofin et al.). Please verify these are publicly available or indicate preprint status correctly.

Circularity Check

1 steps flagged

No material circularity: the main claim is an empirical grid-search observation; the only tautological statement is a minor appendix claim that the best checkpoint outperforms T* by construction.

specific steps
  1. self definitional [Appendix A.1 (Main results tables)]
    "For every method and size, the best training duration outperforms T*."

    The 'best training duration' is defined as the maximum score across the eight checkpoints {0.25, 0.5, 0.75, 1.0, 1.5, 2.0, 3.0, 5.0} x T*, and this set includes T* itself (1.0 x T*). Therefore the maximum over the set is at least the score at T* by construction. The sentence carries no empirical evidence about whether overtraining helps; it is true even if T* were the best duration. This statement is in the appendix and is not load-bearing for the paper's central empirical claim, which depends on where the maximum occurs, not on the fact that the maximum exceeds T*.

full rationale

The central claim is an empirical observation from a systematic grid over training durations and merging methods, not a derived prediction. The paper does not fit a parameter and then predict a closely related quantity: the only tuned hyperparameter, Task Arithmetic's lambda, is selected by grid search and does not determine the duration-dependent pattern. T* is defined objectively per expert as the validation-loss minimum, and the merged-model scores are measured independently. The bias-variance-covariance discussion is explicitly presented as an explanatory hypothesis ('an explanatory hypothesis rather than a proven mechanism') and is not used to fit the reported numbers, so it cannot make the conclusion tautological. There are no load-bearing self-citations: the authors do not cite their own prior work to justify the central premise, and external references (MergeBench, DiWA, Horoi et al.) are either benchmarks or independently falsifiable related work. The lack of multiple seeds and error bars, acknowledged in Section 6, is a statistical robustness limitation rather than a circularity: it affects whether the observed peaks are real, but it does not reduce the conclusions to their inputs. The only circular statement found is the appendix line noting that the best checkpoint outperforms T*, which is true by construction because T* is in the candidate set; it is isolated, non-central, and does not affect the main finding that sparsification methods' maxima occur at overtrained checkpoints.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The paper introduces no new theoretical entities. Its results rest on standard merging methods, benchmarks, and the DiWA framework. The main unquantified load is the grid-selected lambda and the reliance on a single model family and fine-tuning recipe.

free parameters (1)
  • Task Arithmetic scale λ = selected from {0.5, 1.0, 1.5} via grid search
    The paper selects λ per model/domain/duration based on validation scores. This data-dependent choice could influence the duration-method interaction, though the reported pattern for Task Arithmetic is not solely determined by it.
axioms (4)
  • domain assumption Qwen 3.5 models fine-tuned with QLoRA are representative of LLM merging behavior
    Results are only shown for one model family and one PEFT recipe; the authors explicitly note this limitation in the conclusion.
  • domain assumption Subsampled training/validation sets (10K/1K examples) preserve the domain's overfitting characteristics
    Datasets are subsampled from their full versions; this may alter validation loss curves and the sharpness of overfitting.
  • domain assumption The bias-variance-covariance decomposition of DiWA applies to weight-averaged merged models
    The paper applies Rame et al. (2022) to interpret merged model error, but does not derive or verify this decomposition for the merging setting.
  • domain assumption The aggregated average of accuracy across four heterogeneous benchmarks is a meaningful objective
    The paper averages exact-match, pass@1, prompt-level accuracy, and normalized accuracy without weighting or variance normalization, which could affect the apparent pattern.

pith-pipeline@v1.3.0-alltime-deepseek · 245 in / 7624 out tokens · 97587 ms · 2026-08-02T06:45:50.095323+00:00 · methodology

0 comments
read the original abstract

Multi-task model merging combines separately trained expert models into a single model that handles all tasks without co-training. Standard practice merges experts at their optimal validation loss. We challenge this convention by systematically studying how training duration of domain experts affects the quality of the merged model. We fine-tune experts on five domains (Math, Code, Instruction Following, Multilingual, and Safety) across three model sizes (Qwen 3.5 0.8B, 2B, and 4B), saving checkpoints from 25% to 500% of the optimal training steps and evaluating five merging methods at each duration. Our findings reveal a striking method-dependent pattern: simple averaging degrades sharply with overfitting, while sparsification-based methods achieve their best performance well past the validation optimum. We formalize this through bias-variance decomposition analysis, drawing a parallel to random forests where averaging benefits from high-variance individual learners. These results suggest that training duration and merging method should be chosen jointly rather than independently.

Figures

Figures reproduced from arXiv: 2607.11997 by Jack Butler, Nikita Kozodoi, Zainab Afolabi.

Figure 1
Figure 1. Figure 1: Effect of training duration on merged model performance. Each line represents one merging method; the x-axis shows training steps as a multiple of T* (optimal validation checkpoint per expert). Green shading indicates the undertrained regime (<T*); red indicates the overfitted regime (>T*). Results are averaged over the four accuracy-based domains. Optimal T steps is shown as method-specific. 4.2. Training… view at source ↗
Figure 2
Figure 2. Figure 2: Bias-variance-covariance decomposition of the merged model error against training duration, shown across all three model sizes. As training increases past T*, bias drops while variance and covariance rise [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Mode connectivity: average loss barrier between expert pairs vs. training duration. The barrier grows monotonically with training duration but remains moderate even at 5× T*. 5.2. Mode connectivity We evaluate linear mode connectivity between expert pairs to assess whether overfitted experts remain in the same loss basin. For each pair (θA, θB), we interpolate θ(α) = αθA + (1−α)θB for α ∈ [0, 1] and measur… view at source ↗
Figure 4
Figure 4. Figure 4: decomposes the aggregate trends from [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 15 linked inside Pith

  1. [1]

    A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al

    Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: a highly capable lan- guage model locally on your phone.arXiv preprint arXiv:2404.14219,

    Pith/arXiv arXiv

  2. [6]

    He, T. et al. MergeBench: a comprehensive bench- mark for merging in foundation models.arXiv preprint arXiv:2505.10833,

    arXiv

  3. [9]

    Magicoder: empowering code generation with OSS-instruct.arXiv preprint arXiv:2312.02120,

    Luo, Y ., Xu, C., Zhao, P., Sun, R., Zhu, J., Xiao, T., et al. Magicoder: empowering code generation with OSS-instruct.arXiv preprint arXiv:2312.02120,

    Pith/arXiv arXiv

  4. [10]

    Harm- Bench: a standardized evaluation framework for auto- mated red teaming and robust refusal.arXiv preprint arXiv:2402.04249,

    Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., et al. Harm- Bench: a standardized evaluation framework for auto- mated red teaming and robust refusal.arXiv preprint arXiv:2402.04249,

    Pith/arXiv arXiv

  5. [11]

    Ranjan, R., Garg, S., Raman, M., Guestrin, C., and Lip- ton, Z. C. Post-hoc reversal: are we selecting models prematurely?arXiv preprint arXiv:2404.07815,

    Pith/arXiv arXiv

  6. [12]

    (How) learning rates regulate catastrophic overtraining.arXiv preprint arXiv:2604.13627,

    Rofin, M., Varre, A., and Flammarion, N. (How) learning rates regulate catastrophic overtraining.arXiv preprint arXiv:2604.13627,

    Pith/arXiv arXiv

  7. [14]

    Leveraging model soups to classify ICH images from the Mekong Delta

    Tran, H., Nguyen, M., and Pham, Q. Leveraging model soups to classify ICH images from the Mekong Delta. arXiv preprint arXiv:2603.02181,

    arXiv

  8. [15]

    Wei, Y . et al. DART-Math: difficulty-aware rejection tun- ing for mathematical problem-solving.arXiv preprint arXiv:2407.13690,

    Pith/arXiv arXiv

  9. [16]

    Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911,

    Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y ., Zhou, D., and Hou, L. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911,

    Pith/arXiv arXiv

  10. [2001]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    Pith/arXiv arXiv

  11. [2018]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv

  12. [2021]

    Think you have solved question answering? try ARC, the AI2 reasoning chal- lenge.arXiv preprint arXiv:1803.05457,

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try ARC, the AI2 reasoning chal- lenge.arXiv preprint arXiv:1803.05457,

    Pith/arXiv arXiv

  13. [2022]

    Lambert, N., Morrison, J., Pyatkin, V ., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V ., et al. T¨ULU 3: pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124,

    Pith/arXiv arXiv

  14. [2023]

    S., Silvestri, F., and Rodol`a, E

    Zhou, L., Solombrino, D., Crisostomi, D., Bucarelli, M. S., Silvestri, F., and Rodol`a, E. ATM: improving model merg- ing by alternating tuning and merging.arXiv preprint arXiv:2411.03055,

    Pith/arXiv arXiv

  15. [2024]

    C., and Molchanov, P

    Belcak, P., Heinrich, G., Diao, S., Fu, Y ., Dong, X., Mu- ralidharan, S., Lin, Y . C., and Molchanov, P. Small lan- guage models are the future of agentic AI.arXiv preprint arXiv:2506.02153,

    Pith/arXiv arXiv

  16. [2025]

    Horoi, S., Wolf, G., Belilovsky, E., and Dziugaite, G. K. Less is more: undertraining experts improves model up- cycling.arXiv preprint arXiv:2506.14126,

    Pith/arXiv arXiv

  17. [2026]

    Aya dataset: an open-access collection for multilingual instruction tuning

    Singh, S., Vargus, F., Dsouza, D., et al. Aya dataset: an open-access collection for multilingual instruction tuning. arXiv preprint arXiv:2402.06619,

    Pith/arXiv arXiv