{"id":"6acb6a38-9adb-43c6-a708-e63ff8540e00","arxiv_id":"2504.18580","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Merging LoRA checkpoints with weights based on inverse training loss, plus a rank-based penalty, beats uniform averaging on three fine-tuning tasks.","lead":"This paper proposes a simple way to merge several saved checkpoints of a fine-tuned language model into one better model by giving more weight to checkpoints with lower training loss or later training steps. A generalist reader might care because it is a nearly free post-training trick that often beats averaging all checkpoints equally and sometimes beats the final checkpoint.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains are selected from a penalty-factor sweep on the test benchmark; a held-out validation split could show the advantage over uniform merging vanishes.","rationale":"The reader identifies the reliability of loss as a signal as the weakest assumption. I agree that this is a component, but the more critical link in the argument is the evaluation protocol: the reported numbers are the best of many hyperparameter configurations selected on the same test set. Even if loss is a perfect ranking signal, the claim that MWA 'consistently' beats uniform requires that the advantage is robust to choosing the penalty factor without peeking at the test benchmark. The paper's merging steps explicitly implement test-set selection, making the numerical improvements untrustworthy as evidence for generalizable superiority. The concrete test of a validation split would directly settle this: if the improvement persists under a proper selection protocol, the concern is resolved; if not, the paper's contribution reduces to a new weighting formula whose practical benefit is not demonstrated. I therefore keep the CONDITIONAL verdict, matching the reader's overall assessment, but the condition must include a validation-based selection.","tokens_in":142,"tokens_out":5818,"duration_ms":57750,"concrete_test":"Perform a 50/50 random split of the evaluation examples for GSM8K and GSM-Plus (for the math experiment). Using only the first half as a validation set, follow the Section 5.1 merging steps: shortlist uniform soups, then for each shortlisted soup vary p and select the best p on validation accuracy. Evaluate the resulting MWA model on the held-out second half, and compare to the uniform soup chosen on validation in the same way. Repeat for 10 random splits. If the median validation-selected MWA test accuracy is not higher than the corresponding uniform soup test accuracy (or the 95% bootstrap confidence interval of the difference includes 0), the claimed 5.05% improvement is a selection artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 describes a model-selection procedure in which uniform soups are ranked on the evaluation benchmark, the top 1-2 are shortlisted, and then penalty factors are varied to produce MWA soups; the best results (Tables 2-3) are then reported as evidence of consistent superiority. This is a form of test-set overfitting: every choice of k, interval spacing, and p is evaluated on the same benchmark used to claim success, with no held-out validation, multiple random seeds (only one training run per task), or error bars. For the math task, the best MWA model (last_10_loss_pf-0_7) differs from the final checkpoint by an absolute weighted-average gain of 0.004 (0.22826 vs 0.22431), which on GSM8K/GSM-Plus corresponds to roughly a handful of examples and is within the noise of a single evaluation. A sweep over several p values (e.g., 0.5, 0.7, 0.8, 1.0, 1.05) makes an apparent winner almost inevitable. The paper also concedes in Section 5.5 that pairwise MWA often underperforms uniform, contradicting the abstract's 'consistently outperforms'. Without a validation-based selection protocol or significance testing, the central claim that MWA reliably beats uniform averaging is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Metrics-Weighted Averaging (MWA), a heuristic for merging LoRA checkpoints taken from a single fine-tuning run. Checkpoints are weighted by the reciprocal of their training loss (loss-weighted) or by their step count (steps-weighted), with an additional rank-based penalty factor p applied to control the weight spread. Experiments cover three tasks: math SFT (GSM8K/GSM-Plus), alignment via SimPO (ToxiGen/TruthfulQA), and instruction tuning (OpenHermes-2.5 validation loss). The authors report that MWA often beats uniform averaging of the same checkpoints and sometimes beats the final individual checkpoint, with the best math gain stated as +5.05% on the GSM weighted average. They also compare MWA against Ties, Dare-Ties, and SLERP, claiming relative superiority on domain-specific benchmarks, and they discuss limitations and cases where MWA underperforms uniform merging.","tokens_in":14496,"tokens_out":3556,"duration_ms":34454,"significance":"If the claims are supported, MWA would be a simple, low-overhead addition to PEFT checkpoint merging: it requires only one tunable hyperparameter per merge and uses metrics that are already recorded during training. The paper covers three distinct post-training scenarios, includes baseline methods (Ties, Dare-Ties, SLERP), and is candid in Section 5.5 about pairwise merges often underperforming uniform averaging. However, the significance is currently limited by the evaluation protocol: the penalty factor p is selected on the same benchmarks used to report success, all results come from single runs without error bars, and the baseline comparison in Section 5.4 is weakened by an unsubstantiated hardware-difference explanation. The core idea is plausible, but the evidence as presented does not yet support the abstract's claim of 'consistently' outperforming uniform averaging.","major_comments":[{"comment":"The model-selection procedure in Section 5.1 evaluates uniform soups on the target benchmark, shortlists the top 1-2, and then varies the penalty factor p to produce MWA soups, with the best results reported in Tables 2-3. Because p is chosen by maximizing the same GSM, alignment, or validation-loss metric used to claim success, and no held-out validation split is used, the reported gains (e.g., +5.05% in Figure 1) are selected maxima over a small sweep rather than unbiased estimates of MWA's advantage. To support the central claim, the paper should report performance on a held-out validation set or use a nested selection procedure, and should show results for all p values swept.","section":"Section 5.1 and Tables 2-3"},{"comment":"In the math benchmark, the raw baseline results are higher than the best MWA results (e.g., dare_ties_last_3_base_first achieves 0.2295 vs. 0.22826 for last_10_loss_pf-0_7), but the text dismisses this with the statement that 'the two runs used different underlying hardware accelerators' without providing any controlled experiment to support that claim. Since the base-model scores also differ between the two runs (gemma-2b 0.1251 vs. 0.12281 in Tables 2 and 4), the relative comparison does not establish that MWA is superior to Ties, Dare-Ties, or SLERP. The comparison should be conducted under identical hardware and base-model conditions, or the claim of superiority should be withdrawn.","section":"Section 5.4 and Table 4"},{"comment":"The abstract states that MWA 'consistently produces merged models that outperform the naive uniform average of checkpoints,' but Section 5.5 explicitly says that for pairwise merges MWA 'marginally outperforms the Uniform Soup equivalent at best, and often underperforms the latter.' The claim of consistency is therefore too broad; either the abstract and conclusion should be scoped to merges of 4-10 checkpoints, or the pairwise underperformance should be reported as a caveat in the abstract.","section":"Section 5.5 and Abstract"},{"comment":"All reported results are single runs without error bars or significance tests. Some headline improvements are small in absolute terms: in Table 2, last_10_loss_pf-0_7 has weighted average 0.22826 versus 0.22431 for the final checkpoint, a difference of about 0.004 that on GSM8K/GSM-Plus corresponds to a handful of examples and may lie within evaluation noise. Without multiple training seeds or repeated evaluations, the improvements cannot be distinguished from random variation, especially when p is selected post hoc on the same benchmark.","section":"Tables 2-3 and Section 5.2"}],"minor_comments":[{"comment":"The power factor q is said to be 'fixed heuristically based on the distribution of checkpoint metrics values,' but the paper never reports the q values used in any experiment or describes the heuristic concretely. Please state the chosen q values and the rationale.","section":"Section 3.1"},{"comment":"The figure captions define 'baseline' as the last merging checkpoint, while the abstract and text use 'uniform soup' as the baseline. Please clarify which baseline is shown in each figure to avoid inconsistent references.","section":"Figure 1 and Figure 2 captions"},{"comment":"In Figure 3, validation-loss values are labeled only to two decimal places (e.g., 17.78 vs. 17.82), but the differences discussed are as small as 0.02, making the order of bars visually ambiguous. A table or more decimal places would make the results clearer.","section":"Figure 3 and Section 5.3"},{"comment":"The footnote states 'All models at or below the highlighted model show validation losses lower than those of both the final checkpoint and the last merging checkpoint,' but several models above the highlighted row (e.g., ties_last_3_base_last at 17.4391) also have lower validation loss than 17.4189, so the statement is inaccurate as written.","section":"Table 6 footnote"},{"comment":"There are several typographical errors, including 'seection' in Section 3, 'afects' in Section 3.1, and 'simpo_lora_hh-rlhf' with inconsistent hyphenation. The LoRA acronym is also written as both 'LoRA' and 'LoRa'; please standardize.","section":"General presentation"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a simple and potentially useful heuristic, and the authors are transparent about several limitations, which is appreciated. However, the evaluation protocol has a serious selection-on-test-set issue: the penalty factor and checkpoint-count choices are optimized on the same metrics used for the headline numbers, without held-out validation or repeated runs. The central claim of consistent superiority over uniform averaging is not yet supported, and the Section 5.4 baseline comparison is unconvincing because of the unverified hardware explanation. These issues are fixable within the manuscript's scope, but they are load-bearing for the paper's main contribution, so I recommend major revision rather than rejection. If the authors can provide held-out validation, report error bars, and either strengthen or temper the baseline claims, the paper could become a reasonable empirical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nPunchline: this is a modest but legitimate contribution — a one-hyperparameter weighting scheme for merging LoRA checkpoints from a single run — and the paper's central idea is plausible. But the empirical support as written is not as strong as the abstract claims. The gains are largely selected from sweeping the penalty factor on the same benchmarks used to report success.\n\nWhat's new: the specific formula (inverse-loss or step weighting with a rank-based penalty p^pos) applied to PEFT checkpoint merging is not in the cited literature. Prior checkpoint merging work uses Bayesian optimization; weighted averaging methods like Fisher averaging target different settings. The paper does well to test on three distinct post-training tasks (math SFT, SimPO alignment, instruction tuning) and to include TIES/Dare-TIES/SLERP baselines, even if those runs were on different hardware and only relative comparisons are meaningful.\n\nSoft spots: the main problem is selection. Steps 2-5 of the merging protocol shortlist uniform soups on the evaluation benchmark and then vary penalty factors to produce MWA soups; the best result is reported. With several p values and no held-out validation, an apparent winner is close to inevitable. The math task's best MWA gain over the final checkpoint is 0.004 weighted average — a handful of GSM-Plus examples, plausibly within evaluation noise. There are no error bars, no multiple seeds, and no code or data. The abstract's \"consistently outperforms\" is also contradicted by the paper's own Section 5.5 note that pairwise MWA often underperforms uniform averaging. On circularity, I'd go a bit easier than the stress-test note: the weighting formula is not fitted to the benchmark, and the loss-as-quality premise is reasonable for late-stage checkpoints, but the p-selection procedure still inflates the headline numbers. The baseline comparison is weakened by running on different hardware; the paper is transparent about this, but it limits the comparison.\n\nWho it's for: practitioners running LoRA fine-tunes who want a cheap merging heuristic, and anyone who wants a cautionary example of test-set selection in model-soup papers. It deserves serious peer review, but only with a validation-split selection protocol, multiple seeds, and a toned-down claim.\n\nRecommendation: send to review; expect major revision. The core idea may well survive a fairer evaluation.","headline":"A simple, plausible LoRA checkpoint-merging heuristic whose headline gains are partly a product of penalty-factor selection on the test benchmark; weaker than claimed, but worth a serious look.","tokens_in":15053,"tokens_out":2622,"would_cite":false,"duration_ms":24104,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Metrics-Weighted Averaging weights saved LoRA checkpoints by their training loss, and the merged model beats uniform averaging — and often the final checkpoint — by up to 5.05% relative on the GSM benchmark.","keywords":["checkpoint merging","model merging","metrics-weighted averaging","loss-weighted averaging","LoRA","parameter-efficient fine-tuning","model soup","large language models"],"falsifier":"Score every saved checkpoint on the downstream benchmark (e.g., the GSM weighted average) for a fresh run of the same three training recipes. If the inverse training loss ranks checkpoints differently from the benchmark scores, or if the best loss-weighted soup found by sweeping the penalty factor does not beat the uniform soup of the same checkpoints and the final checkpoint, then the central claim is falsified. The paper reports merged-model scores but not per-checkpoint benchmark scores, so this check is open.","tokens_in":14024,"feed_emoji":"⚖️","tokens_out":12025,"duration_ms":102310,"temperature":0.7,"pith_summary":"Checkpoint merging normally averages fully trained models, but this paper asks whether cheaply weighted averages of LoRA adapters from a single fine-tuning run can do better. The proposed method, Metrics-Weighted Averaging (MWA), weights each checkpoint by its training loss (or step count), shaped by a single penalty factor, and averages the adapter parameters. Across math reasoning, preference alignment, and instruction tuning, loss-weighted merges beat the uniform average of the same checkpoints and, on the math and alignment runs, also outperform the final individual checkpoint, with up to 5.05% relative gain on the GSM weighted benchmark. If correct, the result means saved checkpoints plus their loss curves are enough to squeeze extra accuracy out of a fine-tuning run at near-zero extra compute.","feed_headline":"Loss-weighted checkpoint merging beats uniform averaging","feed_subtitle":"A one-knob formula turns saved LoRA checkpoints into a better model, no retraining needed.","key_machinery":"The load-bearing object is the rank-shaped weighting formula $\\varphi_x = p^{q \\cdot \\mathrm{pos}(\\ell_x)} \\cdot \\ell_x^{-1}$ for loss (or $\\varphi_x = p^{q \\cdot \\mathrm{pos}(s_x)} \\cdot s_x$ for steps), normalized by the sum over checkpoints. The penalty factor $p$ controls how fast weight decays with a checkpoint's rank, while the power factor $q$ sets the steepness of the decay; because raw losses differ by only small amounts late in training, the pure reciprocal-loss weights would be nearly uniform, and the rank term is what spreads them out. The same formula is presented as metric-agnostic through a min/max dichotomy, meaning any cheap signal — lower-is-better or higher-is-better — can drive the weights with the same single knob.","core_discovery":"The paper's central claim is that the relative value of a checkpoint for merging can be read off a cheap training signal, and that a geometric penalty on the checkpoint's rank turns that signal into weights that beat naively averaging the same checkpoints. Concretely, for checkpoints with losses $\\ell_x$, MWA sets the unnormalized weight $\\varphi_x = p^{q \\cdot \\mathrm{pos}(\\ell_x)} / \\ell_x$, where $\\mathrm{pos}(\\ell_x)$ is the zero-indexed rank from lowest to highest loss, $p<1$ is the single tuned penalty factor, and $q>1$ is a fixed power factor; steps weighting uses $\\varphi_x = p^{q \\cdot \\mathrm{pos}(s_x)} \\cdot s_x$ instead. After normalization, the LoRA adapters are averaged with these weights. Across the three runs the best models come from loss-weighted merges of 4-10 checkpoints, including interval-spaced ones; the best math result improves the GSM weighted average by 5.05% over the last merging checkpoint and 1.76% over the final checkpoint. The paper further claims that this one-knob procedure outperforms uniform averaging and, measured relative to each run's own reference checkpoints, outperforms the TIES, DARE-TIES, and SLERP baselines on domain benchmarks, while challenging the earlier finding that only pairwise or adjacent checkpoint merges help.","pith_inferences":["The penalty factor $p$ interpolates between a nearly uniform soup and a near-one-hot selection of the best-loss checkpoint, so part of MWA's gain is likely variance reduction or implicit early stopping rather than fine-grained weighting; an ablation that fixes the selected subset and varies only the weights would separate the two effects.","Because the formula needs only a loss curve, the same merging trick should transfer to full fine-tuning and pretraining checkpoints, which the paper did not test; if the loss-to-quality link holds there, pretraining runs could harvest their routinely saved checkpoints the same way.","A cheap held-out validation loss would be a drop-in 'min' metric that may rank checkpoints more faithfully than training loss, especially late in training when training-loss differences are tiny; testing that variant is the most direct next experiment."],"forward_implications":["A fine-tuning run that already logs training loss can produce a better final model without retraining: merging the last 4-10 saved checkpoints with loss-derived weights is the recipe.","The cost of choosing merge weights stops growing with the number of checkpoints: instead of tuning $k-1$ weights, one penalty factor $p$ is swept, so the heuristic's advantage grows as the soup gets larger.","Merging checkpoints that are spaced apart (interval merging) can beat merging adjacent ones, and merges of 4-10 checkpoints beat pairwise merges, contradicting earlier checkpoint-merging conclusions.","The choice of metric decides the outcome: loss-weighted merging beat steps-weighted merging in all three experiments, so the practical rule is to weight by the signal that best tracks downstream quality.","The same weighting formula applies to any lower-is-better or higher-is-better metric, so other cheap training signals can be dropped into the identical one-knob procedure."],"supporting_citations":[{"why":"Supplies the Uniform Soup baseline that MWA must outperform and defines weight-averaging of fine-tuned models.","marker":"[Wortsman et al., 2022]"},{"why":"Defines LoRA, the parameter-efficient adapters whose checkpoints the paper merges.","marker":"[Hu et al., 2021]"},{"why":"The prior checkpoint-merging study whose conclusions (pairwise and adjacent merges best) MWA's results contradict.","marker":"[Liu et al., 2024a]"},{"why":"Provides TIES-Merging, one of the three baseline merging methods in the comparative study.","marker":"[Yadav et al., 2023]"},{"why":"Orca-Math is the training dataset for the math fine-tuning run that produces the headline 5.05% result.","marker":"[Mitra et al., 2024]"},{"why":"SimPO is the preference-optimization method that generates the alignment checkpoints for the second experiment.","marker":"[Meng et al., 2024]"},{"why":"GSM8k is one of the two math benchmarks whose weighted average reports the headline numbers.","marker":"[Cobbe et al., 2021]"},{"why":"GSM-Plus carries 70% of the weighted math benchmark and drives most of the measured improvement.","marker":"[Li et al., 2024]"}],"fun_headline_variants":["Loss-weighted merge beats uniform checkpoints","Metric-weighted averaging outperforms uniform checkpoint merging","One weight knob turns saved LoRA checkpoints into better model","Loss-weighted averaging lifts accuracy over uniform merge","Checkpoint merging: loss-based weights beat simple average"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole scheme assumes training loss is a faithful ranking of a checkpoint's value for merging: if a lower-loss checkpoint is not actually more useful inside the average, the weights are arbitrary.","fun_headline_variants_meta":{"raw":{"variants":["Loss-weighted merge beats uniform checkpoints","Metric-weighted averaging outperforms uniform checkpoint merging","One weight knob turns saved LoRA checkpoints into better model","Loss-weighted averaging lifts accuracy over uniform merge","Checkpoint merging: loss-based weights beat simple average"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001652,"raw_usage":{"total_tokens":6613,"prompt_tokens":1049,"completion_tokens":5564,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":5507}},"tokens_in":665,"tokens_out":5564,"duration_ms":33333,"temperature":1.0,"reasoning_tokens":5507,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:03:24.421405+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score every saved checkpoint on the downstream benchmark (e.g., the GSM weighted average) for a fresh run of the same three training recipes. If the inverse training loss ranks checkpoints differently from the benchmark scores, or if the best loss-weighted soup found by sweeping the penalty factor does not beat the uniform soup of the same checkpoints and the final checkpoint, then the central claim is falsified. The paper reports merged-model scores but not per-checkpoint benchmark scores, so this check is open.","supporting_citations":[{"cited_title":"Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time","cited_arxiv_id":null,"evidence_quote":"Supplies the Uniform Soup baseline that MWA must outperform and defines weight-averaging of fine-tuned models."}],"review_version":1}