Pith. sign in

REVIEW 4 major objections 7 minor 54 references

Using Adam's second-moment statistics as a diagonal curvature proxy, OTA merging with FFG denoising surpasses TIES, linear averaging, and Fisher merging when consolidating five capability-specific SFT experts into one Llama-3.1-8B model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 16:57 UTC pith:GL2JJIS2

load-bearing objection Useful, honest merging paper whose empirical gains come mostly from FFG; the curvature-weighting story is overstated, but the method and analysis deserve a serious referee. the 4 major comments →

arxiv 2509.11167 v1 pith:GL2JJIS2 submitted 2025-09-14 cs.LG cs.AI

Harnessing Optimization Dynamics for Curvature-Informed Model Merging

Pouria Mahdavinia , Hamed Mahdavi , Niloofar Mireshghallah , Mehrdad Mahdavi This is my paper
classification cs.LG cs.AI
keywords model mergingcurvature-aware mergingFisher information approximationAdam second momenttask interferencesparsificationLLM fine-tuningoptimizer state
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that the main obstacle to merging fine-tuned models is not geometric misalignment but interference from low-saliency parameter updates, and that Adam's second-moment accumulator (exp_avg_sq) provides a free diagonal-curvature proxy for identifying and reverting those noisy updates. It introduces OTA merging: FFG sparsifies each expert's task vector by saliency score (Δw)^2·v, then curvature-weighted aggregation averages the surviving deltas. On five SFT experts (math, code, instruction following, general, knowledge) merged into Llama-3.1-8B, OTA reaches 0.582 average, vs TIES 0.565, linear 0.551, Fisher 0.541. Ablations show selection/denoising (FFG+task arithmetic = 0.560) matters more than curvature aggregation alone (0.536). Also evidence that separately fine-tuned models share nearly identical diagonal curvature, explaining why simple averaging works.

Core claim

In the paper's own terms, the central discovery is that the diagonal curvature of the loss landscape, as proxied by Adam's second-moment estimates, is highly conserved across SFT checkpoints fine-tuned on different capabilities, and that a two-stage pipeline—curvature-guided grafting (FFG) followed by curvature-preconditioned aggregation (OTA)—exploits this shared geometry to consolidate capabilities better than existing weight-space methods. The denoising step is the engine: FFG masks most of the task-vector, keeping only parameters whose squared update times second moment is high, reverting the rest to the base weights, and thus removing conflicting low-value edits.

What carries the argument

Optimization Trajectory Aware (OTA) merging (Eq. 7) combines Fast Fisher Grafting (FFG), a saliency score s_i = (Δw_i)^2 · v_i built from Adam's second-moment accumulator, with a curvature-preconditioned average using P = Diag(√v + ε). The same second-moment tensor can be compressed via a rank-one, AdaFactor-style factorization, yielding nearly storage-free curvature access while preserving most of the benefit.

Load-bearing premise

That each SFT checkpoint is perfectly calibrated: the model's predicted label distribution matches the true data distribution, so the Hessian equals the observed empirical Fisher information.

What would settle it

Compute the true diagonal Fisher information for the same checkpoints (by sampling from the model's predictive distribution and backpropagating log-likelihood), compare the resulting saliency masks and merge scores against OTA's exp_avg_sq-based results; if performance diverges substantially, the proxy's claimed equivalence is not the operative mechanism. Also, measuring calibration error on each expert and showing that merge quality collapses when calibration is poor would directly stress Assumption 2.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim is right, model merging can be performed without computing Fisher information separately; the optimizer state already contains the needed curvature signal.
  • Simple linear averaging's strong empirical performance is explained: checkpoints share similar curvature, so linear aggregation is implicitly preconditioned.
  • Memory-light rank-1 compression is sufficient: second moments have stable ranks around 1.3, enabling practical merging at a tiny storage overhead.
  • FFG's structured sparsity (early/late query/key, token embeddings) suggests task localization may guide where to interpolate or adapt in future merges.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The perfect-calibration assumption may be weaker in practice; if it fails, the saliency score is still empirically useful but not literally a Fisher-derived quantity.
  • The method could likely extend to other optimizers (e.g., LAMB) or to non-SFT adaptation such as LoRA deltas, since the second-moment accumulator is optimizer-generic.
  • The low stable rank of the second moments suggests the loss surface is extremely flat in most weight directions; combining OTA with permutation alignment might push merged-model quality even higher.
  • A testable extension is to replace exp_avg_sq with a sampled empirical Fisher or other curvature approximations and check whether the merge gains persist, which would isolate whether the proxy itself or the denoising mechanism drives the improvement.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes OTA-Merging, a two-stage recipe for merging multiple SFT checkpoints of an LLM. The method uses Adam's exp_avg_sq second-moment statistics as a 'free' diagonal Fisher/curvature proxy. Stage one, Fast Fisher Grafting (FFG), scores each task-vector coordinate by (Δw_i)^2 · v_i and grafts low-saliency coordinates back to the base model. Stage two aggregates the remaining task vectors with a preconditioner P* = Diag(√v + ε) (Eqs. 6–7). Stage three compresses v with an AdaFactor-style rank-1 approximation. Experiments merge five Llama-3.1-8B SFT experts (math, code, general instruction following, knowledge recall, precise instruction following). OTA+FFG reaches the highest average score (0.582), ahead of TIES (0.565), linear averaging (0.551), and Fisher merging (0.541). Ablations show that FFG is the main driver, and the rank-1 compressed variant scores 0.571. The paper also presents visual analyses of FFG masks and of shared curvature across experts, arguing that SFT models share a common curvature geometry, which explains why linear merging works.

Significance. If the central claims hold, the paper contributes a practical, nearly storage-free use of optimizer statistics for model merging, plus an interpretable sparsification technique (FFG) that appears to improve merging over strong baselines on an external benchmark. The release of code, training/evaluation scripts, and checkpoints is a concrete asset for reproducibility. The structural analyses of FFG masks are interesting and potentially useful for understanding where SFT knowledge is stored. However, the theoretical bridge connecting Adam's second moment to loss curvature is not established for the actual one-epoch SFT checkpoints, and the aggregation formula uses √v rather than the v that the theory justifies. The ablation evidence also suggests that the aggregation component, taken alone, is not beneficial. For these reasons the paper's 'curvature-informed' framing currently overreaches its support; the empirical recipe may still be valuable, but the narrative needs to be adjusted and the theoretical claims need to be either substantiated or explicitly downgraded to heuristics.

major comments (4)
  1. [§4.3, Eqs. (6)–(7)] The aggregation step is internally inconsistent with the theory in §4.1. Section 4.1 argues that Adam's second moment v is a scaled EMA of the diagonal FIM, so the curvature weighting should use v (as in Fisher merging). Equation (6) instead defines P* = Diag(√v + ε), which is neither the Fisher weight nor Adam's actual preconditioner (Adam divides by √v + ε). No argument in §4.3 derives √v as a curvature metric. Please either derive this weighting from a stated approximation, replace it with v, or present it explicitly as an additional heuristic. As written, the claim that Eq. (6)–(7) is 'curvature-aware' is not supported by the paper's own derivation.
  2. [Appendix A, Assumptions 1–2] The proof that the Hessian equals the observed empirical FIM and that the mini-batch outer product is a scaled FIM requires (A) late-NTK locality and (B) perfect calibration at the fine-tuned checkpoint. For one-epoch SFT checkpoints trained from a base model, neither assumption is plausible: the model is unlikely to be perfectly calibrated, and the final exp_avg_sq is an EMA over the whole trajectory, not a final-point Fisher estimate. Thus Lemma 1 and Lemma 2 do not establish that v is the diagonal Hessian at w*. This is load-bearing because the paper's central interpretive claim rests on it. A concrete empirical check would be to compute the diagonal empirical Fisher (or a Hessian-vector-product estimate) on these checkpoints and compare it with v, or to test sensitivity to calibration. Without such evidence, the 'curvature proxy' language should be treated as a heuristic.
  3. [Table 3, OTA (wo FFG) row] The ablation shows that curvature-only aggregation on unpruned task vectors scores 0.536, below plain linear averaging (0.551). This directly undercuts the claim that the curvature-aware aggregation in Eq. (7) is, on its own, a beneficial component of the method. The full OTA gain is driven by FFG; the aggregation component is either neutral or harmful when FFG is removed. The paper's abstract and introduction attribute interference mitigation to the curvature-aware aggregation, but the data support attributing it to FFG. To support the current claim, please show settings where the √v weighting improves over linear averaging with the same mask, or revise the narrative to present the aggregation as a secondary heuristic.
  4. [§4.4 and §5.5] The rank-1 AdaFactor-style compression is justified by the low stable rank of v (<1.3). Low stable rank is a spectral property and does not guarantee elementwise fidelity, which is what matters for per-coordinate saliency in Eq. (5) and for per-coordinate weighting in Eq. (7). The paper reports that compressed OTA retains performance (0.571 vs 0.582), but it does not report how much the FFG masks change when v is replaced by its rank-1 reconstruction. Please report mask overlap or elementwise error between full and compressed v, so readers can see whether the compression works for the selection task, not just for the final merged score.
minor comments (7)
  1. [Figure 1] The caption says 'FGG mask regions'; should be 'FFG mask regions'.
  2. [§3.3 / §4.2] FFG is named 'Fast Fisher Grafting' and connected to OBS, but the saliency in Eq. (5) is the diagonal OBD score (H_ii Δw_i^2), not OBS's full inverse-Hessian correction. Please align the terminology and references.
  3. [Table 3] The row label 'OTA (w Linear)' is referred to in the text as 'FFG-TA (Selection Only)'. Please use one name consistently to avoid confusion.
  4. [§5.2 / Table 3] No variance or number of seeds is reported for the merging results. Since merging is deterministic but evaluation (e.g., pass@10, IFEval) may have stochastic components, please state evaluation seeds or report standard errors.
  5. [§5.4 / Figure 4] The heatmaps downsample weight matrices by uniform row/column subsampling. Please state the exact subsampling procedure for each displayed matrix and note any distortion this introduces for the visual claims about column/row structure.
  6. [§5.5] The notions of 'input feature curvature' and 'output feature curvature' are informal. Please define them in terms of sums of v entries over rows/columns, or state that they are visual heuristics.
  7. [Appendix D / Stable rank] The stable rank plots show values below 1.3, but stable rank is bounded below by 1 by construction; the displayed range makes differences look larger than they are. Consider plotting from 1.0 with appropriate ticks.

Circularity Check

0 steps flagged

No circular reduction found: benchmark results are external; mild selection loop and interpretive gloss do not rise to circularity.

full rationale

I walked the claimed derivation chain. Section 4.1 argues that Adam's second moment v is a scaled EMA of the diagonal FIM via Lemma 1 and Lemma 2, relying on external references ([27], [28], [36]) and two explicit assumptions (Late NTK Locality, Perfect Calibration). This is a substantive theoretical bridge, not a definitional equivalence: v is not defined in terms of the merged-model quality, and the merged-model quality is not defined in terms of v. FFG (Eq. 5) uses s_i = (Delta w_i)^2 * v_i for mask selection, and OTA aggregation (Eq. 7) uses P* = Diag(sqrt(v)+epsilon). The reported merged-model scores (Table 3) come from external benchmarks (GSM8K, MATH, HumanEval, IFEval, BBH, DROP, POPQA), so they are not forced by construction from the saliency scores or the second-moment matrices. The paper tunes the sparsity ratio per expert to preserve each expert's fine-tuning performance; this is a standard hyperparameter selection loop, not a fitted parameter that is then renamed a 'prediction' of merged performance. The 'shared curvature explains linear merging' section is an interpretative narrative based on v-similarity, not a derivation whose conclusion is presupposed by the method. There are no load-bearing self-citations: the optimizer-as-Fisher-proxy idea is attributed to concurrent external work ([26]) and to Morwani et al. ([36]), and none of the present authors' prior work is invoked to force a uniqueness or ansatz choice. A genuine weakness is that Eq. 6-7 use sqrt(v) whereas the Hessian/Fisher equivalence justifies v (or a scaled v), and no argument derives sqrt(v) as the curvature metric; however, this is an internal inconsistency / missing derivation (a correctness risk), not circularity, because the aggregation result is not defined to be equivalent to the theory. Therefore no significant circularity is present. Score 2 reflects the mild per-expert sparsity tuning loop and the interpretive gloss, neither of which is a definitional reduction.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The method introduces no new physical or mathematical entities. It relies on a set of modeling assumptions about optimizer states, calibration, and low-rank structure, plus one tuned per-expert sparsity ratio. The free parameters and assumptions are modest but real, and the calibration assumption is the most fragile.

free parameters (1)
  • per-expert sparsity ratio rho = not reported; tuned per expert and per method
    Section 5.1 states the sparsity ratio is tuned on a per-expert basis for OTA and baselines; the paper also reports that a fixed ratio makes OTA and TIES no better than linear merging, so the final benchmark result depends on this tuning.
axioms (4)
  • domain assumption Late NTK locality: near the optimum, the network is locally linear so the Hessian is dominated by the Generalized Gauss-Newton term.
    Assumption 1 in Appendix A.1; used to justify approximating the Hessian by the GGN, but known to be a poor approximation early in training and only argued to stabilize near the end, not proven for SFT checkpoints.
  • domain assumption Perfect calibration at fine-tuned checkpoints: the model's predictive distribution equals the true conditional data distribution at convergence.
    Assumption 2 in Appendix A.1; this is the load-bearing premise that equates the Hessian with the observed empirical Fisher and makes the mini-batch cross-terms vanish in Lemma 2.
  • domain assumption The final exponential moving average of squared gradients is a faithful proxy for the diagonal of the Fisher information at the final checkpoint.
    Section 4.1 derives this only under the two assumptions above and through an expectation over mini-batches; the stored exp_avg_sq is a stale EMA, not an exact final Fisher estimate.
  • domain assumption Second-moment matrices have very low stable rank, so rank-1 AdaFactor-style compression preserves their effect.
    Section 5.5 validates this empirically with stable-rank plots, and Section 4.4 uses it to justify the 29.9 GB to 12.6 MB compression claim.

pith-pipeline@v1.3.0-alltime-deepseek · 22885 in / 7792 out tokens · 94452 ms · 2026-08-04T16:57:27.867931+00:00 · methodology

0 comments
read the original abstract

Model merging is an effective post-training strategy for composing capabilities in large language models without joint retraining. We study this in the supervised fine-tuning (SFT) stage, where multiple capability-based SFT checkpoints -- spanning math, code, precise instruction following, general instruction following, and knowledge recall -- must be consolidated into a single model. We introduce Optimization Trajectory Aware (OTA) Merging, a curvature-aware aggregation that leverages optimizer second-moment statistics as a diagonal curvature proxy to reweight parameter edits and mitigate interference. Complementing OTA, we propose Fast Fisher Grafting (FFG), a curvature-driven task-localization step that sparsifies conflicting or low-importance edits. FFG induces extremely low-rank masks concentrated in early attention query/key projections and token embeddings, exploiting shared curvature across capabilities. We further develop a memory-light compression of the second moments that preserves OTA's effect. Across diverse capability-based SFT checkpoints, OTA+FFG improves merged-model quality over strong weight-space baselines, reduces negative transfer, and remains robust across sparsity levels. Analyses reveal substantial curvature overlap between checkpoints, offering a novel lens on why simple linear merging can be effective in practice. Ablations confirm that FFG is critical for reducing task interference and that the compressed second moments retain the gains of the full formulation. To facilitate reproducibility, we open-source all code, training and evaluation scripts, visualization artifacts, and capability-specific SFT checkpoints at https://github.com/pmahdavi/ota-merge.

Figures

Figures reproduced from arXiv: 2509.11167 by Hamed Mahdavi, Mehrdad Mahdavi, Niloofar Mireshghallah, Pouria Mahdavinia.

Figure 1
Figure 1. Figure 1: Left: FFG reveals that task-specific knowledge is highly localized. This heatmap shows the FGG mask regions for three expert models (math, code, instructions) in the token embedding layer, demonstrating clear, low-rank structured sparsity induced by FFG at 40% global density . Right: A capability-based comparison shows that our full OTA method, which combines FFG-based denoising with curvature-aware aggreg… view at source ↗
Figure 2
Figure 2. Figure 2: FFG vs. magnitude pruning across varying density ratios. FFG consistently outperforms, especially at lower [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Layer-wise density distribution at a global 40% task vector pruning density. FFG (right) exhibits strong, emergent role-aware pruning, aggressively sparsifying query/key weights while preserving value/output/FFN weights. Magnitude pruning (left) is far more uniform and less structured. aligns well with our understanding of SFT training paradigms, where the query and key layers of task vectors were shown to… view at source ↗
Figure 4
Figure 4. Figure 4: It is worth noting that this two-region behavior also appears to some extent in layers 1 and 2 before vanishing [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Stable rank analysis of the second-moment matrices ( [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Shared curvature geometry in attention layer 11. The top two rows show the Math and Code SFT models, respectively. The striking consistency in structural patterns validates our shared curvature hypothesis. The bottom row shows the max-min ratio across all models, highlighting regions of highest variance (light color) where task specialization is most pronounced. To validate that these observed differences … view at source ↗
Figure 7
Figure 7. Figure 7: Control Experiment: Shared curvature in attention layer 11 for two Code models. The top row shows a Code model trained with a Cosine LR schedule, and the second row shows a Code model trained on the same data with a WSD schedule. The structural similarity is nearly perfect. The bottom row shows the max-min ratio is consistently close to 1 (dark color), indicating minimal geometric deviation. Acknowledgemen… view at source ↗
Figure 8
Figure 8. Figure 8: Gradient norm trajectories over global training steps for each fine-tuned SFT models [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Training loss trajectories across global steps for each SFT model [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: 3-way magnitude-based comparison for attention components across layers 1, 2, 15, 18, 29, and 30. Columns [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: 3-way magnitude-based comparison for FFN components across layers 1, 2, 15, 18, 29, and 30. Columns [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: 3-way FFG comparison for FFN components across layers 1, 2, 15, 18, 29, and 30. Columns show [PITH_FULL_IMAGE:figures/full_fig_p026_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Shared curvature geometry in FFN layers across specialist models. Log-scaled heatmaps of the square root of the second-moment Adam statistics for layer 11 feed-forward network projection weights. Rows represent: Math specialist, Code specialist, Precise IF specialist, and Max-Min ratio across all models (top to bottom). Columns show Wgate, Wup, and Wdown (left to right). The structural similarity persists… view at source ↗
Figure 14
Figure 14. Figure 14: Control Experiment: Shared curvature geometry in FFN layers for two Code models. Log-scaled heatmaps of the square root of the second-moment Adam statistics for layer 11 feed-forward network projection weights. Rows represent: Code specialist (Cosine LR), Code specialist (WSD LR), and Max-Min ratio across the two models (top to bottom). The near-perfect structural similarity and low max-min ratio provide … view at source ↗
Figure 15
Figure 15. Figure 15: FFG masks exhibit structured sparsity in the self-attention q-projection. Row-wise (top) and column￾wise (bottom) sparsity histograms for layers 0, 1, 29, and 30 (left-to-right, top-to-bottom). Note the concentration near ρ ≈ 1.0, indicating that many rows/columns are almost entirely pruned. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Magnitude pruning produces weaker row/column structure. Histograms for the same layers and weight type as [PITH_FULL_IMAGE:figures/full_fig_p030_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 15 linked inside Pith

  1. [1]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Lit...

    1901

  2. [2]

    Linear mode connectivity and the lottery ticket hypothesis

    Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. InInternational Conference on Machine Learning, pages 3259–3269. PMLR, 2020

    2020

  3. [3]

    What matters for model merging at scale?, 2024

    Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. What matters for model merging at scale?, 2024

    2024

  4. [4]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    Pith/arXiv arXiv 2014

  5. [5]

    Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36:7093–7115, 2023

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36:7093–7115, 2023

    2023

  6. [6]

    Task-specific skill localization in fine-tuned language models

    Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. Task-specific skill localization in fine-tuned language models. InInternational Conference on Machine Learning, pages 27011–27033. PMLR, 2023

    2023

  7. [7]

    Adafactor: Adaptive learning rates with sublinear memory cost.arXiv preprint arXiv:1804.04235, 2018

    Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost.arXiv preprint arXiv:1804.04235, 2018

    Pith/arXiv arXiv 2018

  8. [8]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. InInternational conference on machine learning, pages 23965–23998. P...

    2022

  9. [9]

    Merging models with fisher-weighted averaging

    Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. InInternational Conference on Learning Representations, 2022. URLhttps://openreview.net/forum?id=LSKlp_aceOC

    2022

  10. [10]

    Editing models with task arithmetic.arXiv preprint arXiv:2212.04089, 2022

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic.arXiv preprint arXiv:2212.04089, 2022

    Pith/arXiv arXiv 2022

  11. [11]

    Task arithmetic in the tan- gent space: Improved editing of pre-trained models

    Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tan- gent space: Improved editing of pre-trained models. InAdvances in Neural Information Processing Systems, 2023. URL https://papers.neurips.cc/paper_files/paper/2023/file/ d28077e5ff52034cd35b4aa15320caea-Paper-Conference.pdf

    2023

  12. [12]

    Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa

    Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. InInternational Conference on Learning Representations, 2023. URL https:// openreview.net/forum?id=CQsmMYmlP5T

    2023

  13. [13]

    Merging by matching models in task parameter subspaces.Trans- actions on Machine Learning Research, 2024

    Derek Tam, Mohit Bansal, and Colin Raffel. Merging by matching models in task parameter subspaces.Trans- actions on Machine Learning Research, 2024. URL https://openreview.net/forum?id=qNGo6ghWFB. Certified and published on OpenReview

    2024

  14. [14]

    EMR- merging: Tuning-free high-performance model merging

    Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. EMR- merging: Tuning-free high-performance model merging. InAdvances in Neural Information Pro- cessing Systems, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/ dda5cac5272a9bcd4bc73d90bc725ef1-Paper-Conference.pdf. NeurIPS 2024 Spotlight

    2024

  15. [15]

    Arcee’s mergekit: A toolkit for merging large language models

    Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee’s mergekit: A toolkit for merging large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 477–485, Miami, Florida, US, 2024. Associ...

    doi:10.18653/v1/2024 2024

  16. [16]

    Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems, volume 5, 1992. URL https://proceedings.neurips. cc/paper/1992/hash/303ed4c69846ab36c2904d3ba8573050-Abstract.html. 16

    1992

  17. [17]

    Adapterfusion: Non-destructive task composition for transfer learning

    Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main V olume, pages 487–503, Online, April 2021. Association for Computational Linguistics...

    doi:10.18653/v1/2021.eacl-main.39 2021

  18. [18]

    Lora soups: Merging loras for practical skill composition tasks, 2024

    Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham Kakade, Eran Malach, and Samy Jelassi. Lora soups: Merging loras for practical skill composition tasks, 2024. URLhttps://arxiv.org/abs/2410.13025

    Pith/arXiv arXiv 2024

  19. [19]

    Multi lora meets vision: Merging multiple adapters to create a multi task model, 2024

    Ege Kesim and Selahattin Serdar Helli. Multi lora meets vision: Merging multiple adapters to create a multi task model, 2024. URLhttps://arxiv.org/abs/2411.14064

    Pith/arXiv arXiv 2024

  20. [20]

    Denker, and Sara A

    Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. InAdvances in Neural Information Pro- cessing Systems, volume 2, pages 598–605. Morgan Kaufmann, 1990. URL https://proceedings.neurips. cc/paper/1989/hash/6c9882bbac1c7093bd25041881277658-Abstract.html

    1990

  21. [21]

    Woodfisher: Efficient second-order approximation for neural network compres- sion

    Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compres- sion. InAdvances in Neural Information Processing Systems, volume 33, 2020. URL https://proceedings. neurips.cc/paper/2020/hash/d1ff1ec86b62cd5f3903ff19c3a326b2-Abstract.html

    2020

  22. [22]

    SparseGPT: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,Proceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of Machine Learning Research, pages 10323–...

    2023

  23. [23]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695, 2023. URLhttps://arxiv.org/abs/2306.11695

    Pith/arXiv arXiv 2023

  24. [24]

    Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-tuning. In Advances in Neural Information Processing Systems, volume 33, 2020. URL https://proceedings.neurips. cc/paper/2020/hash/eae15aabaa768ae4a5993a8a4f4fa6e4-Abstract.html

    2020

  25. [25]

    Loss surfaces, mode connectivity, and fast ensembling of dnns

    Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wil- son. Loss surfaces, mode connectivity, and fast ensembling of dnns. InAdvances in Neu- ral Information Processing Systems, volume 31, 2018. URL https://papers.nips.cc/paper/ 8095-loss-surfaces-mode-connectivity-and-fast-ensembling-of-dnns

    2018

  26. [26]

    Fishers for free? approximating the fisher information matrix by recycling the squared gradient accumulator

    YuXin Li, Felix Dangel, Derek Tam, and Colin Raffel. Fishers for free? approximating the fisher information matrix by recycling the squared gradient accumulator. InProceedings of the 42nd International Conference on Machine Learning (ICML), 2025. PMLR 267

    2025

  27. [27]

    Natural gradient works efficiently in learning.Neural computation, 10(2):251–276, 1998

    Shun-Ichi Amari. Natural gradient works efficiently in learning.Neural computation, 10(2):251–276, 1998

    1998

  28. [28]

    New insights and perspectives on the natural gradient method.Journal of Machine Learning Research, 21(146):1–76, 2020

    James Martens. New insights and perspectives on the natural gradient method.Journal of Machine Learning Research, 21(146):1–76, 2020

    2020

  29. [29]

    Optimizing neural networks with kronecker-factored approximate curvature

    James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pages 2408–2417. PMLR, 2015

    2015

  30. [30]

    Merging models with fisher-weighted averaging.Advances in Neural Information Processing Systems, 35:17703–17716, 2022

    Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging.Advances in Neural Information Processing Systems, 35:17703–17716, 2022

    2022

  31. [31]

    Merging by matching models in task parameter subspaces.arXiv preprint arXiv:2312.04339, 2023

    Derek Tam, Mohit Bansal, and Colin Raffel. Merging by matching models in task parameter subspaces.arXiv preprint arXiv:2312.04339, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Optimizing neural networks with kronecker-factored approximate curvature

    James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In Francis Bach and David Blei, editors,Proceedings of the 32nd International Conference on Machine Learning, volume 37 ofProceedings of Machine Learning Research, pages 2408–2417, Lille, France, 07–09 Jul 2015. PMLR. URLhttps://proceedings.mlr.press/...

    2015

  33. [33]

    Optimal brain damage

    Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. InAdvances in neural information processing systems, volume 2, 1989

    1989

  34. [34]

    Second order derivatives for network pruning: Optimal brain surgeon

    Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in neural information processing systems, volume 5, 1992. 17

    1992

  35. [35]

    Adaptive subgradient methods for online learning and stochastic optimization.Journal of machine learning research, 12(7), 2011

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization.Journal of machine learning research, 12(7), 2011

    2011

  36. [36]

    A new perspective on shampoo’s preconditioner.arXiv preprint arXiv:2406.17748, 2024

    Depen Morwani, Itai Shapira, Nikhil Vyas, Eran Malach, Sham Kakade, and Lucas Janson. A new perspective on shampoo’s preconditioner.arXiv preprint arXiv:2406.17748, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Limitations of the ntk for understanding generalization in deep learning.arXiv preprint arXiv:2206.10012, 2022

    Nikhil Vyas, Yamini Bansal, and Preetum Nakkiran. Limitations of the ntk for understanding generalization in deep learning.arXiv preprint arXiv:2206.10012, 2022

    Pith/arXiv arXiv 2022

  38. [38]

    Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel.Advances in Neural Information Processing Systems, 33:5850–5861, 2020

    2020

  39. [39]

    T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguist...

    Pith/arXiv arXiv 2024

  41. [41]

    Arcee’s mergekit: A toolkit for merging large language models

    Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee’s mergekit: A toolkit for merging large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 477–485, 2024

    2024

  42. [42]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch.arXiv.org, 2023

    Le Yu, Yu Bowen, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch.arXiv.org, 2023. doi: 10.48550/arxiv.2311.03099

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.03099 2023

  43. [43]

    Model breadcrumbs: Scaling multi-task model merging with sparse masks

    MohammadReza Davari and Eugene Belilovsky. Model breadcrumbs: Scaling multi-task model merging with sparse masks. InEuropean Conference on Computer Vision, pages 270–287. Springer, 2024

    2024

  44. [44]

    Pong, Simon Sidor, William Saunders, Miles Brundage, Ilya Sutskever, Wojciech Zaremba, John Schulman, and Dario Amodei

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    2021

  45. [45]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2024

    Jian Liu, Chenan Wang, Yushan Zhang, Yixin Fu, Yuan Jiang, Enyi Shen, and Qing Wang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2024

    2024

  46. [46]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Michael Petrov, Bob McGrew, Jerry Tworek, Douwe Kiela, Henrique Ponde de Oliveira Pinto, Jared Kaplan, and Dario Amodei. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  47. [47]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    Pith/arXiv arXiv 2021

  48. [48]

    Ifeval: A new benchmark for evaluating llms on instruction following, 2023

    Yucheng Zhou, Tao Yu, Zihan Wang, Keerti Banweer, Yuning Mao, Pengcheng Yin, and Hai-Tao Zheng. Ifeval: A new benchmark for evaluating llms on instruction following, 2023

    2023

  49. [49]

    Challenging big-bench tasks and whether chain-of-thought can solve them, 2022

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aseem Rawat, Swayam Singhal, Siddhartha Brahma, Jason Wei, Aakanksha Chowdhery, and Denny Zhou. Challenging big-bench tasks and whether chain-of-thought can solve them, 2022

    2022

  50. [50]

    Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V olume 1 (Long and Short Papers...

    2019

  51. [51]

    When not to trust language models: Investigating effectiveness of parametric and non-parametric memories, 2023

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories, 2023

    2023

  52. [52]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  53. [53]

    Transformers as support vector machines.arXiv preprint arXiv:2308.16898, 2023

    Davoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis, and Samet Oymak. Transformers as support vector machines.arXiv preprint arXiv:2308.16898, 2023

    Pith/arXiv arXiv 2023

  54. [54]

    role": "system

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models.Neural Information Processing Systems, 2023. 19 A Theoretical Justifications and Proofs This section provides a detailed derivation of the theoretical insights presented in Section 4.1, which establish Adam’s second-moment acc...

    2023