Non-Uniform Parameter-Wise Model Merging

Albert Manuel Orozco Camacho; Eugene Belilovsky; Guy Wolf; Stefan Horoi

REVIEW 4 major objections 7 minor 42 references

Non-Uniform Parameter-Wise Model Merging

T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes NP Merge, a model-merging technique that learns a distinct interpolation weight for every parameter of two aligned neural networks via gradient-based optimization, and claims this per-parameter flexibility produces…

desk verdict A promising per-parameter merging method that needs a fair comparison against the closest prior work before its claims hold. read the letter →

arxiv 2412.15467 v1 pith:3DZFXSAS submitted 2024-12-20 cs.LG cs.AI

Albert Manuel Orozco Camacho , Stefan Horoi , Guy Wolf , Eugene Belilovsky This is my paper · ORCID

classification cs.LGcs.AI

keywords modelmergingparameterinterpolationgradient-basedoptimizationlinearmodeconnectivityalignmentfederatedlearningensemblemethodsnon-uniform

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces NP Merge, a method for combining two or more neural networks by learning a separate interpolation coefficient for each weight, rather than using one scalar for the whole model. After the models are aligned, the coefficients are optimized by gradient descent on labeled data, with a sigmoid keeping each coefficient between 0 and 1. The paper argues that this per-parameter flexibility better exploits linear mode connectivity, especially when models were trained on different data distributions, and reports accuracy gains over uniform averaging, standard fine-tuning, and prior merging methods in several settings. The central empirical claim is that NP Merge consistently outperforms past merging methods; this matters because model merging offers the memory and inference savings of a single model plus some of the accuracy gains of ensembling.

What carries the argument

The central object is the per-parameter interpolation tensor $\alpha_i$, which assigns one scalar in $(0,1)$ to every weight in every layer, combined with the elementwise blend $W_i = \alpha_i \odot W^A_i + (1-\alpha_i) \odot W^{B'}_i$. Each coefficient is parameterized as $\alpha_i = \sigma(\alpha^{\text{pre}}_i)$, so the learned values stay strictly between 0 and 1 and the merged model remains inside the line segment connecting the two aligned models. This bounded formulation acts as a regularizer that pulls the solution back toward the equal average ($\alpha = 0.5$), and the alignment of the two models is held fixed while the coefficients are optimized by gradient descent.

What would settle it

Construct a pair of models where the optimal merged weight for at least some parameters is known to lie outside the interval between the two base weights (for example, a case where the best combined weight has larger magnitude than either base weight, which could be created by deliberately scaling one model's weights). If NP Merge's accuracy on that pair is worse than unconstrained fine-tuning or even uniform averaging, that would show the convex-hull restriction is a real limitation. A simpler check is to compare NP Merge against uniform averaging on a task where no alignment is applied; if NP Merge fails to beat the trivial average in that regime, the method's advantage disappears.

Watch

Extended reading notes

Core claim

The paper's central claim is that not all parameters of a neural network are equally important when merging, and that assigning a learned, per-parameter weight to each weight in the interpolation between two aligned models yields merged models that are more accurate than the uniform average. Concretely, after aligning two models into the same mode-connectivity basin, the merged weights are given by $W_i = \alpha_i \odot W^A_i + (1-\alpha_i) \odot W^{B'}_i$, where each scalar $\alpha_i$ is optimized by backpropagating through the merged network on labeled data. The search is constrained to the convex hull of the two models' weight vectors because each $\alpha_i$ is produced by a sigmoid, so the coefficients lie strictly between 0 and 1. The paper demonstrates this on CIFAR-10, CIFAR-100, and ImageNet-200 with VGG and ResNet architectures, using Permute or Weight Matching for alignment, and reports that NP Merge remains more stable than fine-tuning when the optimization dataset is as small as one example per class.

Load-bearing premise

The load-bearing premise is that the best merged model can be expressed as a per-parameter convex combination of the two aligned models, with every coefficient between 0 and 1; if the optimal merge requires extrapolating outside that segment, NP Merge cannot reach it.

Editorial extensions

If this is right

NP Merge can be applied on top of any existing alignment method by replacing the uniform scalar with learned per-parameter coefficients, so existing alignment research can be directly upgraded.
With limited optimization data, NP Merge degrades more gracefully than fine-tuning, making it better suited to federated or privacy-restricted settings where labeled validation data is scarce.
Repeated pairwise merging extends NP Merge to 4, 8, or more models with stable accuracy in balanced settings, providing a practical route to merging many decentralized models.
The learned per-parameter coefficients are interpretable as the contribution of each weight from each base model, which could be used to diagnose which layers or units carry the most task-specific information.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The learned coefficients may implicitly approximate a Fisher-information-style weighting, so a testable prediction is that the optimized $\alpha_i$ values correlate with the diagonal entries of each model's Fisher information matrix.
Because the sigmoid biases coefficients toward 0.5, NP Merge behaves like a constrained fine-tuning that begins at the uniform average; relaxing the $(0,1)$ bound in a controlled way might unlock further gains, at the cost of robustness.
NP Merge's success on disjoint and imbalanced data splits suggests it could be applied to task-vector arithmetic, learning per-parameter scaling of task vectors rather than the scalar coefficients used in current task arithmetic.
The current method requires labeled optimization data; a natural extension is to optimize the coefficients unsupervised, for example by matching activation statistics between the merged and base models.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

NP Merge is a straightforward idea: after aligning two models, learn a per-parameter interpolation weight by gradient descent on training data. That is worth something. It is not a new capability, but it is a clean way to get more out of existing alignments, and the paper shows it consistently beats uniform averaging and standard alignment baselines across several settings. The extension to multiple models via pairwise merging is sensible, and the data-efficiency experiments—showing NP Merge degrades more gracefully than fine-tuning when only a few examples per class are available—are a genuinely useful addition.

The problems are two, and they are real. First, the paper never compares against the closest per-parameter baselines it cites: Fisher-weighted averaging (Matena & Raffel) and function-space aggregation (Dhawan et al.). Those methods also assign per-parameter weights, so NP Merge is an alternative to them, not a new category. Without those numbers, the abstract's claim that it "outperforms past methods" simply is not established. Second, the prose goes further than the tables. Section IV-C and the conclusion say NP Merge beats ensemble accuracy, but in every reported table the ensemble is better—often by a lot (Table III: 75.96 vs 62.21 top-1). That is not a minor wording issue; it is a false claim that should be removed.

There is also a smaller point about the sigmoid constraint pinning alpha to (0,1). It is a deliberate regularizer, and the comparison to fine-tuning helps. But the method cannot escape the convex hull of the two aligned models. One sentence acknowledging that limitation would be appropriate.

Overall, the paper is a useful incremental contribution. It is not a revolution, but the method is clean, the experimental coverage is reasonable, and the data-efficiency analysis is a plus. The missing baselines and the ensemble overclaim are correctable. I would send it to a serious referee and ask for a revision that adds the two missing comparisons and tones down the claims. For a reading group, it would be a good case study in how easy it is to overstate when you omit the nearest competitors.

Referee Report

4 major / 7 minor

Summary. The paper proposes NP Merge, a method that learns a per-parameter interpolation coefficient for each weight when merging two neural networks that have already been aligned. The coefficients are constrained to (0,1) via a sigmoid reparameterization and optimized by gradient descent on labeled data, with batch-norm statistics reset after merging. The authors evaluate NP Merge on VGG11 and ResNet20 models trained on CIFAR-10/100 under same-data and non-uniform class splits, on ResNet18 on ImageNet-200, and in a many-model setting via iterative pairwise merging. They compare against uniform averaging, permutation-based and CCA alignment, model ensembles, and fine-tuning. The central claims are that NP Merge outperforms past merging methods and can outperform ensembles, but the reported tables do not support the stronger ensemble claims.

Significance. If the empirical claims were fully supported, NP Merge would be a useful and simple addition to the model-merging toolkit: it is architecture-agnostic, compatible with existing alignment methods, and the data-size robustness analysis and iterative multi-model extension are practically relevant. However, the evaluation omits the closest per-parameter merging baselines (Fisher-weighted averaging and function-space aggregation), and the manuscript's prose claims about outperforming ensembles are contradicted by its own tables. The underlying optimization procedure appears sound, but the central empirical claim needs substantial revision before the result can be considered established.

major comments (4)

[Section IV-C, Tables II and III] The sentence in Section IV-C that 'results in Tables II and III show that NP Merge can outperform all the baseline accuracies, including the ensemble accuracy' is directly contradicted by the numbers in those tables. In Table II (80%-20%) the ensemble reaches 77.84% while NP-P reaches 73.13% and NP-WM 72.94%; in the Dirichlet column the ensemble is 73.77% while both NP variants are 73.45%; in Table III the ensemble Top-1 accuracy is 75.96% while NP-P is 62.21% and NP-WM is 58.55%. The conclusion in Section V that NP Merge 'consistently outperforms traditional merging methods and ensemble techniques' is also false for Table I, where the ensemble achieves 89.65% versus 88.38% (NP-P) and 88.49% (NP-WM) on CIFAR-10, and 73.51% versus 62.88% (NP-P) and 61.68% (NP-WM) on CIFAR-100. These overstatements are load-bearing because the abstract's central claim is 'outperforming past methods'; the manuscript must either weaken the claims to match the data or justify why the ensemble comparison is not the relevant one.
[Section IV-A and Section II-C] The baseline list in Section IV-A omits Fisher-weighted averaging [13] and function-space aggregation [18], the two methods that the paper itself identifies in Section II-C as the closest prior work on per-parameter weighted merging. Because the abstract claims NP Merge 'outperform[s] past methods' and the stated novelty is per-parameter weighting, a fair comparison should include these baselines under the same alignment procedure (Permute or Weight Matching) and the same optimization data. Without those numbers, the central claim that learned per-parameter weights improve over existing per-parameter merging methods is not established.
[Section III-D] The sigmoid parameterization constrains every alpha to (0,1), so the merged model is restricted to the per-parameter convex hull of the two aligned models. The paper presents this boundedness as a regularization advantage, but it also means that any optimal merge requiring extrapolation outside the segment is unreachable. This assumption is load-bearing for the method's design and is not tested. Please report results with unconstrained alpha (or a comparison to fine-tuning in settings where the best weights may lie outside the hull) to characterize when the bounded search space helps and when it limits performance.
[Section IV-D] The comparison with fine-tuning is not fully specified. NP Merge uses a fixed 10 epochs, a fixed learning rate, and the sigmoid-induced regularization, while fine-tuning is said to use 'optimal hyper-parameters' without reporting them. Because one of the paper's stated contributions is that NP Merge is more stable than fine-tuning with little data, the fine-tuning protocol (epochs, learning rate schedule, early stopping, and data budget) needs to be reported so the comparison is budget-matched and reproducible.

minor comments (7)

[Author affiliation] The affiliation contains a typo: 'Insitute' should be 'Institute'.
[Section III-B] The notation 'alpha = S_{i=1} alpha_i tensor' is unclear; use a concatenation or Cartesian-product notation for the per-layer alpha tensors.
[Figure 1] Figure 1 reports trends without error bars or standard deviations, although the tables report them; the 'stable accuracy' claim is hard to evaluate without variability information.
[Tables II and IV] There is a small inconsistency between Table II and Table IV for the Weight Matching training-data accuracy (60.86% in Table II versus 60.20% in Table IV after batch-norm reset); please reconcile or explain.
[Throughout] The manuscript alternates between 'NP Merge' and 'NP-Merge'; please choose one consistent spelling.
[Section IV-A] The learning rate is reported for CIFAR-10 and CIFAR-100 but not for the ImageNet-200 experiment; please clarify the optimization hyperparameters used there.
[Section IV-F] The memory and computational overhead discussion is qualitative; a quantitative measurement (e.g., wall-clock time and memory usage relative to baselines) would strengthen the trade-off discussion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: NP Merge's interpolation weights are fit to training data and evaluated on held-out test accuracy, and no load-bearing claim reduces to its own inputs.

full rationale

I found no circular step in the derivation chain. The paper proposes per-parameter interpolation coefficients optimized by gradient descent on training data (Section IV-A: "We can then backpropagate through these layers, compute the training loss gradient for α, and update them through gradient descent") and reports accuracy on the full test set ("We always evaluate and report the accuracy of the full test set of the task considered"). The claimed contribution, the learned per-parameter weights, is therefore an empirically fitted procedure, not a quantity that is defined in terms of the reported test accuracy. The design builds on external alignment methods (Permute, Weight Matching) and resets Batch Norm statistics per REPAIR, rather than invoking the paper's own conclusions as premises. The only significant self-citation is [17], CCA Merge, used as a baseline rather than as evidence for the method; the omission of Fisher-weighted averaging and function-space aggregation as baselines is a completeness or experimental-design concern, not circularity. The bounded-convex-hull assumption in Section III-D is a design limitation, not a circular reduction. Thus the central empirical claim is self-contained against external test data and the score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the choice to optimize a per-parameter interpolation coefficient for every weight, plus a handful of fixed hyperparameters. No new entities are introduced. The key assumptions are that permutation alignment suffices to make interpolation meaningful and that training the coefficients on labeled data for 10 epochs transfers to the test distribution.

free parameters (4)

per-parameter interpolation weights α (one scalar per model parameter) = learned by gradient descent
The core free parameters of the method; fitted to the training data for each merge. There are as many α values as model parameters.
number of optimization epochs for α = 10
Chosen without reported tuning; affects final accuracy.
learning rate for α optimization = 0.01 (Adam)
Fixed for all experiments; no sensitivity analysis.
α initialization = 0.5
Chosen to avoid biasing toward either model; with sigmoid parameterization this corresponds to α_pre = 0.

assumptions (5)

domain assumption The two models to be merged have identical architectures (same layer sizes).
Stated in Section III-A; all derivations and experiments assume matching architectures.
domain assumption Permutation-based alignment makes the models linearly mode connected, so per-parameter interpolation is meaningful.
The method inherits the Entezari et al. conjecture and prior alignment methods; the paper relies on this to initialize the merge.
domain assumption Gradient-based optimization of α on the training set for 10 epochs yields merged models that generalize to the test set.
No proof; the paper's positive results depend on this empirical claim.
domain assumption Resetting BatchNorm statistics after merging is sufficient to avoid variance collapse during α optimization.
Mentioned in Section IV-A following REPAIR; the procedure during optimization is not detailed.
standard math Backpropagation through the merged weight equation is differentiable and stable.
The merged weights are a linear combination of the frozen model weights, so gradients flow only through α; this is standard.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Non-Uniform Parameter-Wise Model Merging." pith.science (2026). https://pith.science/paper/3DZFXSAS

@misc{pith2026241215467,
  author       = {Pith},
  title        = {Pith review of: Non-Uniform Parameter-Wise Model Merging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3DZFXSAS}},
  note         = {Machine review of arXiv:2412.15467}
}

read the original abstract

Combining multiple machine learning models has long been a technique for enhancing performance, particularly in distributed settings. Traditional approaches, such as model ensembles, work well, but are expensive in terms of memory and compute. Recently, methods based on averaging model parameters have achieved good results in some settings and have gained popularity. However, merging models initialized differently that do not share a part of their training trajectories can yield worse results than simply using the base models, even after aligning their neurons. In this paper, we introduce a novel approach, Non-uniform Parameter-wise Model Merging, or NP Merge, which merges models by learning the contribution of each parameter to the final model using gradient-based optimization. We empirically demonstrate the effectiveness of our method for merging models of various architectures in multiple settings, outperforming past methods. We also extend NP Merge to handle the merging of multiple models, showcasing its scalability and robustness.

Figures

Figures reproduced from arXiv: 2412.15467 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 32 canonical work pages

[13]

Merging models with fisher-weighted averaging,

M. S. Matena and C. A. Raffel, “Merging models with fisher-weighted averaging,” in Advances in Neural Information Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 17 703–17 716. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2022/file/70c26937f...

work page 2022
[18]

Leveraging function space aggregation for federated learning at scale,

N. Dhawan, N. E. Mitchell, Z. Charles, Z. Garrett, and G. K. Dziugaite, “Leveraging function space aggregation for federated learning at scale,” Transactions on Machine Learning Research, 2024, expert Certification. [Online]. Available: https://openreview.net/forum?id=Ytp9KFKZfZ

work page 2024
[1]

Random decision forests,

T. K. Ho, “Random decision forests,” in Proceedings of 3rd international conference on document analysis and recognition , vol. 1. IEEE, 1995, pp. 278–282

1995
[2]

On power laws in deep ensembles,

E. Lobacheva, N. Chirkova, M. Kodryan, and D. P. Vetrov, “On power laws in deep ensembles,” Advances In Neural Information Processing Systems, vol. 33, pp. 2375–2385, 2020

work page 2020
[3]

Communication-Efficient Learning of Deep Networks from Decentralized Data,

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , ser. Proceedings of Machine Learning Research, A. Singh and J. Zhu, Eds., vol. 54. PMLR, 20–22 Apr 2017, pp. 1273–1282....

2017
[4]

Model fusion via optimal transport,

S. P. Singh and M. Jaggi, “Model fusion via optimal transport,” in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 22 045–22 055. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2020/file/fb2697869f56484404c8ceee2985b0...

work page 2020
[5]

Optimizing mode connectivity via neuron alignment,

N. Tatro, P.-Y . Chen, P. Das, I. Melnyk, P. Sattigeri, and R. Lai, “Optimizing mode connectivity via neuron alignment,” in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 15 300–15 311. [Online]. Available: https://proceedings.neurips.cc/pa...

work page 2020
[6]

Git re-basin: Merging models modulo permutation symmetries,

S. Ainsworth, J. Hayase, and S. Srinivasa, “Git re-basin: Merging models modulo permutation symmetries,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id=CQsmMYmlP5T

work page 2023

Show all 42 references

[7]

Re-basin via implicit sinkhorn differentiation,

F. A. G. Pe ˜na, H. R. Medeiros, T. Dubail, M. Aminbeidokhti, E. Granger, and M. Pedersoli, “Re-basin via implicit sinkhorn differentiation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 20 237–20 246

2023
[8]

REPAIR: REnormalizing permuted activations for interpolation repair,

K. Jordan, H. Sedghi, O. Saukh, R. Entezari, and B. Neyshabur, “REPAIR: REnormalizing permuted activations for interpolation repair,” in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/ forum?id=gU5sJ6ZggcX

2023
[9]

Zipit! merging models from different tasks without training,

G. Stoica, D. Bolya, J. Bjorner, T. Hearn, and J. Hoffman, “Zipit! merging models from different tasks without training,” arXiv, 2023

2023
[10]

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time,

M. Wortsman, G. Ilharco, S. Y . Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y . Carmon, S. Kornblith, and L. Schmidt, “Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time,” in Proceedi...

2022
[11]

Editing models with task arithmetic,

G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi, “Editing models with task arithmetic,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id=6t0Kwf8-jrj

2023
[12]

Robust fine-tuning of zero-shot models,

M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs, R. G. Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong, and L. Schmidt, “Robust fine-tuning of zero-shot models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June...

2022
[14]

Population parameter averaging (papa),

A. Jolicoeur-Martineau, E. Gervais, K. Fatras, Y . Zhang, and S. Lacoste-Julien, “Population parameter averaging (papa),” 2023. [Online]. Available: https://arxiv.org/abs/2304.03094

2023 arXiv
[15]

Linear mode connectivity and the lottery ticket hypothesis,

J. Frankle, G. K. Dziugaite, D. Roy, and M. Carbin, “Linear mode connectivity and the lottery ticket hypothesis,” in Proceedings of the 37th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, H. D. III and A. Singh, Eds., vol
[16]

The role of permutation invariance in linear mode connectivity of neural networks,

R. Entezari, H. Sedghi, O. Saukh, and B. Neyshabur, “The role of permutation invariance in linear mode connectivity of neural networks,” in International Conference on Learning Representations ,
[17]

Harmony in diversity: Merging neural networks with canonical correlation analysis,

S. Horoi, A. M. O. Camacho, E. Belilovsky, and G. Wolf, “Harmony in diversity: Merging neural networks with canonical correlation analysis,” in Forty-first International Conference on Machine Learning , 2024. [Online]. Available: https://openreview.net/forum?id=hLuNVjRnY3

2024
[19]

A closer look at memorization in deep networks,

D. Arpit, S. Jastrzebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y . Bengio, and S. Lacoste-Julien, “A closer look at memorization in deep networks,” in Proceedings of the 34th International Conference on Machine Learning, ser. Pr...

2017
[20]

Visualizing the loss landscape of neural nets,

H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, “Visualizing the loss landscape of neural nets,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 31. Curran Associates, Inc., 2018, pp. 6389– 6399

2018
[21]

Exploring the geometry and topology of neural network loss landscapes,

S. Horoi, J. Huang, B. Rieck, G. Lajoie, G. Wolf, and S. Krish- naswamy, “Exploring the geometry and topology of neural network loss landscapes,” in Advances in Intelligent Data Analysis XX , T. Bouadi, E. Fromont, and E. H ¨ullermeier, Eds. Cham: Springer International Publis...

2022
[22]

Topology and geometry of half-rectified network optimization,

C. D. Freeman and J. Bruna, “Topology and geometry of half-rectified network optimization,” in International Conference on Learning Representations, 2017. [Online]. Available: https://openreview.net/ forum?id=Bk0FWVcgx

2017
[23]

Loss surfaces, mode connectivity, and fast ensembling of dnns,

T. Garipov, P. Izmailov, D. Podoprikhin, D. P. Vetrov, and A. G. Wilson, “Loss surfaces, mode connectivity, and fast ensembling of dnns,” in Advances in Neural Information Processing Systems , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, ...
[24]

Essentially no barriers in neural network energy landscape,

F. Draxler, K. Veschgini, M. Salmhofer, and F. Hamprecht, “Essentially no barriers in neural network energy landscape,” in Proceedings of the 35th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. ...

2018
[25]

On the Geometry of Feedforward Neural Network Error Surfaces,

A. M. Chen, H.-m. Lu, and R. Hecht-Nielsen, “On the Geometry of Feedforward Neural Network Error Surfaces,” Neural Computation, vol. 5, no. 6, pp. 910–927, 11 1993. [Online]. Available: https: //doi.org/10.1162/neco.1993.5.6.910

1993 doi
[26]

On the algebraic structure of feedforward network weight spaces,

R. Hecht-Nielsen, “On the algebraic structure of feedforward network weight spaces,” in Advanced Neural Computers , R. ECKMILLER, Ed. Amsterdam: North-Holland, 1990, pp. 129–135. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ B9780444884008500194

1990
[27]

What is being transferred in transfer learning?

B. Neyshabur, H. Sedghi, and C. Zhang, “What is being transferred in transfer learning?” in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 512–

2020
[28]

Rethinking the inception architecture for computer vision,

C. Szegedy, V . Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in 2016 IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 2818–2826

2016
[29]

Averaging weights leads to wider optima and better generalization,

P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson, “Averaging weights leads to wider optima and better generalization,” in Uncertainty in Artificial Intelligence , A. Globerson and R. Silva, Eds., vol. 34. AUAI Press, 2018. [Online]. Available: http://auai.o...

2018
[30]

TIES- merging: Resolving interference when merging models,

P. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal, “TIES- merging: Resolving interference when merging models,” in Thirty- seventh Conference on Neural Information Processing Systems , 2023. [Online]. Available: https://openreview.net/forum?id=xtaX3WyCj1

2023
[31]

Representation surgery for multi-task model merging,

E. Yang, L. Shen, Z. Wang, G. Guo, X. Chen, X. Wang, and D. Tao, “Representation surgery for multi-task model merging,” 2024. [Online]. Available: https://arxiv.org/abs/2402.02705

2024 arXiv
[32]

Pytorch: An imperative style, high-performance deep learning library,

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-pe...
[33]

Very deep convolutional networks for large-scale image recognition,

K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in the 3rd International Conference on Learning Representations (ICLR 2015) , 2015, pp. 1–14

2015
[34]

Deep residual learning for image recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778

2016
[35]

Wide residual networks,

S. Zagoruyko and N. Komodakis, “Wide residual networks,” CoRR, vol. abs/1605.07146, 2016. [Online]. Available: http://arxiv.org/abs/ 1605.07146

2016 arXiv
[36]

Adam: A method for stochastic optimization,

D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations (ICLR) , San Diega, CA, USA, 2015

2015
[37]

Convergent learning: Do different neural networks learn the same representations?

Y . Li, J. Yosinski, J. Clune, H. Lipson, and J. Hopcroft, “Convergent learning: Do different neural networks learn the same representations?” in Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges at NIPS 2015 , ser. Proceeding...

2015
[119]

3259–3269

PMLR, 13–18 Jul 2020, pp. 3259–3269. [Online]. Available: https://proceedings.mlr.press/v119/frankle20a.html

2020
[523]

Available: https://proceedings.neurips.cc/paper/2020/file/ 0607f4c705595b911a4f3e7a127b44e0-Paper.pdf

[Online]. Available: https://proceedings.neurips.cc/paper/2020/file/ 0607f4c705595b911a4f3e7a127b44e0-Paper.pdf

2020
[2018]

Available: https://proceedings.neurips.cc/paper files/ paper/2018/file/be3087e74e9100d4bc4c6268cdbe8456-Paper.pdf

[Online]. Available: https://proceedings.neurips.cc/paper files/ paper/2018/file/be3087e74e9100d4bc4c6268cdbe8456-Paper.pdf

2018
[2019]

Available: https://proceedings.neurips.cc/paper files/ paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf

[Online]. Available: https://proceedings.neurips.cc/paper files/ paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf

2019
[2022]

Available: https://openreview.net/forum?id=dNigytemkL

[Online]. Available: https://openreview.net/forum?id=dNigytemkL

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [13]

Merging models with fisher-weighted averaging,

M. S. Matena and C. A. Raffel, “Merging models with fisher-weighted averaging,” in Advances in Neural Information Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 17 703–17 716. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2022/file/70c26937f...

work page 2022

[2] [18]

Leveraging function space aggregation for federated learning at scale,

N. Dhawan, N. E. Mitchell, Z. Charles, Z. Garrett, and G. K. Dziugaite, “Leveraging function space aggregation for federated learning at scale,” Transactions on Machine Learning Research, 2024, expert Certification. [Online]. Available: https://openreview.net/forum?id=Ytp9KFKZfZ

work page 2024

[3] [1]

Random decision forests,

T. K. Ho, “Random decision forests,” in Proceedings of 3rd international conference on document analysis and recognition , vol. 1. IEEE, 1995, pp. 278–282

1995

[4] [2]

On power laws in deep ensembles,

E. Lobacheva, N. Chirkova, M. Kodryan, and D. P. Vetrov, “On power laws in deep ensembles,” Advances In Neural Information Processing Systems, vol. 33, pp. 2375–2385, 2020

work page 2020

[5] [3]

Communication-Efficient Learning of Deep Networks from Decentralized Data,

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , ser. Proceedings of Machine Learning Research, A. Singh and J. Zhu, Eds., vol. 54. PMLR, 20–22 Apr 2017, pp. 1273–1282....

2017

[6] [4]

Model fusion via optimal transport,

S. P. Singh and M. Jaggi, “Model fusion via optimal transport,” in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 22 045–22 055. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2020/file/fb2697869f56484404c8ceee2985b0...

work page 2020

[7] [5]

Optimizing mode connectivity via neuron alignment,

N. Tatro, P.-Y . Chen, P. Das, I. Melnyk, P. Sattigeri, and R. Lai, “Optimizing mode connectivity via neuron alignment,” in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 15 300–15 311. [Online]. Available: https://proceedings.neurips.cc/pa...

work page 2020

[8] [6]

Git re-basin: Merging models modulo permutation symmetries,

S. Ainsworth, J. Hayase, and S. Srinivasa, “Git re-basin: Merging models modulo permutation symmetries,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id=CQsmMYmlP5T

work page 2023