Training-free Heterogeneous Model Merging

Han Zheng; Jie Song; Li Sun; Mingli Song; Zhengqi Xu

REVIEW 5 major objections 4 minor 33 references

Training-free Heterogeneous Model Merging

T0 review · 5 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Neural networks that differ in depth and width can be merged without retraining, matching homogeneous merging in the paper's experiments.

desk verdict First real attempt at training-free heterogeneous model merging; depth experiments are promising, but the depth-merge derivation has an algebraic flaw and the width method is under-specified. read the letter →

arxiv 2501.00061 v1 pith:7MFCD7PF submitted 2024-12-29 cs.LG cs.AI

Zhengqi Xu , Han Zheng , Jie Song , Li Sun , Mingli Song This is my paper · ORCID

classification cs.LGcs.AI MSC 68T07

keywords modelmergingheterogeneousmodelsdepthheterogeneitywidthtraining-freeneuronzippinglayeralignmentweightaveraging

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tackles a gap in model merging: prior weight-averaging and align-then-average methods require the two networks to have the same architecture, while this work fuses models that differ in layer count and layer width. The proposed framework handles depth by segmenting the deeper model so each segment is averaged against one layer of the shallower model, and handles width by an elastic neuron zipping algorithm that collapses pairs of similar neurons until both models have a common width. The authors report that, across image classification and text-understanding benchmarks, these training-free, heterogeneous merges perform about as well as merging identical architectures. If the claim holds, task-specific models trained on different backbones can be combined into one model by linear-algebra operations on weights and activations alone.

What carries the argument

The load-bearing identity is the weight-average construction of Eq. (6): once a deep segment is paired with one shallow layer, the shallow model is extended with identity layers (non-residual models) or zero-weight residual layers (residual models) so that averaged features equal the features of averaged weights, under layerwise linear feature connectivity. The segment boundaries themselves are chosen by two heuristics: segment-wise model alignment, which maximizes CKA similarity between each segment's output and the corresponding shallow layer's output, and layer-wise model alignment, which additionally considers the internal layers of the segment. Width heterogeneity is handled by elastic neuron zipping, which merges the most similar neurons across the two models until a target width is reached. These pieces convert heterogeneous merging into repeated applications of standard permutation-alignment and weight averaging.

What would settle it

Run a pair of differently deep models on the same task, apply the proposed alignment, and compare the merged model's layerwise feature maps with the true averaged feature maps; if the mismatch grows with segment length, the core identity fails. A sharper test is to insert an activation or normalization change known to destroy linear mode connectivity and check whether depth-heterogeneous merging collapses while homogeneous merging does not.

Watch

Extended reading notes

Core claim

The paper's central claim is that structural heterogeneity does not require retraining: after alignment, a heterogeneous merge is just a sequence of weight averages. For depth, the deeper model is partitioned into as many segments as the shallower model has layers, using representation similarity to choose segment boundaries; then the shallow model is padded with identity layers (or zero-weight residual layers) so each segment and its counterpart can be averaged layer by layer through Eq. (6). For width, an elastic neuron zipping algorithm computes pairwise feature similarity between neurons of the two models and greedily merges the most similar neurons until a chosen width is reached, projecting all weights into a common dimensional space. The paper argues this yields merged models with performance comparable to depth-homogeneous and width-homogeneous merging, based on experiments on CIFAR, ImageNet, and GLUE.

Load-bearing premise

The whole depth-merging construction assumes that a layer's output moves in a straight line as its weights are blended, and that a grouped stack of layers keeps this straight-line behavior once the shallower model is padded with identity layers; if real networks' activations, normalization, or residual shortcuts break that straight-line behavior, the depth merge has no support.

Editorial extensions

If this is right

Two models with different layer counts can be merged by first segmenting the deeper model and then applying one weight average per segment, with no gradient updates.
Two models with different layer widths can be merged by neuron zipping onto a common width, so users do not need to retrain either model to a shared architecture.
The paper's reported joint and per-task accuracies for heterogeneous merges sit close to those of homogeneous merges, so architecture mismatch alone need not block model reuse.
The method extends to residual backbones and transformer-based text models, making the recipe applicable beyond simple feed-forward architectures.
Because both alignment methods are training-free, the construction cost is essentially the cost of forward passes for feature similarity plus one averaging step per segment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

This suggests that a practical model zoo could be merged without standardizing on a backbone; the hard part becomes choosing segment boundaries and target widths, not training.
A natural stress test is to apply the same recipe to models whose normalization layers or activation functions differ, since the linear-feature assumption is the component most likely to break.
The neuron-zipping routine is essentially a greedy projection onto a common width; one testable extension is to use it for partial merges or for merging more than two heterogeneous models at once.
Choosing segments by maximal CKA similarity is a heuristic, so optimizing the segmentation against the final merged model's loss rather than pairwise similarity could change which alignment wins.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the quick take: this is the first paper to attempt training-free merging of models that differ in both depth and width, and the depth experiments are genuinely impressive. But the theoretical derivation for depth merging has a real algebraic flaw, and the width method is under-specified. I'd send it to review, but the authors need to fix or retract the derivation.

What's new: the problem is well chosen. Previous merge methods (Git Re-Basin, Zipit!, OTFusion) require matching widths and mostly matching depths. The segment-wise and layer-wise alignment for depth, plus elastic neuron zipping for width, are original. The empirical scope is solid: CIFAR, ImageNet, and GLUE, with comparisons to vanilla averaging, alignment-based, and zipping baselines. Depth-heterogeneous merging lands close to homogeneous merging in most settings. Code is available—that's real evidence.

The soft spots, in order of severity. First, Eq. 5–6 don't hold as written. The identity padding for the shallow model uses per-layer permutation matrices, but the aligned contribution of a padded identity layer is P_i^B (P_{i-1}^B)^{-1}, which is not the identity unless consecutive permutations are identical. Nothing in the alignment objective constrains them. So the B-branch product in Eq. 5 does not collapse to P_l^B W^B x, and Eq. 6 does not follow. This isn't a quibble about linear feature connectivity; it's an algebraic error in the padding construction. The same issue recurs in the residual extension in Supp. S1. The empirical results might still be valid—the method may just work as a heuristic—but the paper's theoretical grounding is currently missing.

Second, the width method is too vague. We get a picture and a description, but no formal algorithm for how neurons are zipped and weights are projected to a common width. And Table III has no homogeneous merging baseline for the same widths, so the claim that width-heterogeneous merging is comparable to homogeneous merging isn't actually demonstrated.

Third, the derivation leans on layerwise linear feature connectivity [31] without checking whether it holds for ReLU, BatchNorm, or residual blocks in these heterogeneous settings. That's a secondary concern given the algebraic issue, but worth flagging.

Who's this for? Anyone working in model merging or model reuse. It's a useful problem and the results are promising. But I wouldn't take the theory at face value. If the authors can fix the derivation—or explicitly reframe the method as a heuristic with empirical support—this becomes a solid paper. As is, it deserves peer review, but with a request for major revision and a careful re-derivation.

Referee Report

5 major / 4 minor

Summary. The paper proposes a training-free framework for merging neural networks that differ in depth and width. For depth heterogeneity, the deeper model is segmented so that each segment corresponds to one layer of the shallower model, and the segments are aligned and merged via layer-wise weight averaging with identity padding for the shallower model. For width heterogeneity, an elastic neuron zipping algorithm projects neurons of different widths into a common dimensional space and merges the most similar neurons. The method is evaluated on CIFAR-10/100, ImageNet, and GLUE with ResNet, VGG, and BERT architectures, with the central claim that heterogeneous merging achieves performance comparable to homogeneous merging without retraining.

Significance. If the claims hold, this would be a useful practical contribution to model merging, enabling the combination of models with different architectures without costly retraining. The paper includes public code and experiments across vision and NLP tasks, which strengthens its potential impact. However, the theoretical derivation for depth merging is algebraically flawed, the width experiments lack a homogeneous baseline, and some experimental claims are not fully supported by the reported tables. These issues currently limit the reliability of the contribution despite the encouraging empirical results.

major comments (5)

[Sec. III-B, Eq. 5-6] The derivation of the depth-merge formula is invalid as a justification for the weight averaging in Eq. 6. The first equality in Eq. 5 treats the merged feature f^*_{l-1} as if it were the A-branch feature, but the merged feature is the result of applying summed weights to the previous merged feature. The expansion to the second line omits cross-terms: for l=2, the merged forward pass is (W^A_2 + P^B_2 (P^B_1)^{-1})(W^A_1 + P^B_1 W^B)x, which contains W^A_2 P^B_1 W^B x and P^B_2 (P^B_1)^{-1} W^A_1 x; these are absent from the derivation. The layerwise linear feature connectivity of [31] concerns interpolation between two models' feature maps, not the composition of summed weights, so Eq. 6 does not follow from the preceding feature-averaging argument. Note that the specific concern that the identity-padded product does not collapse is not correct: the B-branch product telescopes to P^B_l W^B x regardless of the permutation matrices. The real problem is the unjustified parallel-branch interpretation of a sequentially composed merged model. Please either correct the derivation or explicitly reframe Eq. 6 as a heuristic.
[Sec. IV-A, Models] The GLUE experiments create depth heterogeneity by repeating every even-numbered layer of a 12-layer BERT to obtain a 17-layer model. This produces a deeper model whose added layers are exact duplicates, so the segmentation and alignment problem becomes artificially easy: the repeated layers form segments that are trivially aligned. The NLP results therefore do not demonstrate that the method handles genuinely different Transformer depths. Please test on naturally different architectures (e.g., BERT-base vs BERT-large or models with different layer counts from pretrained checkpoints) or temper the claim about NLP generality.
[Sec. IV-C, Tab. IV] The text states that compared to vanilla averaging, lower loss barriers are observed for homogeneous and heterogeneous merging, but in Tab. IV the barriers for MNLI-mm (0.65 and 0.67) and QNLI (0.76 and 0.71) are higher than vanilla averaging (0.59 and 0.67). The claim is not supported by the per-task results and should be revised or supplemented with an aggregate statistic. Also, the 'Error' column is not defined; please state the metric used for each task (e.g., 1-accuracy for classification, MSE for STS-B).
[Sec. IV-B, Tab. II] The 'Homo Avg' entry in Tab. II reports a joint accuracy of 65.25, which is substantially higher than the single-task joint accuracies (36.70 and 37.28) and inconsistent with the pattern in Tab. I where Homo Avg is the lowest-performing method. This suggests a mislabeled entry or a typo. Please check the result and clarify what is being reported.
[Sec. IV-B, Tab. III] The width-heterogeneous merging experiments in Tab. III contain no homogeneous-merging baseline (e.g., merging two models of identical width). Since the abstract claims performance comparable to homogeneous merging, and the width experiments are a central part of the framework, the absence of such a baseline leaves the claim untested for width heterogeneity. Please add homogeneous counterparts or adjust the claim accordingly.

minor comments (4)

[Sec. III-B, Eq. 5] The first line of Eq. 5 writes the B term as P^B_1 W^B x, which is inconsistent with the product expansion in the second line; this is likely a typo and should be fixed.
[Supp. S2, Algorithm 1 and 2] In the backtrace loops of both algorithms, the variables n and m are not defined; they should be replaced with |L_A| and |L_B| (or otherwise explicitly defined).
[Sec. IV-C] The loss-barrier definition is attributed to Frankle et al. [7], but reference [7] (Frankle and Carbin, 2018) is 'The lottery ticket hypothesis' and does not define the loss barrier; please cite the appropriate source (e.g., Frankle et al., 2020 or Entezari et al., 2021).
[Sec. IV-B] The sentence 'the models merged with depth-heterogeneous architectures not only achieve higher average performance compared to the weight average of depth-homogeneous models but also exhibit similar performance to the models merged with depth-homogeneous architectures' is confusing and appears to contradict itself; please rephrase for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: benchmark evaluations are independent of the representation-similarity inputs, and the only self-citation is not load-bearing.

full rationale

The paper's core claims are evaluated on task accuracy and loss barriers, not on the alignment objectives used to construct the merged models. The depth-merge construction explicitly relies on an external layerwise linear feature connectivity assumption [31], and the segmentation algorithms use representation similarity (CKA) only to choose segments; the reported success metric (downstream accuracy) is not defined by that similarity objective. The width-merge method is likewise validated by task accuracy after merging, not by the neuron-similarity measure used to zip neurons. The only self-citation, MuDSC [27], appears in related work and is not load-bearing for the paper's derivations or experiments. The algebraic concern about Eq. 5/6 (padded identity layers not collapsing unless consecutive alignment permutations coincide) is an internal-consistency issue, not a circularity issue, because the derived merged model is not equivalent by construction to the evaluation metric. The derivation chain is therefore self-contained with respect to circularity, even though its correctness may depend on assumptions about feature linearity and permutation alignment.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The ledger shows the depth derivation depends on layerwise linear feature connectivity and the replaceability of consecutive layers, while the width method depends on the informally chosen zipping width r. No new physical entities are introduced.

free parameters (1)

r (number of remaining neurons in elastic neuron zipping) = set to the maximum width of the models being merged
Section III-C and Fig. 2. This hyperparameter determines the common width after zipping and is a design choice, not derived; it is load-bearing for the width-merging method.

assumptions (5)

domain assumption Layerwise linear feature connectivity: the feature map of layer l behaves like a linear map (up to a scaling factor) on the interpolation between W_A_l and W_B_l.
Invoked in Section III-B, Eq. 5, citing [31]. Used to derive the weight-average form for depth-heterogeneous merging; no proof is given for arbitrary heterogeneous depths, activations, or normalization layers.
domain assumption Adjacent layers of a trained network often have similar representations and consecutive layers can be replaced by a single layer.
Used in Section III-B to justify segmenting the deeper model; based on [14], [26], but assumed for the specific pairs being merged.
domain assumption CKA(fA, fB) = CKA(P_A fA, P_B fB) for reprojection matrices P.
Stated in Section III-B. This identity holds for orthogonal/permutation maps, not arbitrary projections, so the claim is stronger than justified.
ad hoc to paper Residual layers can be written as f=(W+I)x and extended with zero-weight residual layers for padding.
Assumed in Supplementary S1 to extend the shallow model's residual layer across the deep segment; relies on a specific residual formulation and ignores bias, batchnorm, and nonlinearities.
domain assumption Most SGD solutions belong to a set whose elements can be permuted so that no performance barrier exists in linear interpolation (from [4]).
Background assumption for align-then-average merging, stated in Section I and used to justify the whole merging framework.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Training-free Heterogeneous Model Merging." pith.science (2026). https://pith.science/paper/7MFCD7PF

@misc{pith2026250100061,
  author       = {Pith},
  title        = {Pith review of: Training-free Heterogeneous Model Merging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7MFCD7PF}},
  note         = {Machine review of arXiv:2501.00061}
}

read the original abstract

Model merging has attracted significant attention as a powerful paradigm for model reuse, facilitating the integration of task-specific models into a singular, versatile framework endowed with multifarious capabilities. Previous studies, predominantly utilizing methods such as Weight Average (WA), have shown that model merging can effectively leverage pretrained models without the need for laborious retraining. However, the inherent heterogeneity among models poses a substantial constraint on its applicability, particularly when confronted with discrepancies in model architectures. To overcome this challenge, we propose an innovative model merging framework designed for heterogeneous models, encompassing both depth and width heterogeneity. To address depth heterogeneity, we introduce a layer alignment strategy that harmonizes model layers by segmenting deeper models, treating consecutive layers with similar representations as a cohesive segment, thus enabling the seamless merging of models with differing layer depths. For width heterogeneity, we propose a novel elastic neuron zipping algorithm that projects the weights from models of varying widths onto a common dimensional space, eliminating the need for identical widths. Extensive experiments validate the efficacy of these proposed methods, demonstrating that the merging of structurally heterogeneous models can achieve performance levels comparable to those of homogeneous merging, across both vision and NLP tasks. Our code is publicly available at https://github.com/zju-vipa/training_free_heterogeneous_model_merging.

Figures

Figures reproduced from arXiv: 2501.00061 by the authors.

**Figure 2.** Merging two layers with different width. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Visualization of representation similarity. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 25 canonical work pages

[31]

L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. arXiv preprint arXiv:2311.03099 , 2023

arXiv 2023
[1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.ncblock write newline " " before.all 'output.state := FUNCTION new.nccont write " " before.all 'output.state := FUNCTION new.sentence output.state after.block = 'skip out...
[2]

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page
[3]

Ainsworth, J

S. Ainsworth, J. Hayase, and S. Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In ICLR , 2023

work page 2023
[4]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248--255, 2009

work page 2009
[5]

J. Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018

arXiv 2018
[6]

Entezari, H

R. Entezari, H. Sedghi, O. Saukh, and B. Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296 , 2021

arXiv 2021
[7]

Fifty, E

C. Fifty, E. Amid, Z. Zhao, T. Yu, R. Anil, and C. Finn. Efficiently identifying task groupings for multi-task learning. NIPS , 34:27503--27516, 2021

work page 2021

Show all 33 references

[8]

R. A. Fisher and E. J. Russell. On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character , 222(594-604):309--368, 1922

1922
[9]

Frankle and M

J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 , 2018

2018 arXiv
[10]

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770--778, 2016

2016
[11]

Ilharco, M

G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic. In ICLR , 2023

2023
[12]

Imfeld, J

M. Imfeld, J. Graldi, M. Giordano, T. Hofmann, S. Anagnostidis, and S. P. Singh. Transformer fusion with optimal transport. In ICLR , 2024

2024
[13]

Izmailov, D

P. Izmailov, D. Podoprikhin, T. Garipov, D. P. Vetrov, and A. G. Wilson. Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence , 2018

2018
[14]

X. Jin, X. Ren, D. Preotiuc-Pietro, and P. Cheng. Dataless knowledge fusion by merging weights of language models. In ICLR , 2023

2023
[15]

Jordan, H

K. Jordan, H. Sedghi, O. Saukh, R. Entezari, and B. Neyshabur. REPAIR : RE normalizing permuted activations for interpolation repair. In ICLR , 2023

2023
[16]

Kornblith, M

S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In K. Chaudhuri and R. Salakhutdinov, editors, 36th ICML , volume 97 of Proceedings of Machine Learning Research , pages 3519--3529. PMLR, 09--15 Jun 2019

2019
[17]

Krizhevsky, G

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009

2009
[18]

S. Luo, X. Wang, G. Fang, Y. Hu, D. Tao, and M. Song. Knowledge amalgamation from heterogeneous networks by common feature learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI) , 2019

2019
[19]

M. S. Matena and C. Raffel. Merging models with fisher-weighted averaging. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, NIPS , 2022

2022
[20]

Radford, J

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748--8763. PMLR, 2021

2021
[21]

Simonyan

K. Simonyan. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014

2014 arXiv
[22]

S. P. Singh and M. Jaggi. Model fusion via optimal transport. NIPS , 33, 2020

2020
[23]

Stoica, D

G. Stoica, D. Bolya, J. B. Bjorner, P. Ramesh, T. Hearn, and J. Hoffman. Zipit! merging models from different tasks without training. In ICLR , 2024

2024
[24]

Tatro, P.-Y

N. Tatro, P.-Y. Chen, P. Das, I. Melnyk, P. Sattigeri, and R. Lai. Optimizing mode connectivity via neuron alignment. NIPS , 33:15300--15311, 2020

2020
[25]

Verma and M

N. Verma and M. Elbayad. Merging text transformer models from different initializations. ArXiv , abs/2403.00986, 2024

2024 arXiv
[26]

A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE : A multi-task benchmark and analysis platform for natural language understanding. In ICLR , 2019

2019
[27]

Wortsman, G

M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, and L. Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In 39th ICML , 2022

2022
[28]

J. Wu, D. Zhu, L. Fang, Y. Deng, and Z. Zhong. Efficient layer compression without pruning. IEEE Transactions on Image Processing , 32:4689--4700, 2023

2023
[29]

Z. Xu, K. Yuan, H. Wang, Y. Wang, M. Song, and J. Song. Training-free pretrained model merging. In CVPR , pages 5915--5925, 2024

2024
[30]

Yadav, D

P. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal. Resolving interference when merging models. In NeurIPS , New Orleans, USA, 2023. Proceedings of Machine Learning Research

2023
[32]

Zhang and Q

Y. Zhang and Q. Yang. A survey on multi-task learning. IEEE TKDE , 34(12):5586--5609, 2021

2021
[33]

Z. Zhou, Y. Yang, X. Yang, J. Yan, and W. Hu. Going beyond linear mode connectivity: The layerwise linear feature connectivity. In NIPS , volume 36, pages 60853--60877. Curran Associates, Inc., 2023

2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [31]

L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. arXiv preprint arXiv:2311.03099 , 2023

arXiv 2023

[2] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.ncblock write newline " " before.all 'output.state := FUNCTION new.nccont write " " before.all 'output.state := FUNCTION new.sentence output.state after.block = 'skip out...

[3] [2]

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page

[4] [3]

Ainsworth, J

S. Ainsworth, J. Hayase, and S. Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In ICLR , 2023

work page 2023

[5] [4]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248--255, 2009

work page 2009

[6] [5]

J. Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018

arXiv 2018

[7] [6]

Entezari, H

R. Entezari, H. Sedghi, O. Saukh, and B. Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296 , 2021

arXiv 2021

[8] [7]

Fifty, E

C. Fifty, E. Amid, Z. Zhao, T. Yu, R. Anil, and C. Finn. Efficiently identifying task groupings for multi-task learning. NIPS , 34:27503--27516, 2021

work page 2021