RECORD
On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks
Machine Learning · Submitted 29 Oct 2021
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Our experiments indicate that CLAFusion, with an extra finetuning process, improves the accuracy of residual networks on the CIFAR10, CIFAR100, and Tiny-ImageNet datasets.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Layer-wise model fusion via optimal transport, named OTFusion, applies soft neuron association for unifying different pre-trained networks to save computational resources. While enjoying its success, OTFusion requires the input networks to have the same number of layers. To address this issue, we propose a novel model fusion framework, named CLAFusion, to fuse neural networks with a different number of layers, which we refer to as heterogeneous neural networks, via cross-layer alignment. The cross-layer alignment problem, which is an unbalanced assignment problem, can be solved efficiently using dynamic programming. Based on the cross-layer alignment, our framework balances the number of layers of neural networks before applying layer-wise model fusion. Our experiments indicate that CLAFusion, with an extra finetuning process, improves the accuracy of residual networks on the CIFAR10, CIFAR100, and Tiny-ImageNet datasets. Furthermore, we explore its practical usage for model compression and knowledge distillation when applying to the teacher-student setting.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Layer-wise model fusion via optimal transport, named OTFusion, applies soft neuron association for unifying different pre-trained networks to save computational resources. | background 0.99 | 0.10 | 0.05 | asserted 1.00 |
| s1 | While enjoying its success, OTFusion requires the input networks to have the same number of layers. | limitation 0.36 | 0.10 | 0.03 | asserted 1.00 |
| s2 | To address this issue, we propose a novel model fusion framework, named CLAFusion, to fuse neural networks with a different number of layers, which we refer to as heterogeneous neural networks, via cross-layer alignment. | contribution 0.99 | 0.85 | 0.18 | asserted 0.99 |
| s3 | The cross-layer alignment problem, which is an unbalanced assignment problem, can be solved efficiently using dynamic programming. | method 0.84 | 0.28 | 0.06 | asserted 0.99 |
| s4 | Based on the cross-layer alignment, our framework balances the number of layers of neural networks before applying layer-wise model fusion. | method 0.99 | 0.84 | 0.12 | asserted 1.00 |
| s5 | Our experiments indicate that CLAFusion, with an extra finetuning process, improves the accuracy of residual networks on the CIFAR10, CIFAR100, and Tiny-ImageNet datasets. | result 1.00 | 0.96 | 0.51 | hedged possibility 0.88 |
| s6 | Furthermore, we explore its practical usage for model compression and knowledge distillation when applying to the teacher-student setting. | contribution 0.89 | 0.90 | 0.08 | asserted 0.66 |
What was examined, and what was not
- arXiv listing abstract v3
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Layer-wise model fusion via optimal transport, named OTFusion, applies soft neuron association for unifying different pre-trained networks to save computational resources. While enjoying its success, OTFusion requires the input networks to have the same number of layers. To address this issue, we propose a novel model fusion framework, named CLAFusion, to fuse neural networks with a different number of layers, which we refer to as heterogeneous neural networks, via cross-layer alignment. The cross-layer alignment problem, which is an unbalanced assignment problem, can be solved efficiently using dynamic programming. Based on the cross-layer alignment, our framework balances the number of layers of neural networks before applying layer-wise model fusion. Our experiments indicate that CLAFusion, with an extra finetuning process, improves the accuracy of residual networks on the CIFAR10, CIFAR100, and Tiny-ImageNet datasets. Furthermore, we explore its practical usage for model compression and knowledge distillation when applying to the teacher-student setting.
Human review
References
How the literature uses it
4 arXiv papers cite this in 10 sentences
Papers citing it · 9 sentences without a confident label
Cites it as background 1 paper
-
Some researchers propose more complex methods to align the parameters of different models and merge them [ref] .
LM-Cocktail: Resilient Tuning of Language Models via Model Merging · Model Merging · 0.98
What it cites · 3 sentences without a confident label
Uses the method or tool of 1 paper
-
We employ the method in Hinton et al. [ref] to match the logit distribution of the student model to that of the teacher model.
Distilling the Knowledge in a Neural Network · Knowledge distillation · 1.00
Uses data or values from 1 paper
-
We first fuse a RESNET34 and a RESNET18 trained on CIFAR10, CIFAR100 [ref] and Tiny-ImageNet datasets [ref] .
A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets · Experiments · 0.95
Cites as background 1 paper
-
The first approach, knowledge distillation, is a machine learning technique that transfers knowledge from large networks (teachers) into a smaller one (student) [ref] .
Distilling the Knowledge in a Neural Network · Introduction · 0.97
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.