{"id":"987cb8b0-5468-4800-a096-93c7f0dd10ad","arxiv_id":"2501.00061","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A training-free framework merges heterogeneous neural networks of different depth and width by segmenting deeper models and elastically zipping neurons, reaching accuracy close to homogeneous merging.","lead":"This paper proposes a training-free way to merge neural networks that have different depths or widths, by aligning layers and zipping neurons before averaging weights. It is useful because model merging currently requires identical architectures, and removing that constraint would let practitioners combine off-the-shelf models without retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 6's identity padding for the shallow model does not preserve its feature map unless consecutive alignment permutations are equal; the depth-merge derivation is internally inconsistent.","rationale":"The Reader identified layerwise linear feature connectivity as the weakest assumption. My stress-test finds a more concrete and more fundamental problem: even granting that assumption, the identity-padding construction in Eq. 5/6 does not preserve the shallow model's feature path unless all padded-layer permutations are equal, a constraint that the paper neither states nor enforces. This invalidates the derivation of the depth-merge weight formula, which is the main theoretical support for the central claim. I do not, however, reject the empirical results: Tables I, II, and IV may still demonstrate that the heuristic works in practice, and the public code might accidentally impose the needed constraint. The correct verdict is CONDITIONAL: the paper should either prove the permutation constraint, add it explicitly to the algorithm, or replace Eq. 5/6 with a correct derivation. The width-merging under-specification noted by the Reader is also real, but the depth derivation flaw is more load-bearing because it affects the claimed mechanism for one of the paper's two central contributions.","tokens_in":11203,"tokens_out":6626,"duration_ms":69076,"concrete_test":"Instrument the released depth-merge code on a minimal case: Model A has two linear layers, Model B has one linear layer. After running Algorithm 1 or 2, record the two B-side alignment permutations P_1^B and P_2^B used for the original and padded layers. Check whether P_2^B (P_1^B)^{-1} equals the identity matrix. Then feed the same Task B input to the original B model and to the merged model and compare the B-side output at the final layer. If the outputs differ substantially, or if the per-layer B permutations are not constrained to be equal, Eq. 6's identity-padding derivation is invalid and the paper must state and prove the required constraint.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The depth-heterogeneous derivation in Sec. III-B rests on padding the shallow model B with identity layers and then applying one-by-one weight averaging (Eq. 6). In Eq. 5 the B branch is written as P_l^B I (P_{l-1}^B)^{-1} ... P_1^B W^B x. For a true identity layer, the aligned contribution is P_i^B I (P_{i-1}^B)^{-1} = P_i^B (P_{i-1}^B)^{-1}, which is not the identity unless P_i^B = P_{i-1}^B. Nothing in Eq. 4 or in the segment/layer alignment algorithms constrains the B-side permutations of padded identity layers to be equal. As a result, the repeated 'identity' layers actively re-permute the shallow model's feature, and the product does not collapse to P_l^B W^B x. The same issue recurs in the residual extension in Supp. S1, where the zero-weight B layers are still associated with per-layer permutation matrices. This is not a failure of the cited layerwise linear feature connectivity assumption; it is an algebraic flaw in the padding construction itself. Eq. 6 therefore does not follow from the preceding feature-averaging argument, and the theoretical grounding for depth merging is missing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-free framework for merging neural networks that differ in depth and width. For depth heterogeneity, the deeper model is segmented so that each segment corresponds to one layer of the shallower model, and the segments are aligned and merged via layer-wise weight averaging with identity padding for the shallower model. For width heterogeneity, an elastic neuron zipping algorithm projects neurons of different widths into a common dimensional space and merges the most similar neurons. The method is evaluated on CIFAR-10/100, ImageNet, and GLUE with ResNet, VGG, and BERT architectures, with the central claim that heterogeneous merging achieves performance comparable to homogeneous merging without retraining.","tokens_in":11511,"tokens_out":12482,"duration_ms":105913,"significance":"If the claims hold, this would be a useful practical contribution to model merging, enabling the combination of models with different architectures without costly retraining. The paper includes public code and experiments across vision and NLP tasks, which strengthens its potential impact. However, the theoretical derivation for depth merging is algebraically flawed, the width experiments lack a homogeneous baseline, and some experimental claims are not fully supported by the reported tables. These issues currently limit the reliability of the contribution despite the encouraging empirical results.","major_comments":[{"comment":"The derivation of the depth-merge formula is invalid as a justification for the weight averaging in Eq. 6. The first equality in Eq. 5 treats the merged feature f^*_{l-1} as if it were the A-branch feature, but the merged feature is the result of applying summed weights to the previous merged feature. The expansion to the second line omits cross-terms: for l=2, the merged forward pass is (W^A_2 + P^B_2 (P^B_1)^{-1})(W^A_1 + P^B_1 W^B)x, which contains W^A_2 P^B_1 W^B x and P^B_2 (P^B_1)^{-1} W^A_1 x; these are absent from the derivation. The layerwise linear feature connectivity of [31] concerns interpolation between two models' feature maps, not the composition of summed weights, so Eq. 6 does not follow from the preceding feature-averaging argument. Note that the specific concern that the identity-padded product does not collapse is not correct: the B-branch product telescopes to P^B_l W^B x regardless of the permutation matrices. The real problem is the unjustified parallel-branch interpretation of a sequentially composed merged model. Please either correct the derivation or explicitly reframe Eq. 6 as a heuristic.","section":"Sec. III-B, Eq. 5-6"},{"comment":"The GLUE experiments create depth heterogeneity by repeating every even-numbered layer of a 12-layer BERT to obtain a 17-layer model. This produces a deeper model whose added layers are exact duplicates, so the segmentation and alignment problem becomes artificially easy: the repeated layers form segments that are trivially aligned. The NLP results therefore do not demonstrate that the method handles genuinely different Transformer depths. Please test on naturally different architectures (e.g., BERT-base vs BERT-large or models with different layer counts from pretrained checkpoints) or temper the claim about NLP generality.","section":"Sec. IV-A, Models"},{"comment":"The text states that compared to vanilla averaging, lower loss barriers are observed for homogeneous and heterogeneous merging, but in Tab. IV the barriers for MNLI-mm (0.65 and 0.67) and QNLI (0.76 and 0.71) are higher than vanilla averaging (0.59 and 0.67). The claim is not supported by the per-task results and should be revised or supplemented with an aggregate statistic. Also, the 'Error' column is not defined; please state the metric used for each task (e.g., 1-accuracy for classification, MSE for STS-B).","section":"Sec. IV-C, Tab. IV"},{"comment":"The 'Homo Avg' entry in Tab. II reports a joint accuracy of 65.25, which is substantially higher than the single-task joint accuracies (36.70 and 37.28) and inconsistent with the pattern in Tab. I where Homo Avg is the lowest-performing method. This suggests a mislabeled entry or a typo. Please check the result and clarify what is being reported.","section":"Sec. IV-B, Tab. II"},{"comment":"The width-heterogeneous merging experiments in Tab. III contain no homogeneous-merging baseline (e.g., merging two models of identical width). Since the abstract claims performance comparable to homogeneous merging, and the width experiments are a central part of the framework, the absence of such a baseline leaves the claim untested for width heterogeneity. Please add homogeneous counterparts or adjust the claim accordingly.","section":"Sec. IV-B, Tab. III"}],"minor_comments":[{"comment":"The first line of Eq. 5 writes the B term as P^B_1 W^B x, which is inconsistent with the product expansion in the second line; this is likely a typo and should be fixed.","section":"Sec. III-B, Eq. 5"},{"comment":"In the backtrace loops of both algorithms, the variables n and m are not defined; they should be replaced with |L_A| and |L_B| (or otherwise explicitly defined).","section":"Supp. S2, Algorithm 1 and 2"},{"comment":"The loss-barrier definition is attributed to Frankle et al. [7], but reference [7] (Frankle and Carbin, 2018) is 'The lottery ticket hypothesis' and does not define the loss barrier; please cite the appropriate source (e.g., Frankle et al., 2020 or Entezari et al., 2021).","section":"Sec. IV-C"},{"comment":"The sentence 'the models merged with depth-heterogeneous architectures not only achieve higher average performance compared to the weight average of depth-homogeneous models but also exhibit similar performance to the models merged with depth-homogeneous architectures' is confusing and appears to contradict itself; please rephrase for clarity.","section":"Sec. IV-B"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be from the same group as MuDSC [27], and the self-citation is appropriate. The stress-test claim about permutation matrices not collapsing is technically incorrect, as the B-branch product telescopes; the more serious issue is the cross-term omission in the derivation, which the authors should address. The experimental issues in Tables II and IV and the lack of a width-baseline are community-facing and should be corrected before acceptance. The paper's core idea is promising, but the current manuscript does not yet provide a sound theoretical grounding or fully supported experimental claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the quick take: this is the first paper to attempt training-free merging of models that differ in both depth and width, and the depth experiments are genuinely impressive. But the theoretical derivation for depth merging has a real algebraic flaw, and the width method is under-specified. I'd send it to review, but the authors need to fix or retract the derivation.\n\nWhat's new: the problem is well chosen. Previous merge methods (Git Re-Basin, Zipit!, OTFusion) require matching widths and mostly matching depths. The segment-wise and layer-wise alignment for depth, plus elastic neuron zipping for width, are original. The empirical scope is solid: CIFAR, ImageNet, and GLUE, with comparisons to vanilla averaging, alignment-based, and zipping baselines. Depth-heterogeneous merging lands close to homogeneous merging in most settings. Code is available—that's real evidence.\n\nThe soft spots, in order of severity. First, Eq. 5–6 don't hold as written. The identity padding for the shallow model uses per-layer permutation matrices, but the aligned contribution of a padded identity layer is P_i^B (P_{i-1}^B)^{-1}, which is not the identity unless consecutive permutations are identical. Nothing in the alignment objective constrains them. So the B-branch product in Eq. 5 does not collapse to P_l^B W^B x, and Eq. 6 does not follow. This isn't a quibble about linear feature connectivity; it's an algebraic error in the padding construction. The same issue recurs in the residual extension in Supp. S1. The empirical results might still be valid—the method may just work as a heuristic—but the paper's theoretical grounding is currently missing.\n\nSecond, the width method is too vague. We get a picture and a description, but no formal algorithm for how neurons are zipped and weights are projected to a common width. And Table III has no homogeneous merging baseline for the same widths, so the claim that width-heterogeneous merging is comparable to homogeneous merging isn't actually demonstrated.\n\nThird, the derivation leans on layerwise linear feature connectivity [31] without checking whether it holds for ReLU, BatchNorm, or residual blocks in these heterogeneous settings. That's a secondary concern given the algebraic issue, but worth flagging.\n\nWho's this for? Anyone working in model merging or model reuse. It's a useful problem and the results are promising. But I wouldn't take the theory at face value. If the authors can fix the derivation—or explicitly reframe the method as a heuristic with empirical support—this becomes a solid paper. As is, it deserves peer review, but with a request for major revision and a careful re-derivation.","headline":"First real attempt at training-free heterogeneous model merging; depth experiments are promising, but the depth-merge derivation has an algebraic flaw and the width method is under-specified.","tokens_in":11970,"tokens_out":7315,"would_cite":false,"duration_ms":63011,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Neural networks that differ in depth and width can be merged without retraining, matching homogeneous merging in the paper's experiments.","keywords":["model merging","heterogeneous models","depth heterogeneity","width heterogeneity","training-free merging","neuron zipping","layer alignment","weight averaging"],"falsifier":"Run a pair of differently deep models on the same task, apply the proposed alignment, and compare the merged model's layerwise feature maps with the true averaged feature maps; if the mismatch grows with segment length, the core identity fails. A sharper test is to insert an activation or normalization change known to destroy linear mode connectivity and check whether depth-heterogeneous merging collapses while homogeneous merging does not.","tokens_in":11053,"feed_emoji":"🔀","tokens_out":7938,"duration_ms":79075,"temperature":0.7,"pith_summary":"This paper tackles a gap in model merging: prior weight-averaging and align-then-average methods require the two networks to have the same architecture, while this work fuses models that differ in layer count and layer width. The proposed framework handles depth by segmenting the deeper model so each segment is averaged against one layer of the shallower model, and handles width by an elastic neuron zipping algorithm that collapses pairs of similar neurons until both models have a common width. The authors report that, across image classification and text-understanding benchmarks, these training-free, heterogeneous merges perform about as well as merging identical architectures. If the claim holds, task-specific models trained on different backbones can be combined into one model by linear-algebra operations on weights and activations alone.","feed_headline":"No retraining: models with different depths and widths can be merged","feed_subtitle":"Layer alignment and neuron zipping make heterogeneous merging match homogeneous performance on vision and NLP.","key_machinery":"The load-bearing identity is the weight-average construction of Eq. (6): once a deep segment is paired with one shallow layer, the shallow model is extended with identity layers (non-residual models) or zero-weight residual layers (residual models) so that averaged features equal the features of averaged weights, under layerwise linear feature connectivity. The segment boundaries themselves are chosen by two heuristics: segment-wise model alignment, which maximizes CKA similarity between each segment's output and the corresponding shallow layer's output, and layer-wise model alignment, which additionally considers the internal layers of the segment. Width heterogeneity is handled by elastic neuron zipping, which merges the most similar neurons across the two models until a target width is reached. These pieces convert heterogeneous merging into repeated applications of standard permutation-alignment and weight averaging.","core_discovery":"The paper's central claim is that structural heterogeneity does not require retraining: after alignment, a heterogeneous merge is just a sequence of weight averages. For depth, the deeper model is partitioned into as many segments as the shallower model has layers, using representation similarity to choose segment boundaries; then the shallow model is padded with identity layers (or zero-weight residual layers) so each segment and its counterpart can be averaged layer by layer through Eq. (6). For width, an elastic neuron zipping algorithm computes pairwise feature similarity between neurons of the two models and greedily merges the most similar neurons until a chosen width is reached, projecting all weights into a common dimensional space. The paper argues this yields merged models with performance comparable to depth-homogeneous and width-homogeneous merging, based on experiments on CIFAR, ImageNet, and GLUE.","pith_inferences":["This suggests that a practical model zoo could be merged without standardizing on a backbone; the hard part becomes choosing segment boundaries and target widths, not training.","A natural stress test is to apply the same recipe to models whose normalization layers or activation functions differ, since the linear-feature assumption is the component most likely to break.","The neuron-zipping routine is essentially a greedy projection onto a common width; one testable extension is to use it for partial merges or for merging more than two heterogeneous models at once.","Choosing segments by maximal CKA similarity is a heuristic, so optimizing the segmentation against the final merged model's loss rather than pairwise similarity could change which alignment wins."],"forward_implications":["Two models with different layer counts can be merged by first segmenting the deeper model and then applying one weight average per segment, with no gradient updates.","Two models with different layer widths can be merged by neuron zipping onto a common width, so users do not need to retrain either model to a shared architecture.","The paper's reported joint and per-task accuracies for heterogeneous merges sit close to those of homogeneous merges, so architecture mismatch alone need not block model reuse.","The method extends to residual backbones and transformer-based text models, making the recipe applicable beyond simple feed-forward architectures.","Because both alignment methods are training-free, the construction cost is essentially the cost of forward passes for feature similarity plus one averaging step per segment."],"supporting_citations":[{"why":"Supplies the layerwise linear feature connectivity assumption on which the depth-merge weight averaging in Eq. (5)-(6) rests.","marker":"[31]"},{"why":"Provides the neuron-merging baseline that the elastic zipping algorithm generalizes from equal-width to arbitrary-width models.","marker":"[21]"},{"why":"Supplies CKA, the representation similarity used to choose segment boundaries and to measure layer alignments.","marker":"[14]"},{"why":"Supplies the permutation-alignment framework and homogeneous merging baseline that the proposed method extends.","marker":"[1]"},{"why":"States the permutation-invariance assumption that makes align-then-average merging possible in the first place.","marker":"[4]"},{"why":"Provides evidence that adjacent layers have similar representations and that groups of layers can be replaced by fewer layers, motivating the segmentation strategy.","marker":"[26]"},{"why":"Provides the homogeneous text-transformer merging method used as the baseline for the GLUE experiments.","marker":"[23]"}],"fun_headline_variants":["Merge models of different depths and widths without retraining","Training-free merging for models of varying depth and width","No retraining, no same shape: merge heterogeneous models","Align layers, zip neurons: merge models without retraining","Heterogeneous model merging that requires no training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole depth-merging construction assumes that a layer's output moves in a straight line as its weights are blended, and that a grouped stack of layers keeps this straight-line behavior once the shallower model is padded with identity layers; if real networks' activations, normalization, or residual shortcuts break that straight-line behavior, the depth merge has no support.","fun_headline_variants_meta":{"raw":{"variants":["Merge models of different depths and widths without retraining","Training-free merging for models of varying depth and width","No retraining, no same shape: merge heterogeneous models","Align layers, zip neurons: merge models without retraining","Heterogeneous model merging that requires no training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000321,"raw_usage":{"total_tokens":1798,"prompt_tokens":928,"completion_tokens":870,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":793}},"tokens_in":544,"tokens_out":870,"duration_ms":7325,"temperature":1.0,"reasoning_tokens":793,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:24:08.920996+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a pair of differently deep models on the same task, apply the proposed alignment, and compare the merged model's layerwise feature maps with the true averaged feature maps; if the mismatch grows with segment length, the core identity fails. A sharper test is to insert an activation or normalization change known to destroy linear mode connectivity and check whether depth-heterogeneous merging collapses while homogeneous merging does not.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies CKA, the representation similarity used to choose segment boundaries and to measure layer alignments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"States the permutation-invariance assumption that makes align-then-average merging possible in the first place."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence that adjacent layers have similar representations and that groups of layers can be replaced by fewer layers, motivating the segmentation strategy."},{"cited_title":"Stoica, D","cited_arxiv_id":null,"evidence_quote":"Provides the homogeneous text-transformer merging method used as the baseline for the GLUE experiments."}],"review_version":1}