{"id":"70805d74-8b47-4615-a759-d3975074e895","arxiv_id":"2501.06126","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Merging aligned feed-forward sublayers into tied weights can remove over a third of a Transformer's feed-forward parameters with only small performance losses, after a short fine-tuning.","lead":"This paper compresses Transformer models by merging whole feed-forward sublayers into one shared set of weights, then fine-tuning to recover performance. It reports results close to the original models after removing over a third of feed-forward sublayers across language modeling, image classification, and machine translation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'comparable to original' claim rests on an uneven comparison: compressed ViT and OPUS-MT models are fine-tuned on task data while their uncompressed counterparts are not, per Section 4.1. A fine-tuned baseline could widen the reported gap.","rationale":"The method itself is clearly described and the GPT-2 experiments include a properly fine-tuned baseline, but the abstract extends the 'comparable performance' claim to ViT and OPUS-MT, where the comparison is confounded by unequal fine-tuning. The reader's weakest_assumption identifies exactly this issue. I agree that this is the most load-bearing concern because it directly tests the abstract's 'comparable performance' and the 99% ViT claim. The remaining issues, such as single runs without error bars and selection on validation data, are secondary; they would further reduce confidence but are not as directly tied to the central claim. The proposed baseline experiment is straightforward and would settle whether the concern lands. Therefore the reader's CONDITIONAL verdict remains appropriate, and no change is needed.","tokens_in":14909,"tokens_out":13817,"duration_ms":133544,"concrete_test":"Run the missing baselines: fine-tune the unmodified ViT model on ImageNet-1k for up to 50k steps using the Section C.2 hyperparameters, and fine-tune the unmodified OPUS-MT zh-en model on the Tatoeba training data for up to 100k steps using the Section C.3 hyperparameters. Evaluate on the same validation/test sets as Table 10. If the fine-tuned original ViT accuracy and OPUS-MT BLEU exceed the unfine-tuned originals by more than the reported gaps (79.2 vs 80.3 for ViT; 33.5 vs 35.8 for MT), then the compressed models are no longer comparable to a fairly fine-tuned original, and the paper's central claim for these tasks fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of comparable performance is evaluated unevenly for two of the three main tasks. Section 4.1 states that for ViT and OPUS-MT no fine-tuned baseline is provided because the recovery data already appears in their original training data. However, the compressed models receive substantial recovery fine-tuning, up to 100k steps for MT and 50k for ViT, as listed in Appendix C. The paper's own GPT-2 experiment demonstrates that fine-tuning the uncompressed model matters: the fine-tuned GPT-2 baseline reaches 16.16 PPL, while the merged model at 1/3 FF removal reaches 17.27. For ViT and OPUS-MT, the reported reference values (80.3 accuracy and 35.8 BLEU) are from the original models without this recovery fine-tuning. If those original models were fine-tuned with the same task data and hyperparameters, they could plausibly improve, which would increase the gap and weaken the 'comparable to original' and '99% of original performance' claims. This is not a peripheral issue: recovery fine-tuning is a core part of the pipeline, and Table 1 reports only post-fine-tuning performance for all compressed variants.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a post-training compression method for Transformer models that merges adjacent feed-forward (FF) sublayers. The method computes activation-based neuron permutations to align FF sublayers within a window, averages the aligned parameters, and ties the merged weights so that k sublayers occupy the memory of one. The authors evaluate on GPT-2 (language modeling), ViT (image classification), OPUS-MT (machine translation), and OLMo-7B with QLoRA (summarization), comparing against layer pruning and showing compatibility with quantization. The central claim is that more than a third of FF sublayers can be merged with performance comparable to the original models and better than a layer-pruning baseline.","tokens_in":15084,"tokens_out":5349,"duration_ms":49813,"significance":"If the results hold, this is a useful and simple post-training compression technique that is orthogonal to quantization and pruning. The merging equations in Section 3.3 are transparent and mathematically sound. The paper includes a fine-tuned baseline for GPT-2, releases a toolkit, and tests across three Transformer architectures and modalities, with additional robustness checks on window selection and anchor choice. The main weakness is that, for two of the three primary tasks, the comparison against the original model is uneven because the compressed models receive recovery fine-tuning while the original models do not; this is a load-bearing issue for the 'comparable performance' claim.","major_comments":[{"comment":"The 'comparable to original' claim for ViT and OPUS-MT is evaluated against non-fine-tuned baselines. Section 4.1 states that no fine-tuned baseline is provided for these models because the recovery data already appears in their original training data, but the compressed models are still fine-tuned for up to 50k steps (ViT) and 100k steps (MT) on that same data (Appendix C). The GPT-2 experiment shows that such fine-tuning can improve the uncompressed model: the fine-tuned GPT-2 baseline is 16.16 PPL, and this is the reference used in Table 10. For ViT and OPUS-MT, the reference values 80.3 accuracy and 35.8 BLEU are from the original models without this additional recovery fine-tuning. If the original models were fine-tuned under the same protocol, the gap could widen. This directly affects the validity of the central claim for two of the three main tasks. The authors should either run the fine-tuned baselines for ViT and OPUS-MT or explicitly restrict the claim and justify why fine-tuning would not change the reference numbers.","section":"Section 4.1 and Tables 10, 11"},{"comment":"The claim that 'performance is almost identical to the original model' at 1/3 FFs removed is unevenly supported by the reported metrics. In Table 10, at 4/12 FFs removed, ViT accuracy drops from 80.3 to 79.2 (about 1.1% relative), GPT-2 PPL increases from 16.16 to 17.27 (about 6.9% relative increase, where lower is better), and OPUS-MT BLEU drops from 35.8 to 33.5 (about 6.4% relative). Given the baseline fairness issue in the first comment, the 'almost identical' phrasing is too strong, especially for GPT-2 and OPUS-MT. The authors should report relative changes and qualify the comparison to account for the fact that the ViT and MT references are not recovery-fine-tuned.","section":"Section 5.1, Figure 2"}],"minor_comments":[{"comment":"The text says 'for simplicty' and should be 'for simplicity'.","section":"Appendix A"},{"comment":"The reference for Dettmers et al. (2022) is formatted as 'Gllm.int8 ()' and should be 'LLM.int8()'; the reference for Voita et al. is printed as 'V oita'.","section":"References"},{"comment":"The y-axis label reads 'BLEU on T atoeba' and should be 'BLEU on Tatoeba'.","section":"Figure 4"},{"comment":"The phrase 'apply our method alongsize 4-bit QLoRA' should be 'alongside 4-bit QLoRA'.","section":"Section 4.5"},{"comment":"The column label 'Compression' is ambiguous: it is not clear whether the values (e.g., 78%, 20%) represent the percentage of original model size remaining or the percentage reduction. Please clarify the definition in the caption or text.","section":"Table 3"},{"comment":"In Algorithm 1, the loop variable i is not passed to the COMPRESS function, so the pseudocode does not specify which window of k adjacent layers is being compressed in each iteration. This should be corrected for reproducibility.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The core method appears sound, and the manuscript is well within the scope of the journal. The main barrier to acceptance is the evaluation fairness for ViT and OPUS-MT. I recommend major revision rather than rejection because the issue is addressable by adding fine-tuned baselines or by carefully qualifying the claims. If the authors can show that fine-tuning the original ViT and OPUS-MT does not change their performance, the central claims are likely acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core result: you can take a pre-trained Transformer, align the neurons of adjacent feed-forward sublayers with permutations, average their weights, and tie them so they share one parameter set. Doing this for a third of the FF sublayers costs 1 PPL point on GPT-2, 1 accuracy point on ViT, and 2 BLEU on zh-en MT. That is the real result, and it is a decent one. The method is simple, the equations in Section 3.3 are correct, and the paper makes a fair comparison to a strong layer-pruning baseline, which it matches or beats.\n\nWhat is new is applying model-merging permutation alignment inside a single model, not across models. That is a genuine extension, and the paper does a careful job across decoder-only, encoder, and encoder-decoder architectures. The QLoRA and quantization extensions are useful, and the CKA analysis showing structured similarity among FF sublayers is a nice empirical observation.\n\nThe main soft spot is the baselines. For ViT and OPUS-MT, the original uncompressed models are not fine-tuned on the task data, while the merged models receive substantial recovery fine-tuning (up to 50k and 100k steps). The paper discloses this in Section 4.1, but it means the 'comparable to original' claim is only properly tested for GPT-2, where the fine-tuned baseline reaches 16.16 PPL versus 17.27 after merging a third of the FFs. For the other two tasks, the gap could widen if the originals got the same treatment. This does not sink the method—the comparison to the layer-pruning baseline is fair because that baseline is also fine-tuned—but the abstract's phrasing overstates the evidence.\n\nOther issues are minor: no error bars or multiple seeds, and the code link appears truncated. The sliding-window selection of which layers to merge is a form of validation-based model selection; it is standard, though the reported numbers are for the best window.\n\nOverall, this is a solid empirical paper. A serious referee would ask for fine-tuned baselines on ViT and MT and some variance estimates, but the method is well-motivated, the experiments are broad, and the core claim is convincing. I would send it to review.","headline":"A clean post-training compression method that merges feed-forward sublayers via permutation alignment; the main weakness is uneven baselines for two of three tasks, but the core result holds.","tokens_in":15701,"tokens_out":2305,"would_cite":true,"duration_ms":20015,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adjacent feed-forward sublayers in trained Transformers can be aligned, averaged, and tied into one shared sublayer, removing over a third of feed-forward layers while keeping performance after recovery fine-tuning.","keywords":["model compression","weight tying","feed-forward sublayers","permutation alignment","transformers","vision transformers","layer pruning","quantization"],"falsifier":"Fine-tune the original ViT and OPUS-MT models on the same data used for recovery fine-tuning and measure their accuracy and BLEU against the merged-and-fine-tuned models at the same compression ratio; if the fine-tuned originals outperform the merged models by more than the reported 1 accuracy-point or 2 BLEU-point gaps, the claim of comparable performance to the original models does not hold for those settings.","tokens_in":14608,"feed_emoji":"🗜️","tokens_out":11940,"duration_ms":102475,"temperature":0.7,"pith_summary":"This paper argues that a large part of a trained Transformer is redundant at the level of whole feed-forward sublayers, and that redundancy can be exploited by merging instead of pruning. The method aligns the neurons of adjacent sublayers with permutations, averages the aligned weights, and ties them so that several sublayers occupy the memory of one. Across language modeling, image classification, and machine translation, the paper reports performance close to the original models after merging more than a third of feed-forward sublayers, and better performance than a strong layer-pruning baseline at matched compression ratios. It also reports that feed-forward sublayer activations show clear blocks of high similarity, which it offers as evidence that the mergeability is grounded in the models' actual behavior.","feed_headline":"Merging, not pruning, trims 21% of a Vision Transformer","feed_subtitle":"Aligned neurons let feed-forward layers be averaged and tied, keeping accuracy while shrinking memory and disk use.","key_machinery":"The load-bearing device is the permutation-aligned sublayer merge. For a window of $k$ adjacent feed-forward sublayers, the first sublayer is the anchor; the paper collects activations just before the nonlinearity, computes a cross-correlation matrix between the anchor and each other sublayer, solves the resulting linear assignment problem for the best neuron permutation, applies that permutation to the input weights and bias and its inverse to the output weights of each non-anchor sublayer, then averages the $k$ parameter sets and ties them so they are stored once. A sliding window over all possible contiguous groups, evaluated on validation data, chooses which sublayers to merge, and a short recovery fine-tuning step restores task performance. For SwiGLU feed-forward blocks the same recipe operates on the gated-product features.","core_discovery":"On the paper's own terms, the central discovery is that post-training merge-and-tie compression of feed-forward sublayers works across decoder-only, encoder, and encoder-decoder Transformers. For example, merging four of twelve ViT feed-forward sublayers and fine-tuning removes over 21% of total parameters while keeping 99% of original ImageNet accuracy; merging twelve of thirty-six GPT-2 sublayers raises test perplexity by roughly one point, and merging four of twelve OPUS-MT sublayers drops BLEU by about two points. In each case the permutation-aligned merge beats the no-permutation version and matches or beats a layer-dropping baseline. The paper further shows that feed-forward sublayer outputs are often highly similar under CKA even when isolated from residual connections, a pattern it does not find in attention sublayers.","pith_inferences":["Because the reported CKA similarity forms contiguous blocks, a cheap selection rule that scores windows by average activation similarity could replace exhaustive validation, extending the method to much deeper models.","Combining the merge step with quantization changes the error surface in a way the experiments do not isolate; comparing merged-then-quantized models against pruned-then-quantized models at the same storage budget would separate the two sources of error drift.","If feed-forward sublayer similarity is a general property of trained Transformers rather than a quirk of these three architectures, the align-average-tie recipe could transfer to other architectures with repeated sublayers, though that transfer is outside the paper's experiments."],"forward_implications":["Merging can be applied to an already-trained model without retraining from scratch, so it fits into existing deployment pipelines as a post-training step.","Because merged weights are tied rather than zeroed, the compression is realized immediately in memory and on disk, and can be combined with quantization for roughly doubled savings at fixed performance.","A validation-selected layer-dropping baseline is weaker at the same compression ratios, so layer redundancy may be better exploited by merging than by deletion.","The observed high similarity between feed-forward sublayer activations suggests mergeability is not an artifact of residual connections and may be a general property of trained Transformers.","The method carries over to SwiGLU-based models, so it can shrink current large language models that use this nonlinearity."],"supporting_citations":[{"why":"Defines the Transformer architecture and its feed-forward sublayers, the compression target of the paper.","marker":"Vaswani et al., 2017"},{"why":"Introduces cross-correlation based permutation alignment of neurons, the core of the alignment step.","marker":"Li et al., 2015"},{"why":"Formulates neuron alignment as a linear assignment problem, which the paper solves for each sublayer pair.","marker":"Tatro et al., 2020"},{"why":"Supplies the Jonker-Volgenant assignment solver used to compute the neuron permutations.","marker":"Crouse, 2016"},{"why":"Shows that a single widened tied feed-forward block can support translation, motivating the usefulness of sharing feed-forward parameters.","marker":"Pires et al., 2023"},{"why":"Provides the Centered Kernel Alignment measure used to demonstrate high sublayer similarity.","marker":"Kornblith et al., 2019"},{"why":"Supplies GPT-2 as the language-modeling testbed.","marker":"Radford et al., 2019"},{"why":"Supplies the Vision Transformer used for image classification, the 21% parameter-reduction result.","marker":"Dosovitskiy et al., 2020"},{"why":"Supplies the OPUS-MT translation model tested in the machine-translation experiments.","marker":"Tiedemann and Thottingal, 2020"},{"why":"Represents the layer-dropping approach whose performance the merging method is compared against.","marker":"Gromov et al., 2024"}],"fun_headline_variants":["Merge feed-forward layers to shrink Transformers","Compress Transformers by merging, not pruning","Merging sublayers cuts ViT parameters by 21%","Tie similar layers to compress Transformers with little loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison for ViT and machine translation assumes the original, uncompressed models are a fair baseline even though only the compressed models receive recovery fine-tuning; if the originals were fine-tuned on the same data, the performance gap could be larger.","fun_headline_variants_meta":{"raw":{"variants":["Merge feed-forward layers to shrink Transformers","Compress Transformers by merging, not pruning","Merging sublayers cuts ViT parameters by 21%","Tie similar layers to compress Transformers with little loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1459,"prompt_tokens":898,"completion_tokens":561,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":496}},"tokens_in":514,"tokens_out":561,"duration_ms":4861,"temperature":1.0,"reasoning_tokens":496,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:05:34.616941+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune the original ViT and OPUS-MT models on the same data used for recovery fine-tuning and measure their accuracy and BLEU against the merged-and-fine-tuned models at the same compression ratio; if the fine-tuned originals outperform the merged models by more than the reported 1 accuracy-point or 2 BLEU-point gaps, the claim of comparable performance to the original models does not hold for those settings.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Formulates neuron alignment as a linear assignment problem, which the paper solves for each sublayer pair."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that a single widened tied feed-forward block can support translation, motivating the usefulness of sharing feed-forward parameters."}],"review_version":1}