{"id":"3122335f-8a73-40cf-a587-f58149b4dd03","arxiv_id":"2501.00530","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Merging two GPT-2 variants with B-spline-blended hidden states plus autoencoders yields a single model that keeps both English and French perplexity closer to the best expert than linear interpolation.","lead":"This preprint describes a method for merging two language models into one by blending their inner representations with learned coefficients and small autoencoders. The authors say the merged model keeps much of each original model's ability, and they test it by mixing an English and a French GPT-2.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim of superiority over linear interpolation and task arithmetic rests on 'perplexity during the last epoch of training' (Sec. 4.2.2), not on held-out evaluation; the trained autoencoder may simply memorize the training data.","rationale":"The reader correctly locates the central claim in Section 4.2.2, but the most load-bearing weakness is not primarily the label-free inference assumption from Section 3.5.4: it is that the headline numbers appear to be training perplexity, not held-out evaluation. A trained module with roughly 1.7 million parameters per autoencoder will naturally achieve lower training perplexity than static, untrained baselines, so the reported gap of 13 to 14 perplexity points may reflect capacity or memorization rather than a genuine merging advantage. The only clean held-out results in the paper (Table 1) compare the merged model to the base and fine-tuned models, not to the interpolation baselines; those results show the merged model is worse than the relevant expert on each language (English 43.14 vs base 28.88; French 33.20 vs fine-tuned 29.57), which is consistent with a compromise but does not validate the Section 4.2.2 claim. The label-free mechanism is also untested, but it is secondary because the central comparative claim fails regardless of whether labels are needed if the reported metric is training perplexity. A concrete held-out test with repeated runs, per-language reporting, and a no-autoencoder control would settle whether the claimed advantage survives. The reader's conditional verdict remains appropriate, pending that evidence.","tokens_in":10734,"tokens_out":7092,"duration_ms":73668,"concrete_test":"Re-run the Section 4.2.2 comparison on the 10% held-out validation split (and a fresh test split) after the final epoch, with identical tokenization and decoding, for M-PPL, I-PPL, and TA-PPL across at least three independent runs with mean plus or minus standard deviation. Include a control that trains the B-spline blending and language-model loss without any autoencoder or reconstruction target, so the contribution of the autoencoder can be separated from the effect of adding many trainable parameters. If the held-out gap is not reproduced, or the no-autoencoder control matches M-PPL, the central claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.2.2 reports the central comparison as 'perplexity ... during the last epoch of training': M-PPL=47.01 vs I-PPL=60.29 and 61.30. This is not a held-out metric. The method trains roughly 1.7 million parameters per autoencoder (Section 4.1) on the same combined corpus, whereas the linear-interpolation and task-arithmetic baselines are described as static merges with no training or extra parameters. Lower training perplexity can therefore be produced by overfitting or memorization and does not by itself demonstrate that the superposition mechanism generalizes. No error bars, seeds, or per-language held-out numbers for these baselines are given, and the validation split from Section 4.1 is not reused for this comparison. Unless the same gap is shown on held-out data with the baselines given the same evaluation protocol, the central claim is unsupported as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a model-merging method in which a base and a fine-tuned transformer are combined by interpolating their per-layer hidden states with B-spline-generated coefficients, with autoencoders inserted at selected layers to reconstruct either the base or fine-tuned hidden state. The training objective combines language modeling, reconstruction, and optional alpha-regularization losses. Experiments on English/French GPT-2 models compare the merged model's perplexity and accuracy with linear interpolation and task arithmetic, and analyze neuron sparsity, polysemanticity, and hidden-state geometry. The central quantitative claim is that the autoencoder-merged model reaches lower perplexity (47.01) than linear interpolation (60.29) or task arithmetic (61.30) at the last training epoch.","tokens_in":11011,"tokens_out":5087,"duration_ms":48011,"significance":"The idea of merging two models through a low-cost learned interpolation with autoencoder reconstruction is interesting and potentially practical, and the paper is honest in listing limitations, including the absence of intra-input switching. However, the significance of the reported results is not yet established: the headline comparison is computed on training data, the label-free inference mechanism is unvalidated, and the paper vacillates between hidden-state blending and parameter-space merging. If the authors supply held-out evaluations with a clear protocol and resolve the method inconsistency, the contribution could be a modest but useful addition to the model-merging literature.","major_comments":[{"comment":"The central comparison is reported as perplexity “during the last epoch of training,” not on held-out data. The validation split described in Section 4.1 is not used for this comparison, no seeds or error bars are given, and the protocol for the static interpolation and task-arithmetic baselines across epochs is unclear. Since the merged method trains roughly 1.7 million extra parameters per autoencoder, lower training perplexity may reflect memorization rather than generalization. Please report held-out perplexity and accuracy on the validation or test set for all methods under an identical protocol, ideally with multiple seeds and error bars.","section":"Section 4.2.2, Figure 2, Section 4.1"},{"comment":"The autoencoder is trained with domain labels to decide whether to reconstruct h_base or h_fine, but labels are not available at inference. The paper assumes the autoencoder will infer the correct expert state from the blended hidden state alone; no experiment isolates this mechanism. Please add an experiment that measures reconstruction error of the merged model's output relative to h_base versus h_fine for English and French inputs without using labels, or compare with an oracle-label upper bound.","section":"Section 3.5.4 and Sections 3.3.2, 4.3.1"},{"comment":"The method is defined inconsistently. Section 3.2.2 and Figure 1 blend hidden states h_base_l and h_fine_l, while Algorithm 1 first merges parameters theta_l = (1-alpha)theta_base + alpha theta_fine and then runs the autoencoder on the activations of the merged parameter set. These are different operations and should lead to different results. Please specify which forward pass was actually trained and evaluated.","section":"Section 3.2.2, Figure 1, Appendix Algorithm 1"},{"comment":"The abstract claims the method “supports dynamic switching between model states during inference,” but Section 5 states that the approach “does not support dynamic state switching within a single input.” This is a direct contradiction and should be resolved by removing the overclaim or by demonstrating the dynamic-switching scenario.","section":"Abstract, Section 3.3.1, Section 5"}],"minor_comments":[{"comment":"The heading contains a typo: “learing” should be “learning,” and “F orgetting” has a stray space.","section":"Section 2"},{"comment":"The task arithmetic perplexity is labeled “I-PPL=61.30”; this appears to be a typo for “T-PPL.”","section":"Section 4.2.2"},{"comment":"The table does not state whether the perplexities are on the validation split, the training split, or a separate test set, and no confidence intervals are given.","section":"Table 1"},{"comment":"The figure lacks a description of how many epochs the static baselines were run for; linear interpolation and task arithmetic do not train, so it is unclear what “perplexity evolution” means for them.","section":"Figure 2"},{"comment":"Section 3.5.3 says backpropagation updates “alpha and the autoencoder parameters,” but Algorithm 1 updates only the control points and autoencoder parameters; the set of trainable variables should be stated consistently.","section":"Section 3.5.3 and Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"I see no conflict of interest. The paper should be reconsidered after the authors provide held-out evaluation for the central comparison, validate the label-free reconstruction mechanism, and clarify whether the trained model performs hidden-state blending or parameter-space merging."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a new combination - B-spline layer-wise blending of hidden states plus an autoencoder reconstruction target, trained jointly with both source models frozen - and the per-language held-out numbers in Table 1 show the merged model legitimately sits between the base and fine-tuned models. The specific architecture is not in the cited literature, so the novelty claim is fair.\n\nThe soft spot is load-bearing. The central comparison in Section 4.2.2 reports 'perplexity ... during the last epoch of training': M-PPL=47.01 versus I-PPL=60.29 and I-PPL=61.30. That is training-data perplexity. The autoencoders add roughly 1.7 million trainable parameters per layer and are trained on the combined corpus; the linear interpolation and task arithmetic baselines are static merges with no training and no extra parameters. Lower training perplexity can simply be overfitting. No error bars, no seeds, and no held-out evaluation of the baselines under the same protocol, so the claimed advantage is unsupported as written. The stress-test note is right.\n\nTwo further problems. First, training uses domain labels to pick the reconstruction target (Section 3.5.4), but inference has no labels. The paper assumes the autoencoder will infer the correct expert state from the blended hidden state alone, and no experiment isolates that mechanism. Second, the abstract claims 'dynamic switching between model states during inference', but the Limitations explicitly say the approach 'does not support dynamic state switching within a single input.' That is a direct contradiction.\n\nThe 'mixture of experts' framing overstates things - there is no routing, just a static blend plus reconstruction. The polysemantic-neuron analysis uses a threshold the authors admit is unreliable, and the clustering-based diversity analysis is suggestive but informal.\n\nWhat is genuinely good: the method is clearly described, the objective is explicit, and Table 1 does evaluate on held-out data for the three models, showing a real (if modest) bridging effect. The idea of reconstructing one source model's hidden state after blending is worth exploring.\n\nVerdict: this deserves a serious referee, but with major revision required. The next version should report held-out perplexity for all methods under an identical protocol, average over multiple seeds, and include an experiment testing whether the autoencoder can pick the right target without labels. If the training-perplexity gap disappears on held-out data, the main claim disappears. I would not cite it yet, but I would send it back for revision rather than reject it out of hand.","headline":"New merging idea, but the headline comparison is measured on training perplexity, so the main claim is unsupported as written.","tokens_in":11490,"tokens_out":3294,"would_cite":false,"duration_ms":27958,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Merging two GPT-2 checkpoints by B-spline-blending hidden states and reconstructing them with autoencoders preserves both languages better than linear interpolation or task arithmetic.","keywords":["mixture of experts","model merging","catastrophic forgetting","B-spline blending","autoencoder reconstruction","transformer hidden states","polysemantic neurons","parameter-efficient fine-tuning"],"falsifier":"Train the merging procedure with domain labels withheld and compare the unlabeled merged model's perplexity against linear interpolation on both languages; or feed the trained model a mixed English–French sequence and compare its reconstructed layer-4 states with the base and fine-tuned target states. If unlabeled training no longer beats the interpolation perplexity, or if the reconstructed states do not align with the correct expert model on mixed inputs, the adaptive label-free switching the method depends on is not actually happening.","tokens_in":10516,"feed_emoji":"🔀","tokens_out":10560,"duration_ms":88674,"temperature":0.7,"pith_summary":"This paper tries to establish that two transformers—a base GPT-2 and the same model fine-tuned on French—can be merged into a single compact checkpoint without retraining either one. The merger blends each layer's hidden states with a learned B-spline coefficient and passes the blend through small autoencoders trained to reconstruct the base or fine-tuned state depending on the input's domain. On the reported experiments the merged model keeps English perplexity at 43.14 (base: 28.88) and French at 33.20 (fine-tuned: 29.57), and its overall perplexity of 47.01 beats linear interpolation (60.29) and task arithmetic (61.30). The intended payoff is a lightweight alternative to mixture-of-experts: two specializations live superposed in one parameter space and are recovered on demand, mitigating catastrophic forgetting without separate expert modules.","feed_headline":"Autoencoder-blended GPT-2 beats linear merging on two languages","feed_subtitle":"One checkpoint keeps English near base level and French near fine-tuned level, with no separate experts.","key_machinery":"Per layer $l$, the blended hidden state is $h_l = (1-\\alpha(l))h^{\\text{base}}_l + \\alpha(l)h^{\\text{fine}}_l$, with $\\alpha(l) = \\operatorname{clamp}\\left(\\sum_{i=1}^N c_i B_{i,k}(l) + b_l, 0, 1\\right)$, where $B_{i,k}$ are B-spline basis functions, $c_i$ are trainable control points, and $b_l$ is a layer bias. At selected layers an autoencoder compresses the blend to a bottleneck $z_l = \\operatorname{Encoder}(h_l)$ and reconstructs $\\hat{h}_l = \\operatorname{Decoder}(z_l)$, trained to match either $h^{\\text{base}}_l$ or $h^{\\text{fine}}_l$ while a language-modeling loss keeps the output fluent. The B-spline makes $\\alpha(l)$ vary smoothly across layers, the bottleneck creates polysemantic pressure, and joint training lets $\\alpha(l)$ shift toward the expert state the autoencoder can reconstruct most faithfully for the current input.","core_discovery":"The paper's central claim is that a transformer merged by B-spline-blending hidden states plus autoencoder reconstruction outperforms both linear interpolation and task arithmetic for combining a base and a fine-tuned model. Concretely, it reports overall perplexity 47.01 and next-token accuracy 0.3270 for the autoencoder-merged model, versus 60.29 and 0.3039 for linear interpolation and 61.30 and 0.2957 for task arithmetic. The authors interpret this as evidence that the autoencoders act as a learned gate over the representation space: for English inputs the reconstructed states align with the base model, for French inputs they align with the fine-tuned model, and the bottleneck forces the two expert states to share neurons, increasing the fraction of polysemantic neurons. The base and fine-tuned weights stay frozen throughout, so the only trained parameters are B-spline control points, layer biases, and autoencoders; the paper calls the result an 'in-model' superposition.","pith_inferences":["Extension: bottleneck width should trace a measurable polysemanticity-versus-domain-accuracy frontier; a sweep of bottleneck sizes on the same two models could map that frontier without retraining either expert.","Extension: because the learned blending coefficient depends only on layer index, the $\\alpha(l)$ profile is interpretable as a layer-wise routing policy, and comparing that profile across language pairs could reveal which layers are consistently more transferable.","Extension: the same autoencoder-gated blending could in principle merge more than two models by extending the reconstruction target to several expert states, although the paper explicitly leaves multi-model scaling unresolved.","Extension: if the domain labels used in training are truly optional, the method points toward a fully self-supervised model-merging objective in which the autoencoder's reconstruction error alone selects the expert state; an experiment that trains without labels would settle this."],"forward_implications":["A base model and a fine-tuned model can be collapsed into one set of merged parameters plus small autoencoders, so one checkpoint can serve two specializations without storing both original models separately.","Because the base and fine-tuned weights stay frozen and only the B-spline control points, biases, and autoencoders are trained, the cost of merging is small relative to fine-tuning or to building a mixture of experts with separate expert modules.","The merged model's lower perplexity and higher next-token accuracy than linear interpolation and task arithmetic imply that smooth layer-wise blending with reconstruction is a stronger parameter-efficient merging baseline than simple weight averaging.","The reported rise in polysemantic neurons under the bottleneck suggests the method trades some per-domain specialization for shared, reusable features, making bottleneck size a direct control on the specialization-versus-generality balance."],"supporting_citations":[{"why":"Supplies the GPT-2 architecture and the base model that is merged with its French fine-tune.","marker":"[21]"},{"why":"Defines catastrophic forgetting, the problem the merging method is designed to mitigate.","marker":"[3]"},{"why":"Provides the sparsely-gated mixture-of-experts architecture the paper contrasts with its in-model superposition.","marker":"[8]"},{"why":"Gives LoRA as the parameter-efficient fine-tuning baseline the approach is positioned as orthogonal to.","marker":"[11]"},{"why":"Represents the model-merging literature, the baseline family that includes the linear interpolation and task arithmetic comparisons.","marker":"[20]"},{"why":"Supplies the sparse-autoencoder view of transformer activations that motivates the reconstruction autoencoders and the polysemanticity analysis.","marker":"[18]"}],"fun_headline_variants":["Autoencoder merge beats linear and arithmetic for two languages","B-spline blending with autoencoders stops catastrophic forgetting","Superpose hidden states: autoencoder gate beats single-model merge","One frozen base, one fine-tune: autoencoder superposes both","No separate experts: autoencoder blend keeps both skills intact"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the autoencoder, at inference time, will infer from the blended hidden state alone whether the input belongs to the base or fine-tuned domain and reconstruct the right expert state—training uses domain labels (Section 3.5.4), no experiment isolates the label-free case, and the Limitations section concedes that switching within a single input is not supported.","fun_headline_variants_meta":{"raw":{"variants":["Autoencoder merge beats linear and arithmetic for two languages","B-spline blending with autoencoders stops catastrophic forgetting","Superpose hidden states: autoencoder gate beats single-model merge","One frozen base, one fine-tune: autoencoder superposes both","No separate experts: autoencoder blend keeps both skills intact"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1480,"prompt_tokens":870,"completion_tokens":610,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":526}},"tokens_in":486,"tokens_out":610,"duration_ms":6120,"temperature":1.0,"reasoning_tokens":526,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:48:19.635804+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the merging procedure with domain labels withheld and compare the unlabeled merged model's perplexity against linear interpolation on both languages; or feed the trained model a mixed English–French sequence and compare its reconstructed layer-4 states with the base and fine-tuned target states. If unlabeled training no longer beats the interpolation perplexity, or if the reconstructed states do not align with the correct expert model on mixed inputs, the adaptive label-free switching the method depends on is not actually happening.","supporting_citations":[{"cited_title":"Language Models are Unsupervised Multitask Learners","cited_arxiv_id":null,"evidence_quote":"Supplies the GPT-2 architecture and the base model that is merged with its French fine-tune."},{"cited_title":"McCloskey and N","cited_arxiv_id":null,"evidence_quote":"Defines catastrophic forgetting, the problem the merging method is designed to mitigate."},{"cited_title":"Outrageously large neural net- works: The sparsely-gated mixture-of-experts layer","cited_arxiv_id":null,"evidence_quote":"Provides the sparsely-gated mixture-of-experts architecture the paper contrasts with its in-model superposition."},{"cited_title":"Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen","cited_arxiv_id":null,"evidence_quote":"Gives LoRA as the parameter-efficient fine-tuning baseline the approach is positioned as orthogonal to."},{"cited_title":"Merging mod- els with fisher-weighted averaging","cited_arxiv_id":null,"evidence_quote":"Represents the model-merging literature, the baseline family that includes the linear interpolation and task arithmetic comparisons."},{"cited_title":"Towards Monosemanticity: Decomposing Language Mod- els With Dictionary Learning","cited_arxiv_id":null,"evidence_quote":"Supplies the sparse-autoencoder view of transformer activations that motivates the reconstruction autoencoders and the polysemanticity analysis."}],"review_version":1}