{"id":"e9b11546-d6d9-48aa-9914-d0ebdb1da1e9","arxiv_id":"2505.14436","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Directly transferring parameters between differently-sized language models is unreliable; the paper proposes a pre-alignment method (LaTen) and explains the failure via 'Neural Incompatibility'.","lead":"Researchers tried to transplant knowledge directly from a larger AI model's internal parameters into a smaller one, and found the two models' parameters are too different for the transfer to work reliably. The paper introduces a new 'align before injecting' method and shows that even with alignment, the transfer only helps on some tasks and is unstable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'unbridgeable' claim is underdetermined: only an assumed index-based layer/neuron correspondence is tested, so the observed gap could reflect a wrong mapping rather than intrinsic Neural Incompatibility.","rationale":"The reader and I converge on the same structural weakness. Section 4.2 defines extraction by ranking 13B layers and taking the top Ls in descending score order, then taking top-Ns/ds neurons in each layer; no transformation is learned across models. All unaligned and aligned transfer results in Sections 3 and 5 inherit this assumed correspondence. If the true correspondence between Llama-2-13B and Llama-2-7B is non-monotonic, distributed across layers, or requires a learned rotation or permutation, the extracted delta parameters are misdirected regardless of how good the alignment hypernetwork is. The paper's universal negative claim ('Neural Incompatibility ... unbridgeable') is therefore not established by its experiments. The paper does provide useful empirical evidence: direct unaligned injection of SEEKING-style deltas fails badly (Figure 2), and LaTen gives a modest but real improvement over no-transfer in some benchmarks; the distillation comparisons in Tables 7-8 are a reasonable sanity check. However, all of this is compatible with a weaker conclusion: the particular index-based matching tested here does not transfer well. To settle the question, one experiment should fit a learned per-layer linear or orthogonal map between activations of the two models, map the extracted deltas through it, and compare against the top-index baseline. If the learned map changes outcomes materially, Neural Incompatibility is not intrinsic; if it does not, the claim gains real support. For these reasons I keep the reader's CONDITIONAL verdict rather than strengthening or weakening it.","tokens_in":20189,"tokens_out":5704,"duration_ms":52286,"concrete_test":"Run the same 13B-to-7B transfer on GSM8K and MMLU, but replace the index-based matching of Section 4.2 with a learned correspondence: collect paired activations from both models on 256 shared prompts, fit a per-layer orthogonal Procrustes map (or a small MLP) from 13B neuron rows to 7B neuron rows, map the extracted delta parameters through it, and then apply LaTen's alignment and injection exactly as in the paper. Compare against the top-index baseline in Table 1. If the learned map produces consistent gains (e.g., more than 2 points on GSM8K or 1 point on MMLU), Neural Incompatibility is at least partly an artifact of the assumed correspondence; if it produces no gain, the incompatibility claim is substantially supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that cross-scale parametric transfer fails because of Neural Incompatibility, an intrinsic ethological/parametric structural difference (Abstract; Section 5.3). This is a universal negative, but every experiment is generated by one assumed correspondence: Section 4.2 ranks the 13B layers by attribution score and takes the top Ls in that order, then takes the top Ns/ds neurons within each layer by rank; no mapping between the two models' internal organizations is learned or validated. Under this protocol, low transfer performance conflates 'this index-based correspondence is wrong' with 'no correspondence exists.' The claim is also in tension with the paper's own result: LaTen on GSM8K reaches 20.47 versus the 13B model's 20.55 (Table 1), i.e., near-complete transfer of the task gap on that benchmark, so 'unbridgeable' overstates the data. Confounds remain: alignment uses fewer than 100 examples and 2-8 training steps, and the 'stronger Ml' comparisons use differently fine-tuned checkpoints (WizardCoder/WizardMath), which introduce distribution shift independent of scale. The incompatibility conclusion therefore needs a test that varies the correspondence before it can be called fundamental.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines cross-scale Parametric Knowledge Transfer (PKT) as transferring knowledge from a larger LLM (M_l) to a smaller LLM (M_s) through weights, and argues that alignment in parameter space is a prerequisite for successful transfer. It distinguishes Post-Align PKT (PostPKT, e.g., SEEKING with LoRA initialization followed by fine-tuning) from a proposed Pre-Align PKT (PrePKT) paradigm, and introduces LaTen (Locate-Then-Align), which uses neuron-level attribution and a small hypernetwork to align extracted delta parameters with the target model using fewer than 100 examples. Experiments on MMLU, GSM8K, HumanEval, and MBPP with Llama-2-7B/13B-Chat show that unaligned transfer fails, that LaTen yields small and sometimes inconsistent gains, and that representations between the two scales have low CKA similarity. The paper attributes these failures to 'Neural Incompatibility,' defined as ethological and parametric structural differences between LLMs of different scales, and concludes that cross-scale parametric transfer faces fundamental, 'unbridgeable' obstacles.","tokens_in":20423,"tokens_out":4361,"duration_ms":45139,"significance":"If the central claim were established, the paper would provide a useful negative result for parametric knowledge transfer: cheap weight-based transfer from larger to smaller LLMs would be fundamentally limited, and the community would need to focus on learned alignment or distillation. The paper also makes a concrete proposal (LaTen) and releases code, and its controlled demonstration that unaligned injection degrades performance is a clean and reproducible observation. However, the significance is currently limited because the 'unbridgeable gap' claim rests on experiments that all use one fixed index-based correspondence between source and target layers/neurons, and because the reported gains for LaTen are small, partly within likely noise, and not reported with error bars for the PrePKT results. The stronger claims therefore need additional controls before they can support the paper's title-level conclusion.","major_comments":[{"comment":"Section 5.3's CKA analysis is presented as evidence of 'ethological similarity' between scales, but the CKA values are computed between layer representations at the same layer indices (Figure 3). If the true correspondence between layers is nonlinear or non-monotonic, the low CKA could be an artifact of the index-aligned comparison. The paper should either show that the low-similarity conclusion is robust under learned feature-space alignments (e.g., using a fitted linear map before computing CKA) or state explicitly that the CKA result is conditional on index-based layer correspondence.","section":"Section 4.2 and Section 5.3"}],"minor_comments":[{"comment":"There are typos in the variable names: 'Delta Theta^T_algin' and 'Delta Theta^T_extarct' should be 'align' and 'extract'. Please correct these throughout.","section":"Equations (1)-(2)"},{"comment":"The heading 'Detailed Experimens' should be 'Detailed Experiments'.","section":"Appendix C.1"},{"comment":"The word 'promissing' should be 'promising' in the third bullet point.","section":"Section 1 (Contributions)"},{"comment":"The term 'Neuron Incompatibility' is used in the text ('We call this Neuron Incompatibility') while the abstract and title use 'Neural Incompatibility.' Please make the terminology consistent.","section":"Section 5.3"},{"comment":"The label 'Post-Align on D_train (=1000)' is misleading for MBPP, since Table 4 reports a training size of 300 for MBPP and the text in Section 5.1 says the training set is 1000 'except MBPP.' Please use the actual sizes in the table or in a footnote.","section":"Table 1"},{"comment":"The alignment set sizes (32, 64, 48, 128) and the number of steps (2, 4, 3, 8) vary considerably across benchmarks. Please discuss how these were chosen and whether the results are sensitive to these choices; currently the paper gives no sensitivity analysis.","section":"Section 5.1 and Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful empirical core: the unaligned-injection failure is clearly demonstrated, and the idea of Pre-Align PKT with a low-data hypernetwork is worth reporting. My main concern is that the title and abstract promise a universal negative ('unbridgeable gap') that the experiments do not substantiate, because every experiment uses one fixed index-based correspondence and because the paper's own best result (GSM8K) nearly closes the gap. I would like the editors to require the authors to either add the correspondence-varying control and error bars, or substantially soften the 'Neural Incompatibility' claim to a limitation of current index-based transfer. With such revisions, the paper would be a valuable contribution; in its present form, the central claim overreaches the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful negative result with a new framing, but the central claim is bigger than the experimental setup can support.\n\nThe paper does two things well. It shows, across several simple baselines, that directly injecting extracted parameters from a 13B model into a 7B model without alignment degrades performance — that is robust and convincing. And it organizes the space into Pre-Align and Post-Align PKT, which is a legitimate and clarifying way to think about it. LaTen, the hypernetwork alignment method, is a real new piece; on GSM8K it moves the 7B model from 16.07 to 20.47 against a 13B ceiling of 20.55, using only a few examples. The SEEKING vs PiSSA comparison — self-derived LoRA initialization beating cross-scale initialization — is also interesting evidence.\n\nBut the 'Neural Incompatibility' conclusion goes beyond the data. The experiments only test one assumed correspondence: the top-ranked layers and neurons of the 13B model, in index order, mapped directly to the 7B model. No learned mapping, no permutation test, no validation that this is the right alignment. Low transfer performance under that protocol conflates 'our assumed mapping is wrong' with 'no mapping exists.' And the paper's own best result is a counterexample to the word 'unbridgeable' — a gap of 0.08 on GSM8K is a bridge, not an unbridgeable chasm.\n\nThere are also scope issues. One model family, one size pair, no error bars on the main LaTen results. The 'stronger Ml' experiments use WizardCoder, CodeLlama, and WizardMath, which differ from Llama-2-13B in fine-tuning data as well as scale; the observed drop may be distribution shift, not incompatibility. And the 'first to comprehensively define' claim is a bit much given SEEKING's prior parametric transfer work.\n\nThe CKA and cosine similarity analyses are suggestive but correlational. The analogy to cross-species neuroscience is evocative, not explanatory.\n\nI would send this to a serious referee. The setup is reproducible, the negative result is potentially effort-saving, and the PrePKT/LaTen piece deserves scrutiny. But the authors should be pushed to test the correspondence assumption directly, add uncertainty estimates, and soften the 'unbridgeable' claim to 'challenging under simple index-based alignment.' As is, it reads as a conditional negative result, not a law of nature.","headline":"Useful negative result on cross-scale parameter transfer, but the 'unbridgeable' conclusion is stronger than the evidence: only one assumed layer/neuron correspondence was tested, and the paper's own GSM8K result nearly closes the gap.","tokens_in":20987,"tokens_out":3236,"would_cite":false,"duration_ms":28611,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Copying weights between LLM sizes fails without alignment, and even aligned transfer stays unstable.","keywords":["parametric knowledge transfer","cross-scale LLMs","neural incompatibility","LoRA initialization","knowledge localization","model alignment","hypernetwork","Llama-2"],"falsifier":"Fit a learned permutation between the 13B and 7B layers, or a soft assignment between their neurons, before performing the same transfer; if transfer improves substantially or becomes stable, then Neural Incompatibility is largely an artifact of rank-based matching rather than a fundamental barrier.","tokens_in":19955,"feed_emoji":"🧠","tokens_out":7295,"duration_ms":58171,"temperature":0.7,"pith_summary":"This paper asks whether the knowledge stored in a larger language model's weights can be transplanted directly into a smaller model as a cheap alternative to distillation or fine-tuning. It argues that alignment of the two parametric spaces is a fundamental prerequisite, and that transfer between models of different scales remains unstable even when pre-alignment is attempted, because of what it calls Neural Incompatibility: the behavioral and weight-structure differences between models of different sizes. The paper introduces LaTen, a low-cost pre-alignment method that locates task-relevant neurons in the larger model and projects them into the smaller model's space with a small hypernetwork, but reports that results are not consistently stable across tasks. If the claim holds, weight-based cross-scale transfer is not a reliable shortcut, and any working parametric transfer must first solve the correspondence problem between the two models.","feed_headline":"Copying weights between LLM sizes fails without alignment","feed_subtitle":"Even with its new pre-alignment method, copied weights give unstable gains and cannot replace distillation.","key_machinery":"The central machinery is the Locate-Then-Align (LaTen) pipeline for pre-aligned transfer, together with the diagnostic metrics used to expose incompatibility. LaTen first applies a static neuron-level attribution method to score each FFN and MHSA neuron in the larger model by how much removing it changes the predicted token's log-probability; it selects the top-ranked layers and the top neurons per layer, then sends the extracted delta parameters through a two-layer MLP hypernetwork trained on fewer than 100 examples with standard language-modeling loss, so the projected delta can be added directly to the smaller model's weights. The incompatibility diagnosis uses Centered Kernel Alignment (CKA) to compare layer representations between the two models and cosine similarity between LoRA parameters and the original or residual weights, contrasting PiSSA (which decomposes the target model's own weights) with SEEKING (which imports weights from the larger model).","core_discovery":"The paper's central claim is that alignment in parametric space is a necessary condition for transferring knowledge between differently sized language models through weights, and that this alignment is hard because the two models are 'neurally incompatible'. It redefines prior work such as SEEKING as Post-Align PKT, where extracted parameters initialize LoRA and alignment happens during later fine-tuning, and proposes Pre-Align PKT with LaTen, which uses neuron-level attribution to locate task-relevant FFN and MHSA parameters, an MLP hypernetwork to map them to the smaller model, and injection before any training on the task. The evidence shows that unaligned transfer severely damages the target model, that LoRA initialized from the larger model underperforms LoRA derived from the target model itself (PiSSA), that a task-specialized larger model transfers no better than a generic one, and that representation similarity measured by CKA between the 7B and 13B models is low, especially in attention modules. The paper concludes that Neural Incompatibility, expressed as weak ethological and parametric structural similarity between cross-scale LLMs, poses a fundamental challenge to achieving effective parametric knowledge transfer.","pith_inferences":["If incompatibility is truly structural, then the search for universal neuron-to-neuron correspondences across model sizes may be fundamentally limited, and transfer may need to operate on distributed or functional units rather than individual neurons.","The failure of a stronger specialized source suggests the blocking factor is the source-target gap itself, not the strength of the extracted knowledge; a learned permutation or soft assignment between the two models' layers might reveal how much of the observed incompatibility is an artifact of rank-based matching.","LaTen's instability may reflect the optimization landscape of the hypernetwork rather than an intrinsic ceiling on cross-scale transfer, so a testable extension is to regularize the hypernetwork or search over different source layer orders.","The same diagnostic (CKA plus parametric cosine similarity) could be applied to same-scale models from different checkpoints to test whether incompatibility is specific to scale or a general property of transferring weights between any two independently trained models."],"forward_implications":["Weight-based transfer from a larger to a smaller LLM cannot serve as a drop-in replacement for distillation or fine-tuning, because unaligned injection collapses task performance.","LoRA initialization derived from the target model's own weights (PiSSA) is a stronger baseline than initialization from a larger model's extracted parameters, so cross-model initialization must justify its added cost.","Pre-alignment with very few examples can yield task gains (for example, +4.40 on GSM8K) but is unstable, meaning practical use would require checkpoint selection and repeated runs.","A larger model that is specialized for a task transfers no better, and sometimes worse, than a generic larger model, so source capability alone does not predict transfer success.","Future parametric transfer methods must model the correspondence between source and target layers and neurons rather than assuming an index-based match."],"supporting_citations":[{"why":"Supplies the static neuron-level attribution method used to locate task-relevant neurons in the larger model.","marker":"(Yu and Ananiadou, 2024)"},{"why":"Provides the SEEKING method, the main Post-Align baseline that initializes LoRA from extracted cross-scale parameters.","marker":"(Zhong et al., 2023)"},{"why":"Provides PiSSA, the self-derived LoRA initialization that serves as the strong comparison showing same-model parameters transfer better.","marker":"(Meng et al., 2024)"},{"why":"Supplies Centered Kernel Alignment, the metric used to measure ethological representation similarity between the two models.","marker":"(Kornblith et al., 2019)"},{"why":"Defines the Llama-2 model family used for all cross-scale transfer experiments.","marker":"(Touvron et al., 2023)"},{"why":"Establishes the key-value memory view of FFN neurons that motivates selecting individual neurons as transfer units.","marker":"(Geva et al., 2020)"},{"why":"Provides the LoRA injection mechanism through which Post-Align PKT transfers extracted parameters.","marker":"(Hu et al., 2021)"},{"why":"Supplies the WizardCoder-13B-Python model used as the task-specialized larger source in code-generation transfer tests.","marker":"(Luo et al., 2023b)"}],"fun_headline_variants":["Parametric alignment alone won't fix cross-scale LLM transfer","Neural incompatibility blocks weight copying between LLM sizes","Pre-aligning LLM weights still yields unstable knowledge transfer","Copying weights across LLM scales fails without neural alignment","LaTen pre-aligns LLMs, but transfer remains fundamentally unstable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study assumes that the top-ranked layers and neurons of the larger model correspond to the smaller model's layers and neurons in the same order, so matching them by rank is meaningful; if the real correspondence is scrambled or distributed, the observed incompatibility may be partly an artifact of that assumption.","fun_headline_variants_meta":{"raw":{"variants":["Parametric alignment alone won't fix cross-scale LLM transfer","Neural incompatibility blocks weight copying between LLM sizes","Pre-aligning LLM weights still yields unstable knowledge transfer","Copying weights across LLM scales fails without neural alignment","LaTen pre-aligns LLMs, but transfer remains fundamentally unstable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1538,"prompt_tokens":1090,"completion_tokens":448,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":706,"completion_tokens_details":{"reasoning_tokens":364}},"tokens_in":706,"tokens_out":448,"duration_ms":4965,"temperature":1.0,"reasoning_tokens":364,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:34:03.008036+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit a learned permutation between the 13B and 7B layers, or a soft assignment between their neurons, before performing the same transfer; if transfer improves substantially or becomes stable, then Neural Incompatibility is largely an artifact of rank-based matching rather than a fundamental barrier.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the static neuron-level attribution method used to locate task-relevant neurons in the larger model."}],"review_version":1}