{"id":"6be643ed-7612-4ba9-8eac-bc6db3dd4b9d","arxiv_id":"2505.13515","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A modular LoRA-transfer pipeline reuses old LoRA weights across LLM upgrades, claiming gains over small-scale retraining and sometimes over full retraining.","lead":"LoRASuite is a method that converts LoRA adapters trained on an older LLM into usable starting points for a newer LLM, using embedding-derived transforms, CKA layer matching, and Hungarian head matching. The paper reports results that match or beat full LoRA retraining on several benchmarks while cutting time by roughly 78% and memory by 5.5 GB.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The transfer matrix W_h = E_o^{-1}E_n (Sec. 3.1) is undefined for every reported upgrade because embedding matrices are rectangular (e.g., 73k×1536 vs 123k×2304); the construction is not instantiable as written, and near-vanilla 'w/o LFT' rows tie the reported gains to the unmatched fine-tuning…","rationale":"I agree with the reader that the undefined inverse in W_h = E_o^{-1}E_n is the weakest load-bearing step. It is not a matter of taste: Section 3.1's formula is dimensionally impossible for every model pair in Table 2, and Eq. (3), the head-level transformation at the heart of Algorithm 2, inherits that ill-defined matrix. A methods paper whose central construction cannot be instantiated from the text, with no code release, cannot support its headline empirical claims. The paper's own evidence sharpens the concern rather than resolving it: 'LoRASuite w/o LFT' is essentially vanilla in the flagship MiniCPM setting, and the sensitivity analysis shows the advantage over LoRA (Small) almost vanishes at LR 1e-4, so the unmatched 1e-3/no-warmup schedule of the LFT stage is a confound for the central comparative claim. I take the reader's CONDITIONAL verdict as unchanged. The concern is addressable: a least-squares or procrustes-style W_h could be specified and the results re-verified, so REJECT would be too harsh given the breadth of the evaluation. The paper also deserves credit for the CKA layer-mapping ablation (vs CCA, PWCCA, Procrustes), the head-mapping ablation, and the honest Limitations paragraph admitting the fine-tuning step is required. These strengthen my view that the gap is one of specification and baseline matching rather than of fabricated results; the conditional acceptance should hinge on the authors providing the exact W_h construction and a matched-schedule baseline.","tokens_in":19167,"tokens_out":12192,"duration_ms":109158,"concrete_test":"Require the authors to state exactly how W_h was computed for the MiniCPM-S-1B→MiniCPM-2B results in Tables 4–5 (E_o ∈ R^{73,008×1,536}, E_n ∈ R^{123,040×2,304}), i.e., the precise pseudo-inverse or least-squares objective, and then re-run the math benchmark of Table 4 with (i) that specified W_h and (ii) LoRA (Small) under LoRASuite's own schedule (LR 1e-3, warmup 0). If the gap over LoRA (Small) shrinks to the 0.27-point level seen at LR 1e-4, the 'consistent outperformance' is attributable to the unmatched schedule rather than to the transfer construction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines the hidden-space transfer matrix as W_h = E_o^{-1}E_n; Eq. (3) and Algorithm 2 then compose this matrix into every head-level LoRA update. For every pair in Table 2, E_o and E_n are rectangular, with vocab dimension far exceeding hidden dimension (e.g., MiniCPM-S-1B: 73,008×1,536 → MiniCPM-2B: 123,040×2,304; Llama-2-7B: 32,000×4,096 → Llama-3-8B: 128,256×4,096). E_o^{-1} does not exist in the stated sense; a one-sided or pseudo-inverse could be intended, but no such construction, rank condition, least-squares objective, or regularization is given, nor for W_i = W_o^{-1}W_hW_n. The shared-token intersection step does not resolve this, since V_shared ≫ d. With no code release, Table 4's numbers cannot be traced to any definite construction. Two further observations make this gap load-bearing rather than cosmetic. First, 'LoRASuite w/o LFT' matches vanilla MiniCPM-2B almost exactly (math 23.96 vs 23.85; commonsense 32.69 vs 32.72), and the Limitations section concedes the fine-tuning step is required for optimal performance — the transferred initialization itself contributes essentially nothing in the flagship setting. Second, the gains come from a 100-example fine-tune at LR 1e-3 with zero warmup, a schedule never given to the LoRA (Small) baseline (LR 3e-4, warmup). Figure 4(b) shows LoRASuite's edge over LoRA (Small) collapses to 0.27 points at LR 1e-4. Until W_h is specified and baselines matched, the claim that the transfer construction 'consistently outperforms' vanilla LoRA is not supported by the described mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LoRASuite, a method for adapting LoRA adapters trained on an older LLM to a newer version of the same architecture family without retraining from scratch. The authors categorize model upgrades into six types (vocabulary size, hidden size, intermediate size, layer depth, head count, attention type) and propose modular remedies: a transfer matrix W_h = E_o^{-1}E_n computed from the embedding weights of the two models (for vocab/hidden changes), a similar W_i for intermediate sizes, a dynamic-programming layer mapping driven by minibatch CKA similarities, and a Hungarian-based attention-head mapping using cosine similarity of input-independent interaction matrices; the transferred parameters are then decomposed by SVD and optionally refined by a small-scale fine-tuning step (LFT). Experiments across six backbone upgrade pairs and fifteen math/commonsense datasets report that LoRASuite with 100-1000 examples outperforms same-scale vanilla LoRA and, for MiniCPM and Qwen, exceeds full-scale LoRA retraining, while cutting memory by 5.5 GB and time by 78.23%. The paper also reports ablations of the layer- and head-mapping components.","tokens_in":19598,"tokens_out":16586,"duration_ms":153983,"significance":"If the central claims held, the contribution would be practically significant: recycling existing LoRA adapters across model upgrades would avoid expensive retraining, and the modular treatment of six upgrade factors is a sensible decomposition of a problem that has received little direct attention. The evaluation is broad (six backbone pairs, fifteen datasets), and the ablations (Figures 11-13) at least demonstrate that the layer- and head-mapping components affect post-fine-tuning accuracy. Credit is also due for the honest limitations paragraph, which concedes that the fine-tuning step is required for optimal performance. However, the paper's core construction is currently not well-defined: W_h uses an inverse of rectangular embedding matrices that does not exist in the stated sense, Eq. (3) is asserted without derivation or code, the CKA inputs are unspecified, and the headline comparisons are run with different learning-rate schedules for LoRASuite and its baselines. The Yi-6B to Yi-1.5-9B results (w/o LFT far above vanilla) also contradict the paper's own explanation that the transfer is numerically unstable.","major_comments":[{"comment":"The central transfer construction is not instantiable as written. The paper defines W_h = E_o^{-1} E_n, but in every reported upgrade with a hidden-size change the embedding matrices are rectangular with the vocabulary dimension far exceeding the hidden dimension (e.g., MiniCPM-S-1B: 73,008x1,536 to MiniCPM-2B: 123,040x2,304; Bloom-560m to Bloomz-1B1: 250k x 1,024 to 250k x 1,536), so the inverse E_o^{-1} does not exist. The shared-token intersection step does not fix this, since the number of shared tokens is still far larger than the hidden dimension. The same problem affects W_i = W_o^{-1} W_h W_n for up/down projections (e.g., Llama-2-7B intermediate 11,008 to Llama-3-8B intermediate 14,336). No pseudo-inverse, least-squares objective, rank condition, or regularization is stated, and since no code is released, Tables 4-18 cannot be traced to any definite construction. The authors should specify the exact construction (e.g., E_o^+ E_n with a stated SVD cutoff or ridge objective), verify it on the reported settings, and state whether any of the reported numbers change.","section":"Section 3.1, Eq. (3), Algorithm 2"},{"comment":"The main comparison that supports the abstract's claim ('consistently outperforms small-scale vanilla LoRA') is confounded by unequal training recipes. In Tables 7 and 8, LoRASuite's fine-tuning step uses LR 1e-3 with zero warmup, while LoRA baselines use LR 3e-4 with warmup ratio 0.1 (or 100 warmup steps); the main tables therefore do not show a like-for-like comparison at the same data scale. The paper's own sensitivity analysis (Figure 4(b)) shows that when the learning rate is lowered to 1e-4 the LoRASuite advantage over LoRA (Small) shrinks to 0.27 points, and no matched-recipe baseline is reported for the other five backbone pairs in Figures 2-3. As a result, the reported gains and the 'exceeds full-scale LoRA retraining' claims rest on a narrow, unstated hyperparameter window. Please report matched-schedule baselines (same LR, same warmup, same data scale) in all main tables, plot both methods across the LR grid with per-task numbers, and report variance over seeds.","section":"Section 4.1, Tables 4-5, Tables 7-8, Figure 4(b)"},{"comment":"The CKA computation and the head-level update are under-specified. It is not stated which corpus or prompt set is used to collect the activations for the CKA similarity matrix S, how many minibatches are used, what batch size and sequence length, or whether activations come from the base models or from LoRA-adapted models; the heatmaps in Figures 7-10 therefore cannot be reproduced. Equation (3) is asserted without derivation: the composition W_h^T (Delta W_Q)_o (W_Q)_o^T W_h (W_Q)_n mixes the LoRA update, the original projection weights, and the target projection weights in a way that is not explained, and the dimension bookkeeping when both head count and hidden size change (H_o != H_n, d_o != d_n) is not spelled out. A derivation of Eq. (3) (or a reference to the linear-algebra identity it relies on), together with the concrete CKA configuration, is needed before the method can be evaluated independently.","section":"Section 3.1, Algorithm 1, Eq. (3)"},{"comment":"The interpretation of the 'w/o LFT' rows is internally inconsistent. On MiniCPM the paper notes (correctly) that 'LoRASuite w/o LFT' matches the vanilla model (Tables 4-5: 23.96 vs 23.85 math; 32.69 vs 32.72 commonsense) and attributes this to numerical instability of matrix-multiplication-only transforms. But for Yi-6B to Yi-1.5-9B the same 'w/o LFT' configuration is far above vanilla (math 73.07 vs 65.86; commonsense 74.29 vs 56.78), and in Table 9 the fine-tuned LoRASuite is actually worse than 'w/o LFT' on AQuA (30.31 vs 38.58). One of these two patterns must be wrong, or the mechanism must be something other than 'numerical instability' (e.g., the eval pipeline has high variance, or the Yi numbers reflect an uncontrolled component). The paper should explain this discrepancy and report per-seed variance, since the flagship claim that the transfer itself is valuable rests entirely on the post-LFT numbers.","section":"Section 4.1, Tables 9-10"}],"minor_comments":[{"comment":"Typos and formatting issues: 'increase the learning rate to to compensate' (Section 3.2), 'fintuing' (Section 4.1), 'Sensitive Analysis' (Section 4.2), inconsistent 'LoRa'/'LoRA' capitalization, and the 'SV AMP' column header should be typeset correctly.","section":"Throughout"},{"comment":"No randomness control is reported: the tables show single-run numbers without seeds or confidence intervals, which matters in particular for the small-scale (100-example) settings where accuracy differences of 1-2 points are used to support the component ablations.","section":"Tables 4-18 and Figures 4, 11-13"},{"comment":"The pseudocode's input/output notation ('/# Store...') is inconsistent with the surrounding text, and the complexity claim in Section 3.3 writes O(nlayer(Delta_layer^2 + n_head^3)) without defining the scope of the constants; please clean up the pseudocode formatting.","section":"Algorithm 1, Section 3.3"},{"comment":"The abstract's memory saving of 5.5 GB and time reduction of 78.23% are reported as headline numbers, but Figure 1 measures a single setting (MiniCPM); please state explicitly that these figures are setting-specific.","section":"Abstract, Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern from the internal review is valid and is reflected in major comment 1: the W_h construction is undefined for the reported settings, so the experimental pipeline cannot be audited without a code release or an explicit pseudo-inverse specification. My assessment differs from the strongest version of the skeptic's note in one respect: Figure 4(b) suggests that, at a matched learning rate, LoRASuite does retain an advantage at the paper's chosen LR, so the comparison is not purely vacuous, but the main tables should be re-run with matched recipes to make that point. The Yi-6B to Yi-1.5-9B anomaly (major comment 4) is the kind of result that, if it survives re-evaluation, would actually strengthen the paper if explained; as written it contradicts the stated narrative. I see no circularity issue: the transfer matrices are computed from model weights and CKA similarities, not from the benchmark labels. Fit with the journal is acceptable, provided the reproducibility gaps are closed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWorth a look if you care about PEFT lifecycle issues. This is the first paper I know to ask how to recycle a trained LoRA when the backbone is upgraded, and it breaks the problem into six concrete mismatches. That framing is genuinely useful, and the experiments span five backbone pairs and many tasks, with big wins on MiniCPM and Qwen math.\n\nWhat the paper does well: the modular design is sensible — CKA for layer mapping, Hungarian for attention heads, and an embedding-derived matrix for hidden-size changes — and the ablation section compares against simpler alternatives like first/middle/last mapping, CCA/PWCCA/Procrustes, and no head mapping. Those ablations are honest and informative.\n\nThe soft spots are real, though. The central transfer matrix is written as W_h = E_o^{-1}E_n in Section 3.1, and for every reported upgrade the embedding matrices are rectangular, so that inverse does not exist. A pseudo-inverse or least-squares construction may have been intended, but it is not stated, and Equation (3), which composes W_h into head-level updates, is simply asserted. Until that is specified, the mechanism cannot be reproduced.\n\nThe second issue is that the reported advantage of LoRASuite is not cleanly attributable to the transfer. In the MiniCPM tables, LoRASuite without the fine-tuning step lands almost exactly at vanilla accuracy on both math and commonsense, and the same pattern appears in the other backbone plots. The Limitations section essentially concedes that the fine-tuning step is required. Meanwhile, the fine-tuning step uses a learning rate of 1e-3 with no warm-up, while LoRA (Small) uses 3e-4 with warm-up. Figure 4(b) shows the edge nearly vanishes at LR 1e-4. So the headline claim that the transfer “consistently outperforms” vanilla LoRA is, as written, supported mainly by a scheduling difference rather than by the described mechanism.\n\nI would not cite this in its current form, and the lack of a code release makes it hard to verify. But the problem is real, the experimental scope is substantial, and the flaws are addressable rather than fatal. A serious referee could push the authors to define W_h properly, release code, match the optimization schedule across baselines, and isolate what the transferred initialization contributes on its own. I would send it out for review, with the expectation of heavy revision.\n\nCandidly yours.","headline":"Real problem, first framing, but the transfer matrix is undefined for the actual rectangular embeddings and the headline gains seem to come from the high-LR fine-tuning step rather than the transfer.","tokens_in":20149,"tokens_out":3487,"would_cite":false,"duration_ms":34318,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LoRASuite claims old LoRA adapters can be recycled across LLM upgrades, beating full retraining on some backbones.","keywords":["LoRA","parameter-efficient fine-tuning","LLM upgrade","transfer matrix","centered kernel alignment","Hungarian algorithm","attention head mapping","low-rank adaptation"],"falsifier":"For an upgrade such as MiniCPM-S-1B to MiniCPM-2B (hidden size 1536 to 2304), identify the exact matrix used as $W_h$ and check whether it satisfies $W_h E_o = E_n$ on the shared-token rows; if no such matrix is specified, substitute a random matrix of the same shape in 'LoRASuite w/o LFT' and compare — if accuracy barely changes, the transfer matrix is not carrying the claimed effect.","tokens_in":18929,"feed_emoji":"♻️","tokens_out":8896,"duration_ms":83606,"temperature":0.7,"pith_summary":"Every time a model family releases a new version, the LoRA adapters trained for the old version are usually discarded and retrained from scratch. LoRASuite argues they can instead be transplanted: given both checkpoints, it builds a transfer matrix from the embedding weights, aligns layers with representation similarity and attention heads with cosine similarity, then pays for a tiny fine-tuning pass. The paper reports that recycled adapters consistently beat same-scale vanilla LoRA and, on MiniCPM and Qwen math benchmarks, also beat full-scale LoRA retraining by +1.4 and +6.6 average points while cutting memory by 5.5 GB and time by about 78%. If correct, adapter repositories for app-specific tasks would survive backbone upgrades almost for free.","feed_headline":"Recycled LoRA adapters beat full retraining on some LLM upgrades","feed_subtitle":"LoRASuite transplants old LoRA weights into new model versions with 100 samples, cutting time by 78%.","key_machinery":"The central object is the embedding-derived transfer matrix $W_h = E_o^{-1}E_n$, which is composed into every head-level update, e.g. $(\\Delta W_Q^j)_n = W_h^T (\\Delta W_Q^i)_o ((W_Q^i)_o)^T W_h (W_Q^j)_n$, so that a single linear map is responsible for moving LoRA updates across hidden-space changes. Around it sit the layer-mapping machinery (minibatch CKA with a dynamic-programming alignment under a maximum-offset constraint), the head-mapping machinery (cosine similarity over input-independent interaction matrices $W_{QK}$ and $W_{VO}$, assigned by the Hungarian algorithm), and an SVD step that re-factors each transformed update into low-rank $B_n, A_n$. A deliberate small fine-tuning stage compensates for the numerical instability of purely matrix-multiplication transforms.","core_discovery":"LoRASuite's central claim is that a LoRA update $B_o A_o$ trained on an older LLM can be transplanted into a newer LLM of the same architecture by transforming it through known parameter differences rather than retraining. Hidden-size and vocabulary changes are handled with a transfer matrix $W_h = E_o^{-1}E_n$ built from the two models' embedding weights (with a shared-token intersection step when vocabularies differ); intermediate-size changes use $W_i = W_o^{-1} W_h W_n$; layer-depth changes are aligned by a dynamic program over minibatch CKA similarities; and attention-head differences are assigned with the Hungarian algorithm on cosine similarities between per-head interaction matrices $W_{QK}$ and $W_{VO}$. The relocated head-level updates are factored back into low-rank form by SVD, and a short fine-tuning pass is added because the transformed parameters come from matrix products rather than backpropagation. On five backbone upgrade pairs the method consistently beats same-scale vanilla LoRA, and on MiniCPM-S-1B to MiniCPM-2B and Qwen-1.5-1.8B to Qwen-2.5-3B it beats full-scale LoRA retraining on math tasks by +1.4 and +6.6 average points.","pith_inferences":["Because the reported hidden-size upgrades all involve rectangular embedding matrices, the written formula $W_h = E_o^{-1}E_n$ cannot be evaluated as stated; replacing it with a least-squares pseudo-inverse is the natural minimal variant to test and is not reported in the paper.","If the embedding-derived map is genuinely carrying the transfer, the same construction should extend to cross-family upgrades that share a tokenizer; the paper restricts itself to same-architecture upgrades, so this is an open testable extrapolation.","The finding that LoRASuite's advantage jumps from +0.27 points at learning rate $10^{-4}$ to +21.30 at $9 \\times 10^{-4}$ suggests the transferred initialization matters mainly through the fine-tuning trajectory, so a sweep over optimizers and schedules is a direct way to separate initialization value from fine-tuning value.","The method's modularity means any improved component (better similarity metric than CKA, better assignment than Hungarian) can be swapped in independently; comparing components one at a time would isolate which stage contributes most."],"forward_implications":["LoRA weights trained for an older model version can be transplanted into the new version rather than discarded, removing the main retraining cost of model upgrades.","With only 100 fine-tuning samples, LoRASuite beats full 10,000-sample LoRA retraining on MiniCPM math tasks (43.80 vs 42.39 average) and on Qwen math tasks (56.88 vs 50.32).","The same recipe transfers DoRA adapters to within less than 1 point of full-scale retraining, so the approach generalizes beyond vanilla LoRA.","Memory drops by 5.5 GB and training time by 78.23% compared with full LoRA retraining, which is what makes per-app adapters on devices economically plausible."],"supporting_citations":[{"why":"Defines LoRA and the low-rank decomposition $B_o A_o$ that the transfer pipeline starts from.","marker":"[1]"},{"why":"Supplies the minibatch estimator of CKA that makes layer similarity cheap enough for large models.","marker":"[28]"},{"why":"Defines centered kernel alignment, the similarity index used to build the layer-similarity matrix.","marker":"[29]"},{"why":"Provides the unbiased HSIC estimator used inside the minibatch CKA computation.","marker":"[30]"},{"why":"The Hungarian algorithm solves the attention-head assignment at cubic cost.","marker":"[33]"},{"why":"Gives the prompt templates and normalization used in the commonsense and math evaluations.","marker":"[41]"}],"fun_headline_variants":["LoRA transfer beats full retraining on some LLM upgrades","Recycle LoRA weights for new LLMs, save 78% time","LoRASuite: adapt LoRA across model versions without retraining","Transplant LoRA adapters, cut compute 78% on math tasks","Efficient LoRA reuse: beat full retraining on math tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one linear map computed from the two models' embedding weights can carry LoRA updates from the old hidden space to the new one; in the reported upgrades the embedding matrices are rectangular, so the stated inverse $W_h = E_o^{-1}E_n$ does not exist and the paper does not specify its replacement.","fun_headline_variants_meta":{"raw":{"variants":["LoRA transfer beats full retraining on some LLM upgrades","Recycle LoRA weights for new LLMs, save 78% time","LoRASuite: adapt LoRA across model versions without retraining","Transplant LoRA adapters, cut compute 78% on math tasks","Efficient LoRA reuse: beat full retraining on math tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000883,"raw_usage":{"total_tokens":3865,"prompt_tokens":1047,"completion_tokens":2818,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":2723}},"tokens_in":663,"tokens_out":2818,"duration_ms":19612,"temperature":1.0,"reasoning_tokens":2723,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:46:40.802152+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For an upgrade such as MiniCPM-S-1B to MiniCPM-2B (hidden size 1536 to 2304), identify the exact matrix used as $W_h$ and check whether it satisfies $W_h E_o = E_n$ on the shared-token rows; if no such matrix is specified, substitute a random matrix of the same shape in 'LoRASuite w/o LFT' and compare — if accuracy barely changes, the transfer matrix is not carrying the claimed effect.","supporting_citations":[{"cited_title":"Similarity of neural network representations revisited,","cited_arxiv_id":null,"evidence_quote":"Defines centered kernel alignment, the similarity index used to build the layer-similarity matrix."},{"cited_title":"Lora: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"Defines LoRA and the low-rank decomposition $B_o A_o$ that the transfer pipeline starts from."},{"cited_title":"Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth,","cited_arxiv_id":null,"evidence_quote":"Supplies the minibatch estimator of CKA that makes layer similarity cheap enough for large models."},{"cited_title":"Feature selection via dependence maximization,","cited_arxiv_id":null,"evidence_quote":"Provides the unbiased HSIC estimator used inside the minibatch CKA computation."},{"cited_title":"Algorithms for the assignment and transportation problems,","cited_arxiv_id":null,"evidence_quote":"The Hungarian algorithm solves the attention-head assignment at cubic cost."}],"review_version":1}