{"id":"f0ff6414-a2a8-4397-9cda-6de6c48b2c73","arxiv_id":"2501.14713","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A block-pruning and model-extension method for LLMs that replaces pruned blocks with weight-shared blocks plus low-rank adapters, reporting state-of-the-art recovery on several benchmarks.","lead":"FlexiGPT prunes large language models by removing transformer blocks and replacing them with shared weights from remaining blocks plus small low-rank adapters, and also extends small models by repeating blocks with similar adapters. The paper reports that the method recovers more performance after pruning than several existing block-pruning baselines on LLaMA-2 7B and other models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FlexiGPT's per-position LoRA adapters cannot be merged into shared weights, so the advertised '30%' compression omits persistent adapter/norm parameters; the Appendix's own 3.67% parameter increase suggests actual savings are ~27-28%, making Table 1 an unequal-parameter comparison.","rationale":"The reader's weakest-assumption analysis focuses on the SVD-based distance metric in Eq. (2), asking whether it truly identifies blocks recoverable by low-rank adapters. That is a reasonable concern, but the paper's ablations partially support the metric: replacing it with Frobenius distance or omitting high-rank pruning degrades PPL, and the selection is used only as an initialization/assignment heuristic before a 1B-token recovery phase. I find a more directly falsifiable weakness in the paper's parameter accounting. The method's deployed model includes block-specific LoRA adapters and output-normalization parameters that cannot be folded into the shared base weights, so the nominal '30%' and '40%' compression ratios overstate the actual parameter savings. The Appendix's admission of a 3.67% relative parameter increase over ShortGPT, combined with the unmergeable adapter structure, makes this a concrete arithmetic issue rather than a subjective modeling judgment. This does not refute the method outright; corrected parameter accounting and matched-budget comparisons could still support the central claim. It does, however, strengthen the conditions under which the paper should be accepted: the authors need to report true deployed parameter counts, release checkpoints for verification, and rerun the key comparisons at equal parameter budgets. Since the reader already assigned a CONDITIONAL verdict, this concern reinforces rather than changes that verdict.","tokens_in":119,"tokens_out":14740,"duration_ms":199042,"concrete_test":"Count deployed parameters from the actual checkpoints for FlexiGPT and ShortGPT+LoRA at nominal 30% and 40% settings: sum every stored tensor, counting shared base weights once and counting LoRA A/B and normalization matrices for every pruned position. Verify whether FlexiGPT's true parameter reduction equals 30%/40%. Then rerun the Table 1 comparison at matched deployed-parameter budgets, for example by lowering FlexiGPT's LoRA rank or giving ShortGPT+LoRA equivalent extra capacity. If the PPL and accuracy margins shrink or reverse, the headline compression claim is not supported at the advertised rates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central parameter-efficiency claim depends on the reported compression ratio, but that ratio appears to count only block removal, not the deployed parameter cost of the replacement scheme. In FlexiGPT, each pruned block is replaced by a shared base block plus a block-specific LoRA adapter and output normalization. Because several pruned positions share the same base weights, the LoRA matrices for different positions cannot be merged into the shared base without corrupting the other positions sharing it; they must remain as separate stored parameters at inference. This is different from ShortGPT+LoRA, where each retained block is unique and its adapter can be merged into that block after fine-tuning, leaving the model at the nominal compression level. Appendix A states that 'Compared to ShortGPT, our method incurs a 3.67% relative increase in total parameters for the main experiment setting of Table 1.' For LLaMA-2 7B at nominal 30% block removal, ten rank-256 LoRA sets add roughly 200M parameters on top of the ~4.6B retained-block model, putting the actual reduction near 27-28% rather than 30%. The observed margins in Table 1 (PPL 6.55 vs 6.71; average accuracy 62.68% vs 61.40%) may therefore reflect extra deployed capacity rather than the weight-sharing mechanism itself. This undercuts the 'state-of-the-art at 30%/40% compression' claim unless parameter counts are reported and controlled.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FlexiGPT, a block-pruning and model-extension method for LLMs. Pruned transformer blocks are replaced by weight-shared unpruned blocks chosen via a low-rank SVD distance metric, plus block-specific LoRA adapters initialized from the SVD of the difference between the pruned and base blocks, and an output feature normalization. The same machinery is used to extend smaller models by repeating blocks with unique adapters. Experiments on LLaMA-2 7B, LLaMA-3 8B, and OPT models report lower perplexity and higher zero-shot accuracy than ShortGPT and ShortGPT+LoRA at 30% and 40% block removal, and extension experiments on TinyLLaMA report improved average accuracy after continued pretraining.","tokens_in":14707,"tokens_out":6868,"duration_ms":61583,"significance":"If the central efficiency claim held, the paper would be a useful contribution to the block-pruning literature: replacing removed blocks with shared lightweight bases plus adapters is a practical recipe, and the ablation isolating output normalization, SVD initialization, and high-rank pruning helps the community understand which ingredients matter. The paper is also commendable for addressing an extension setting that is rarely studied in pruning papers and for discussing compute/throughput trade-offs. However, the advertised parameter savings are not matched to the deployed parameter count, and several experimental controls are missing, so the significance of the current empirical claims is substantially reduced.","major_comments":[{"comment":"The nominal compression ratio is not the deployed parameter reduction. Because multiple pruned positions share the same base block, the per-position LoRA adapters cannot be merged into that shared base at inference; the adapters must remain as separate stored parameters for every pruned position. Appendix A states that FlexiGPT incurs a 3.67% relative increase in total parameters compared with ShortGPT for the main Table 1 setting. For LLaMA-2 7B at nominal 30% block removal, ten rank-256 LoRA sets plus normalization parameters add roughly 250M parameters on top of the retained-block model, which reduces the actual parameter savings from about 30% to about 26-27%. ShortGPT+LoRA, by contrast, applies adapters to unique retained blocks and can merge them after recovery training, so Table 1 compares models with different deployed parameter budgets. The reported margins (PPL 6.55 vs 6.71; average accuracy 62.68% vs 61.40%) may therefore reflect the extra capacity rather than the weight-sharing mechanism. The paper must report the true deployed parameter count and memory footprint for each configuration and, when claiming state-of-the-art at 30% or 40% compression, must match parameter budgets across methods.","section":"Appendix A; Table 1; Table 6"},{"comment":"The extension experiment lacks the critical control of continued training on the base model. The 22-layer base is evaluated after its original training only, while the 36-layer FlexiGPT model receives 10B additional tokens of continued pretraining. Without a matched baseline of the 22-layer model trained on the same 10B tokens, the observed improvements (e.g., HellaSwag 59.77% vs 59.20%, PPL 6.73 vs 6.84) cannot be attributed to the extension mechanism rather than to the additional data. Please add a 'Base + continued training' control with the same token budget and training setup.","section":"Section 5.2; Table 5"},{"comment":"All results are single runs with no error bars or significance tests, and several comparisons are drawn from baselines imported from other papers. Many of the reported advantages are small (e.g., ARC-e 62.84% vs 62.50%, WinoGrande 66.78% vs 66.61%), so the differences could easily be within the noise of a single seed and evaluation protocol. Please report multiple seeds and standard deviations, or at least multiple evaluation runs, and standardize the evaluation harness and prompts across all methods. In addition, Tables 2 and 3 omit the ShortGPT+LoRA or ShortGPT+FT baselines that are used in Table 1; without them, the cross-model generalization claims in Section 4.2 are supported only by comparisons to unrecovered ShortGPT and not to the strongest available baseline.","section":"Appendix A; Tables 1-3"},{"comment":"The selection metric in Eq. (2) contains several hand-chosen ingredients (rank r=256, the residual term Delta_{i-j}, and the high-rank pruning step) that are validated only through end-task performance after 1B tokens of recovery training. Table 4 shows that removing the high-rank-pruning component hurts PPL, which is useful, but it does not directly test whether low values of d(W_i,W_j) identify bases from which the pruned block's function can actually be recovered. Please add a more direct diagnostic, such as per-block output similarity before and after adaptation, or control experiments with nearest-index base selection and random base assignment under matched recovery training. Without such a test, the centrality of the proposed metric to the method's success remains an interpretation rather than a demonstrated mechanism.","section":"Section 3.2; Eq. (2); Table 4"}],"minor_comments":[{"comment":"There is an internal inconsistency in the training-token budget: Section 5.2 says the extension models underwent 'continued pre-training on 1B tokens', while Section 5.1, Table 5, and the caption say 10B tokens. Also, the text says TinyLLaMA was trained on '30T tokens', but the TinyLLaMA reference reports about 3T tokens; the 0.3% figure is consistent with 10B/3T, so please correct the typo.","section":"Section 5.2; Table 5; Abstract"},{"comment":"Equation (6) should specify whether the output normalization is per-feature or per-hidden-unit, the initialization value of gamma, and how this normalization interacts with the existing LayerNorm that is already part of each transformer block. These details are needed for reproducibility.","section":"Section 3.3; Eq. (6)"},{"comment":"The column header 'A verage' appears to be a formatting error and should read 'Average'. Additionally, reporting standard errors around the zero-shot accuracy averages would make the single-run results more interpretable.","section":"Tables 1, 2, 5"},{"comment":"The throughput tables report normalized time and throughput but not peak memory or the actual number of stored parameters after recovery. Since the paper's motivation is on-device deployment under memory constraints, please report measured memory footprint alongside parameter counts.","section":"Table 6; Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely salvageable, but the headline compression claim must be re-derived with true deployed parameter counts. If the authors cannot match parameter budgets without losing the reported gains, the 'state-of-the-art' phrasing should be withdrawn or substantially qualified. The extension experiment also needs a matched data-control baseline before any claim of 'boosting performance' can be supported. I would not reject the paper on the idea's merits; the weight-sharing-plus-adapters recipe is reasonable and the ablations are informative. The current version, however, overstates the empirical support for its central claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FlexiGPT is worth reading for the core trick: instead of just removing blocks like ShortGPT, you find a similar surviving block, share its weights, and attach a LoRA adapter trained from the SVD of the difference. That combination is new, and the ablations actually earn the design—removing output normalization or the SVD initialization hurts. The extension experiment, where a 22-layer TinyLLaMA gets repeated blocks with adapters and gains a bit on all six benchmarks after 10B tokens, is a nice demonstration of the same machinery.\n\nThe soft spot is the parameter accounting. The paper says '30% compression,' but that counts only block removal. Because multiple pruned positions share the same base block, the LoRA adapters can't be merged into the shared weights; they have to stay as separate parameters. Appendix A concedes a 3.67% relative parameter increase over ShortGPT, which on a 7B model is roughly 180M parameters. So the real savings are closer to 27–28%, and the ShortGPT+LoRA baseline—where each retained block is unique and the adapter can be merged away—is at a true 30% with no extra inference-time parameters. That doesn't kill the method, but it makes Table 1 an unequal comparison and undercuts the 'state-of-the-art at 30%' framing until the deployed parameter counts are reported.\n\nOther concerns are the usual ones: single run per configuration, no error bars, SliceGPT and LLM Surgeon numbers copied from other papers with possibly different evaluation harnesses, and the extension margins are small enough that they could be noise. None of that is fatal to the central idea. The paper is honest about the compute-vs-parameter tradeoff and includes a genuine limitations section.\n\nI'd send this to review. The method is novel enough, the ablations are informative, and the parameter-count issue is fixable with a more careful accounting and maybe a plot of accuracy vs. actual stored parameters. If I were doing pruning work, I'd cite it as the clearest example of block replacement with weight sharing.","headline":"The weight-sharing + LoRA trick is a real contribution, but the 30% compression claim quietly ignores persistent adapter parameters—real savings are closer to 28%, and ShortGPT+LoRA gets to merge its adapters away, so Table 1 compares unequal configurations.","tokens_in":15343,"tokens_out":2468,"would_cite":true,"duration_ms":21222,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pruned LLM blocks can be repaired with shared weights and low-rank adapters.","keywords":["large language model pruning","block pruning","weight sharing","low-rank adapters","LoRA initialization","output normalization","model extension","singular value decomposition"],"falsifier":"Run the identical pipeline but choose the weight-sharing base for each pruned block at random, keeping the SVD-based adapter initialization and output normalization. If the final perplexity after 1B recovery tokens lands near the reported 6.55, the selection metric is not carrying the result; if it degrades substantially toward the ablation value of 6.77 or worse, the metric is doing real work.","tokens_in":1953,"feed_emoji":"✂️","tokens_out":7065,"duration_ms":122314,"temperature":0.7,"pith_summary":"The paper claims that a transformer block removed from a large language model can be replaced by a weight-shared copy of a similar surviving block, with a small low-rank adapter (LoRA) initialized from the SVD of the difference between the original and replacement, plus an output normalization that starts small and lets the model ease into the change. After a short recovery-training phase, this scheme restores quality better than plain block pruning, reaching 6.55 perplexity and 62.68% average zero-shot accuracy on a 30%-compressed LLaMA-2 7B model. The same machinery works in reverse: repeating blocks of a 22-layer TinyLLaMA model to 36 layers, each repetition carrying its own adapter and normalization, improves all six evaluated benchmark scores after only about 0.3% additional training tokens. If the recipe generalizes, it gives memory-constrained deployments a way to cut or grow model capacity without retraining from scratch.","feed_headline":"FlexiGPT repairs pruned LLM blocks with shared weights and LoRA","feed_subtitle":"At 30% compression, LLaMA-2 7B keeps 6.55 perplexity and 62.68% zero-shot accuracy; the same trick extends small models.","key_machinery":"The load-bearing machinery is a block-similarity score computed in a low-rank space, combined with an adapter initialization and a stabilizing scale. For each pruned block $i$, FlexiGPT scores every surviving block $j$ by $d(W_i,W_j)=\\Vert \\hat W_i-(\\hat W_j+\\Delta_{i-j})\\Vert_F$, where $\\hat W$ are rank-256 SVD reconstructions and $\\Delta_{i-j}$ is the rank-reduced approximation of the difference; the surviving block with the smallest score becomes the shared weight base. The same SVD of $W_i-W_j$ initializes the LoRA adapter, and an output layer-normalization scale is initialized small. What this machinery does is make replacement cheap and smooth: weight sharing adds almost no parameters, the adapter absorbs the residual difference, and the small normalization scale gives the model time to grow into the replacement instead of suffering a large initial perplexity jump.","core_discovery":"The central claim is that a pruned block can be replaced by another block already inside the model, and the mismatch between the two can be absorbed by a rank-256 LoRA adapter initialized from the SVD of $W_i-W_j$, while output feature normalization at a small initial scale prevents a perplexity spike during recovery. The block that becomes the shared-weight base is chosen by a distance metric on low-rank reconstructions, $d(W_i,W_j)=\\Vert \\hat W_i-(\\hat W_j+\\Delta_{i-j})\\Vert_F$, where $\\hat W$ are rank-256 SVD reconstructions and $\\Delta_{i-j}$ is the rank-reduced approximation of the difference; this makes blocks close in depth look similar and avoids collapsing every pruned block onto one base. After 1B tokens of recovery training on SlimPajama, the method reports the best perplexity and zero-shot averages among compared methods on LLaMA-2 7B at both 30% and 40% compression, with the same direction of results on LLaMA-3 8B and OPT 1.3B/6.7B. In the extension setting, repeating blocks with unique adapters turns a 22-layer TinyLLaMA 1.1B into a 36-layer model that outperforms the base on all six tasks with minimal parameter overhead.","pith_inferences":["Beyond the paper, the same weight-sharing-plus-adapter repair could be applied iteratively: prune, recover, re-score, and prune again, potentially reaching deeper compression than a single pass.","Because the distance metric makes blocks close in depth look similar, one could make the replacement choice input-dependent by scoring candidate bases against the activations of a given prompt, which might improve recovery on out-of-distribution text.","The extension result suggests a cheap route to model growth for families of small models: reuse an already-trained block as a prior and learn only adapters, standing between full pretraining and no training at all.","The weight-sharing replacement scheme should compose with other compression methods such as quantization or distillation, since the adapters and normalization operate on continuous weights and need not interfere with discrete quantization steps."],"forward_implications":["A 30%-compressed LLaMA-2 7B keeps a 6.55 perplexity and 62.68% average zero-shot accuracy after 1B recovery tokens, compared with 6.71 and 61.40% for the strongest compared pruning baseline with LoRA.","At 40% compression, FlexiGPT beats the compared block-pruning baselines on all six reported benchmarks, reaching 58.38% average zero-shot accuracy.","The same components transfer to LLaMA-3 8B and OPT 1.3B/6.7B, where 30% and 40% block removal is otherwise much more destructive to perplexity.","Extending TinyLLaMA from 22 to 36 layers with unique adapters improves average zero-shot accuracy from 55.41% to 56.13% while adding little parameter overhead and using about 0.3% of the original training budget.","Every ingredient contributes: ablating high-rank pruning in the selection score, output normalization, or SVD adapter initialization raises final perplexity to 6.77, 6.68, or 6.63 respectively, versus 6.55 for the full method."],"supporting_citations":[{"why":"Supplies the Block Influence score used to decide which blocks to prune.","marker":"(Men et al., 2024)"},{"why":"Supplies LoRA, the low-rank adapter mechanism that absorbs the block mismatch.","marker":"(Hu et al., 2021)"},{"why":"Supplies layer normalization, the basis for the output normalization that stabilizes recovery.","marker":"(Ba et al., 2016)"},{"why":"Defines the transformer block structure that the method prunes, shares, and extends.","marker":"(Vaswani et al., 2017)"},{"why":"Provides LLaMA-2 7B, the main model for the pruning evaluation.","marker":"(Touvron et al., 2023)"},{"why":"Provides LLaMA-3 8B, one of the pruning evaluation models.","marker":"(AI@Meta, 2024)"},{"why":"Supplies the SlimPajama corpus used for post-pruning recovery and extension training.","marker":"(Soboleva et al., 2023)"},{"why":"Supplies the MiniPile validation subset used for Block Influence computation and perplexity evaluation.","marker":"(Kaddour, 2023)"},{"why":"Provides TinyLLaMA, the base model for the extension experiments.","marker":"(Zhang et al., 2024)"},{"why":"Provides the evaluation harness used to measure zero-shot benchmark accuracy.","marker":"(Gao et al., 2021)"}],"fun_headline_variants":["FlexiGPT swaps pruned LLM blocks for shared weights and LoRA","LLM compression: replace pruned blocks with shared weights + LoRA","FlexiGPT: prune and extend LLMs via low-rank weight sharing","Shared weights and LoRA let LLMs survive 40% pruning","FlexiGPT boosts pruned LLMs by reusing internal blocks"],"cache_read_input_tokens":17280,"weakest_assumption_plain":"The whole scheme leans on the belief that the low-rank SVD distance score, with its hand-chosen rank of 256 and the additional difference term, really identifies blocks whose functions a small adapter can restore; if that similarity measure points at the wrong block, the normalization and adapter tricks have nothing good to adapt.","fun_headline_variants_meta":{"raw":{"variants":["FlexiGPT swaps pruned LLM blocks for shared weights and LoRA","LLM compression: replace pruned blocks with shared weights + LoRA","FlexiGPT: prune and extend LLMs via low-rank weight sharing","Shared weights and LoRA let LLMs survive 40% pruning","FlexiGPT boosts pruned LLMs by reusing internal blocks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000409,"raw_usage":{"total_tokens":2165,"prompt_tokens":1030,"completion_tokens":1135,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":1040}},"tokens_in":646,"tokens_out":1135,"duration_ms":8238,"temperature":1.0,"reasoning_tokens":1040,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:52:34.232207+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical pipeline but choose the weight-sharing base for each pruned block at random, keeping the SVD-based adapter initialization and output normalization. If the final perplexity after 1B recovery tokens lands near the reported 6.55, the selection metric is not carrying the result; if it degrades substantially toward the ablation value of 6.77 or worse, the metric is doing real work.","supporting_citations":[{"cited_title":"For perplexity perfor- mance evaluations, we used the validation MiniP- ile (Kaddour,","cited_arxiv_id":null,"evidence_quote":"Provides the evaluation harness used to measure zero-shot benchmark accuracy."}],"review_version":1}