{"id":"80b6fcbb-eca3-4528-926e-3eaa589be64c","arxiv_id":"2501.18596","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Sharing weights between transformer layers with low-rank delta corrections compresses LLMs by 12 to 25 percent while retaining about 90 percent of benchmark performance.","lead":"DeltaLLM compresses large language models by making neighboring layers share weights and adding small, trainable low-rank correction matrices to recover the lost performance. With only 30 to 40 million training tokens, the method keeps most of the original model's accuracy while cutting 12 to 25 percent of its parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 90% retention and parity with from-scratch training rest on a single thin experiment: 4 tasks, no error bars, and a compressed Llama that drops from 0.58 to 0.50 average accuracy.","rationale":"The reader's weakest_assumption is that low-rank deltas trained on 30-40M tokens can recover the layer's function without updating shared weights. My concern is more specific about where that assumption directly supports the headline numeric claims: the 90% retention figure, the parity-with-from-scratch claim, and the DeltaPhi-2.9B 'similar results' claim all rest on Table 1 and Table 2 without error bars or seed variance. The reader identified the same underlying assumption, so my agreement is 'agree'. However, I want to sharpen the test: the single most load-bearing claim is that 37M tokens beats billion-token from-scratch training, and that is only tested indirectly. A matched from-scratch baseline is the decisive experiment. The concern is not that the method is wrong; it is that the headline quantitative claims are underdetermined by the reported evidence. The reader's CONDITIONAL verdict is appropriate: the architecture is plausible, the ablations are internally consistent, and quantization results show the compressed form is robust, but reproducibility is blocked by missing code and model links. If the matched-budget experiment shows DeltaPhi wins, the verdict could move to ACCEPT. If it shows the baseline wins, the central claim fails and REJECT would be warranted. Until then, CONDITIONAL is the fair verdict.","tokens_in":17421,"tokens_out":2093,"duration_ms":16694,"concrete_test":"Train one matched-budget from-scratch baseline: take the identical Phi-3.5 architecture trimmed to 3.35B parameters (e.g., by reducing intermediate MLP dimension and number of layers), train it on the same 37M Ultrachat tokens with the same hyperparameter budget, and compare zero-shot accuracy on the five Table 1 tasks against DeltaPhi 3.35B. If the from-scratch baseline matches or beats DeltaPhi, the '30-40M tokens suffices to match from-scratch training' claim fails. Additionally, run each DeltaLLM benchmark evaluation across 3-5 random seeds and report mean plus standard deviation; if the DeltaPhi 3.35B versus Qwen 2.5 gap on the average is within one standard deviation, the parity claim is not established.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The abstract's two central claims are (1) DeltaLLM retains 90% of base-model performance at 12% compression, and (2) 30-40M tokens of delta-only training matches training from scratch. Table 1 is the only direct evidence: DeltaPhi 3.35B drops from 0.66 to 0.59 average (89% retention), but DeltaLlama 2.52B drops from 0.58 to 0.51 (88%), and DeltaLlama 2.41B drops to 0.50 (86%). These numbers are on a 5-task average with no standard errors; individual tasks like ARC-Challenge drop from 0.61 to 0.35 (57%) for DeltaLlama. The claim of parity with from-scratch training is supported only by a comparison to Qwen 2.5 3.2B and Llama 3.2 3.2B at different parameter counts and training budgets; no matched-budget, matched-architecture from-scratch baseline is trained by the authors. The method assumes rank-r deltas (r=100, giving about 90MB for the whole Phi 2.9B model) trained on ~37M tokens can re-specialize shared MLP weights across 6-9 blocks. This is plausible but unanalyzed: the paper provides no scaling curve as a function of rank, number of compressed blocks, or training tokens, so there is no evidence about when the assumption breaks. The 24%-compressed DeltaPhi 2.9B actually drops on ARC-Challenge from 0.51 to 0.44 and HellaSwag from 0.70 to 0.61, so the 'similar results' claim holds only on MMLU-Pro and WinoGrande. The strongest unguarded statement is that 37M tokens suffices to match billion-token training; the only support is an indirect comparison to Qwen/Llama with different vocab sizes, data, and parameter counts. If those benchmark deltas are within run-to-run noise—which the paper cannot rule out because no seeds or error bars are reported—the abstract's quantified claims are not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DeltaLLM, a post-training compression scheme that restructures selected Transformer blocks as a shared anchor weight plus a low-rank delta matrix (Eqs. 1-2), and trains only the delta modules via progressive module replacement and knowledge distillation from the original model. The authors compress Phi-3.5 and Llama-3.2 into DeltaPhi and DeltaLlama variants with 12-25% parameter reductions, and report zero-shot benchmark results in Table 1, comparisons to JointDrop/SliceGPT/ShortGPT/LaCo in Tables 2-3, quantization results in Table 4, and ablations in Tables 5-6. The central claims are that 30-40M tokens of delta-only training suffices to match models trained from scratch, and that the compressed models retain about 90% of base-model performance.","tokens_in":17939,"tokens_out":5923,"duration_ms":49348,"significance":"If the token-efficiency claim held, DeltaLLM would be a practically valuable contribution for on-device deployment: 30-40M tokens is orders of magnitude cheaper than pruning-plus-retraining pipelines such as MINITRON, which require billions of tokens. The paper combines known ingredients (cross-layer weight sharing, LoRA-style deltas, progressive module replacement) in a new configuration for decoder-only LLMs, and it includes useful ablations on block selection, layer type, and delta initialization. The release of DeltaPhi/DeltaLlama models is a further strength. However, the headline comparisons to from-scratch training and the '90% retention' figure rest on thin evidence, so the significance is conditional on additional controlled experiments.","major_comments":[{"comment":"The claim that 30-40M tokens of delta-only training is 'sufficient to achieve performance on par with LLMs of comparable sizes trained from scratch' is not supported by the evidence presented. Table 1 compares DeltaPhi/DeltaLlama to Qwen 2.5 and Llama 3.2, which are public models trained on different data, with different token budgets, and at different parameter counts; no matched-architecture, matched-budget from-scratch baseline is trained by the authors. At minimum, the paper should either train a small from-scratch model on the same data and compute budget, or rephrase the claim as 'competitive with existing SLMs of similar size.'","section":"Abstract and Section 4.2, Table 1"},{"comment":"The abstract's 'retaining 90% of the performance' is not reproducible from Table 1: DeltaLlama 2.52B drops from 0.58 to 0.51 average (88%) and DeltaLlama 2.41B drops to 0.50 (86%); on ARC-Challenge DeltaLlama 2.52B drops from 0.61 to 0.35 (57%). The averages have no standard errors, no confidence intervals, and no multiple-seed variation, and the benchmark set has only five tasks. Please report per-task variance or at least state the range explicitly rather than a single rounded percentage.","section":"Table 1 and Section 4.2"},{"comment":"The method relies on the empirical premise that a rank-r delta (r=100 or r=1000) trained on 32-37M tokens can re-specialize shared MLP or attention weights across several blocks, but the paper offers no analysis of when this premise fails. There is no scaling curve varying rank, number of compressed blocks, or number of training tokens; Table 1 only gives two operating points per model family. Such curves are needed to justify the choice of r and to bound compression limits, especially since the larger compression variants show substantial per-task drops.","section":"Section 3.2, Eq. (2), and Section 4.1"},{"comment":"The comparison to SOTA compression methods is fairer than the abstract suggests. Table 2 shows LaCo on Phi achieving higher MMLU-Pro (0.33 vs 0.32) and WinoGrande (0.71 vs 0.70) than DeltaPhi, with no training; the paper's text acknowledges this only partially. In Table 3 the baseline compression percentages vary from 18% to 25% while the DeltaLlama variants are 21% and 25%, so the 'same number of parameters removed' phrasing is imprecise. Please report the training data and token budgets used for all baselines and use matched parameter reductions or, failing that, state the mismatch explicitly.","section":"Section 4.6, Tables 2 and 3"}],"minor_comments":[{"comment":"The text says 'using the procedure outlined in Section 4', but the training procedure is described in Section 3.3; the cross-reference should be corrected.","section":"Section 4.1"},{"comment":"The loss term L_logits is not defined in Eq. (3); the surrounding text mentions KL divergence only later. Please define the distillation loss formally at first use.","section":"Eq. (3)"},{"comment":"There is an inconsistency in the reported rank for DeltaLlama 2.41B: Table 1 lists r100/30MB, while Table 7 lists 'DELTALLAMA (9 seq MLP δ, r1000) 2.41B 25%'. Please reconcile the two entries.","section":"Tables 1 and 7"},{"comment":"The y-axis label 'WinoGrade' is misspelled; it should be 'WinoGrande'.","section":"Figure 3"},{"comment":"The PMR claim is based on comparing different epoch counts (PMR 2 epochs vs no-PMR 5 epochs on Alpaca), but on Ultrachat the no-PMR model achieves lower perplexity (7.24 vs 7.49). The text should qualify the 'faster convergence' claim and explain the Ultrachat discrepancy.","section":"Section 4.4, Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core architecture idea is plausible, but the two headline claims need stronger evidence before the paper can be accepted. The from-scratch parity claim is particularly problematic because it is stated in the abstract without a controlled baseline. I would also flag that related work on recursive transformers with layer-wise LoRA (Bae et al., 2024) is cited but not compared experimentally, which weakens the novelty narrative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you care about post-training compression for on-device LLMs. The combination—cross-layer weight sharing with low-rank deltas, trained via progressive module replacement and distillation—is a genuine extension of BERT-of-Theseus-style PMR to decoder-only models, and the ablations on block/layer choice (Section 5.7) are the best part of the paper. The reported storage savings (12-25% fewer parameters, 90MB of deltas for the 2.9B Phi) are plausible, and the orthogonality to quantization is a nice practical touch.\n\nThe weak spots are the two headline claims. \"On par with models trained from scratch\" is supported only by comparisons to Qwen and Llama checkpoints with different data and training budgets; the authors never train a matched-budget from-scratch baseline. The retention numbers are averages over five tasks with no error bars or seeds. Look at Table 1 directly: DeltaLlama drops from 0.58 to 0.50 average, and ARC-Challenge goes from 0.61 to 0.35—well below 90% retention on a task that matters. The \"similar results\" for DeltaPhi 2.9B versus SlicedPhi holds on MMLU-Pro and WinoGrande but not on ARC or HellaSwag. So the abstract's precision ('retaining 90%', 'on par') outruns the evidence.\n\nAlso, the core expressiveness assumption—rank-100 deltas trained on ~37M tokens can re-specialize shared MLP weights across 6-9 blocks—is never stress-tested. No scaling curves in rank, token count, or number of compressed blocks, so I can't tell when the method breaks down. The paper itself flags some of this in Section 5, which I appreciate, but the abstract doesn't carry those caveats.\n\nOn the practical side, the comparisons to JointDrop, ShortGPT, SliceGPT and LaCo look fair: same compression levels, and the Delta models beat them on average accuracy. The abstract says the models are released, though I couldn't find links in the v2 text—worth checking the final version.\n\nMy recommendation: this deserves a serious referee, not a desk reject. It's a competent empirical study with a useful result, but it needs revision before acceptance: error bars or seeds, a matched from-scratch baseline (or reworded claims), and ideally code and model artifacts. If I were editing, I'd send it out with a request for those changes.","headline":"A useful compression recipe with honest ablations; the headline claims about parity with from-scratch training and 90% retention outrun the evidence.","tokens_in":18425,"tokens_out":2934,"would_cite":true,"duration_ms":24893,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DeltaLLM compresses LLMs by letting later layers share an anchor layer's weights plus small trained low-rank delta matrices; with roughly 30–40M tokens of delta-only training it matches similarly sized models trained from scratch and…","keywords":["LLM compression","weight sharing","low-rank delta","progressive module replacement","knowledge distillation","post-training compression","transformer redundancy","parameter efficiency"],"falsifier":"After a DeltaLLM model is trained, set all delta matrices to zero and re-run the same benchmarks; if the zeroed-delta model scores within noise of the full model, the deltas are not doing the work the paper attributes to them, which would falsify the central mechanism.","tokens_in":17202,"feed_emoji":"🧩","tokens_out":12667,"duration_ms":106453,"temperature":0.7,"pith_summary":"DeltaLLM introduces a post-training compression scheme that restructures a transformer so that several later layers reuse the weights of an earlier anchor layer, with each reused layer's difference from the original captured by a small low-rank matrix called a delta. The paper claims that training only these delta modules with progressive module replacement and knowledge distillation on roughly 30 to 40 million tokens is sufficient to reach performance on par with similarly sized models trained from scratch, while a 12% parameter reduction retains about 90% of the base model's average accuracy on common reasoning benchmarks. The method is also reported to outperform JointDrop, SliceGPT, ShortGPT, and LaCo at comparable parameter savings, and it tolerates additional quantization. If correct, this gives a cheap way to shrink LLMs for storage-constrained devices without removing computations, and it suggests that much of the difference between adjacent transformer layers is low-rank.","feed_headline":"Low-rank deltas let LLMs share layers and keep 90% accuracy","feed_subtitle":"Replacing later layers with shared anchors plus trained low-rank differences needs only 30-40M tokens.","key_machinery":"The load-bearing mechanism is the weight-decomposition identity of Eq. (1): every replaced layer $l+i$ is written as the anchor weight $W_l$ plus a delta $\\tilde{\\delta}_l^{l+i}$, a low-rank approximation of $W_{l+i} - W_l$ parameterized in the same two-matrix $A B$ form as LoRA. The delta modules are trained while the shared anchor weights remain frozen, under a progressive module replacement schedule in which original layers are gradually swapped for the delta layers; the loss combines cross-entropy with a KL-divergence distillation term from the original model. This configuration is what lets the whole compression run on tens of millions of tokens rather than billions, because the search space is small and the teacher provides dense signal.","core_discovery":"The central discovery is that a pretrained transformer layer can be re-expressed as an anchor layer's weights plus a low-rank delta, $W_{l+i} = W_l + \\tilde{\\delta}_l^{l+i}$, where $\\tilde{\\delta}$ is a low-rank approximation of the actual difference $W_{l+i} - W_l$. The paper shows that when several later blocks share an anchor block, storing the anchor once plus the deltas saves parameters, and that training only these deltas—using progressive module replacement and a KL-divergence distillation loss—on 37M tokens for DeltaPhi and 32M tokens for DeltaLlama yields models whose average accuracy on MMLU-Pro, WinoGrande, ARC-Challenge, HellaSwag, and PIQA stays within about 90% of the base model while parameters drop by 12%. A 24%-reduced DeltaPhi 2.9B matches a recovery fine-tuned SlicedPhi 3.3B that is roughly 400M parameters larger, despite DeltaPhi receiving no recovery fine-tuning. These properties hold for both the Phi and Llama families, with MLP-sharing variants giving lower perplexities than attention-sharing variants at the same block count.","pith_inferences":["If the low-rank delta structure is as expressive as reported, the difference between adjacent transformer layers should exhibit a small effective rank on natural pretrained models; this could be verified directly by SVD on layer weight differences, and a positive result would explain why the deltas recover performance so cheaply.","The same anchor-plus-delta recipe could plausibly be used to pretrain small models from scratch by progressively growing depth, turning the compression method into an architecture design principle; the paper only gestures at this as a future direction.","Because the deltas are small, they could be stored at higher precision than quantized anchors with only a small memory cost, a trade-off that the paper's quantization results suggest but do not optimize.","A natural testable extension is to vary the data budget downward (e.g., 5M, 10M, 20M tokens) and measure the performance curve; if performance plateaus well before 30M tokens, even cheaper compression is possible."],"forward_implications":["A 12% parameter reduction retains roughly 90% of the base model's average zero-shot accuracy across MMLU-Pro, WinoGrande, ARC-Challenge, HellaSwag, and PIQA.","Training only the delta modules (frozen anchors) with progressive module replacement converges faster than plain distillation, so the whole compression can be done with roughly 30–40M tokens instead of billions.","Compressing MLP layers yields lower perplexity than compressing attention layers at the same number of replaced blocks, so the method's recommended use is MLP weight sharing.","The method is orthogonal to quantization: 8-bit and 4-bit quantization of the anchors degrades performance only slightly, and keeping the small delta layers in FP16 gives a marginal gain.","DeltaPhi 2.9B with a 24% reduction and no recovery fine-tuning matches a recovery fine-tuned SlicedPhi 3.3B with a 12% reduction, despite being about 400M parameters smaller."],"supporting_citations":[{"why":"Supplies the progressive module replacement schedule that DeltaLLM adapts to train the delta layers.","marker":"Xu et al. (2020)"},{"why":"Defines the low-rank matrix parameterization (LoRA) used to build each delta from two small matrices.","marker":"Hu et al. (2021)"},{"why":"Provides the KL-divergence distillation loss choice and the pruning-plus-distillation recipe that the method follows.","marker":"Muralidharan et al. (2024)"},{"why":"Supplies the JointDrop baseline and the similarity-based layer-importance metric that informs which layers to compress.","marker":"He et al. (2024)"},{"why":"Provides the ShortGPT baseline and evidence that middle and later transformer layers are redundant, motivating anchor-plus-delta sharing.","marker":"Men et al. (2024)"},{"why":"Provides the SliceGPT baseline that DeltaPhi 2.9B is compared against after recovery fine-tuning.","marker":"Ashkboos et al. (2024a)"},{"why":"Provides the LaCo baseline that the paper compares against at equal parameter savings.","marker":"Yang et al. (2024)"},{"why":"The Phi-3 teacher model from which DeltaPhi is compressed.","marker":"Abdin et al. (2024)"},{"why":"The Llama-3 teacher model from which DeltaLlama is compressed.","marker":"Dubey et al. (2024)"}],"fun_headline_variants":["Shared weights plus low-rank deltas shrink LLMs by 12%","LLMs share layers and add low-rank deltas, saving 12% params","Low-rank deltas on shared weights keep 90% accuracy with 12% fewer params","Train only low-rank deltas on shared layers: 12% smaller, 90% accuracy","DeltaLLM: share weights, add low-rank deltas, match larger models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a frozen anchor layer plus a low-rank delta trained on only 30 to 40 million tokens can reproduce the replaced layer's function on the deployment distribution, without any updating of the shared weights.","fun_headline_variants_meta":{"raw":{"variants":["Shared weights plus low-rank deltas shrink LLMs by 12%","LLMs share layers and add low-rank deltas, saving 12% params","Low-rank deltas on shared weights keep 90% accuracy with 12% fewer params","Train only low-rank deltas on shared layers: 12% smaller, 90% accuracy","DeltaLLM: share weights, add low-rank deltas, match larger models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000596,"raw_usage":{"total_tokens":2827,"prompt_tokens":1020,"completion_tokens":1807,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":1698}},"tokens_in":636,"tokens_out":1807,"duration_ms":11872,"temperature":1.0,"reasoning_tokens":1698,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T22:52:11.957977+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"After a DeltaLLM model is trained, set all delta matrices to zero and re-run the same benchmarks; if the zeroed-delta model scores within noise of the full model, the deltas are not doing the work the paper attributes to them, which would falsify the central mechanism.","supporting_citations":[],"review_version":1}