{"id":"177a8af3-9168-4d69-85cc-0a09f9804ed7","arxiv_id":"2505.11344","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A dynamic base-model shift plus two per-task scalars recovers much of the accuracy lost by extreme delta pruning and quantization, though the evaluation tunes the scalars on test-set samples.","lead":"This paper proposes a small add-on for delta compression: instead of always subtracting the original pretrained model from a fine-tuned model, it first shifts the base model toward an average of all fine-tuned models, then rescales the compressed difference. The authors report large accuracy recoveries at extreme compression, but the headline numbers come from tuning two scalars on part of the test data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains are confounded by fitting λ on test-set subsets; evaluation needs a clean held-out split.","rationale":"The strongest claim requires that DBMS improve over data-free baselines. The current protocol violates that comparison because test inputs are used for model selection while baselines receive no data. A transductive setup can be legitimate only if baselines are given the same unlabeled inputs and if metrics are reported on data not used for fitting; neither condition holds. The paper's own Appendix E acknowledges the need for partial unlabeled data and additional training, but it does not address benchmark contamination. Appendix A.2's variance argument is also only a norm decomposition and does not by itself prove that lower variance makes pruning or quantization easier; however, the evaluation flaw is the primary blocker. If a clean held-out evaluation confirms the gains, the paper would present a useful orthogonal method.","tokens_in":15967,"tokens_out":2159,"duration_ms":25524,"concrete_test":"Hold out a disjoint split: fit λ per task on 10% of the original test inputs, then evaluate on the remaining 90% (or on an entirely separate benchmark collection). Re-run Tables 1, 2, 5, and 6 with this protocol. If the average gains over DARE and BitDelta shrink to near zero, the reported advantage is largely an artifact of transductive test-set fitting; if the gains persist, the dynamic-base claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that shifting the base toward the averaged task vector and rescaling the compressed delta preserves accuracy at 99.8% compression—rests on Tables 1–7. In Section 3.2.2 (Eq. 8), λ1 and λ2 are optimized on X_t, described as 10% of the test inputs, and all reported numbers are on the same benchmarks. This gives DBMS access to test inputs that the baselines never see, and the evaluation set contains the very samples used for fitting. The comparison is therefore not a fair test of the plug-in: the large gains (e.g., DARE 99.8% on ViT-B/32 from 8.9 to 80.8) may largely reflect two per-task scalars fit to the benchmark's own inputs, rather than the dynamic-base construction. The closed-form initialization in Eq. 7 is a legitimate parameter-free contribution, but the headline post-training numbers do not cleanly support it. Appendix E acknowledges the need for partial unlabeled data and training cost, but does not address the benchmark-contamination issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Dynamic Base Model Shift (DBMS), a plug-in method for delta compression. Instead of using the pretrained model as the static base, DBMS constructs a per-task shifted base W_base = W_pre + λ1 * BitDelta(W_avg - W_pre), then applies a per-task rescaling λ2 to the compressed delta before reconstruction via Eq. 9. The two scalars are initialized in closed form (Eq. 7) and then refined by MSE minimization on 10% of the test inputs (Eq. 8). Experiments cover GPT-2, RoBERTa, Flan-T5, ViT-B/32, ViT-L/14, and BEiT3, reporting large gains at high compression rates (e.g., DARE 99.8% on ViT-B/32 from 8.9 to 80.8). The claim is that the pretrained base is suboptimal and that the dynamic shift plus rescale preserves accuracy at extreme compression.","tokens_in":16285,"tokens_out":3007,"duration_ms":32558,"significance":"If the empirical claims survive a clean evaluation protocol, DBMS would be a useful and refreshingly simple contribution: the closed-form initialization of λ1 in Eq. 7 is parameter-free, and the method is explicitly orthogonal to both pruning-based (DARE) and quantization-based (BitDelta) compressors, with experiments spanning multiple modalities. The paper also identifies a real limitation of existing delta-compression work. However, the headline post-training results are currently confounded by test-set adaptation, so the significance is contingent on a re-evaluation with a proper held-out split.","major_comments":[{"comment":"The post-training evaluation is circular in a benchmark sense: λ1 and λ2 are optimized on X_t, described in Section 3.2.2 as '10% of the input data randomly selected from the test datasets', and the reported accuracies in Tables 1-7 are measured on the full test benchmarks, which include those same 10% of inputs. Baselines such as DARE and BitDelta never see any test inputs, so the comparison is not apples-to-apples. The gains attributed to the dynamic-base construction may largely reflect two per-task scalars fitted to the evaluation distribution itself. Please re-run the protocol with a disjoint split: either fit on training/validation data and evaluate on the full test set, or fit on a 10% test subset and report accuracy only on the remaining 90%. The paper should also state clearly which evaluation protocol was used for every table.","section":"Section 3.2.2 (Eq. 8) and Tables 1-7"},{"comment":"The theoretical support is mathematically unsound as written. From the least-squares initialization, minimizing ||W_pre + λ1 τ_base - W_t||^2 implies τ_base · (W_t - W_pre - λ1 τ_base) = 0, so the vectors λ1 τ_base and Δ_ours are orthogonal as vectors. However, Eq. 17 then writes Var(Δ_ori) = Var(Δ_ours) + (λ1)^2 Var(τ_base), which is not implied by vector orthogonality. Variance of a sum of random vectors requires zero-mean or uncorrelated components; elementwise orthogonality of two fixed vectors does not make their variances additive. This step needs a corrected derivation, an explicit zero-mean assumption on the delta parameters, or removal of this theoretical claim.","section":"Appendix A.2, Eq. 17"}],"minor_comments":[{"comment":"The training loop description is slightly inconsistent: Algorithm 1 loops once over x in X[t], while Figure 3 reports loss curves for 'the first 100 epochs'. Please clarify how many epochs/iterations are used and how X_t is sampled in each epoch.","section":"Section 3.2.2 / Algorithm 1"},{"comment":"There are several typos: 'commpression' (Section 4.1), 'Trainig' (Table 7 header), 'perfromance' (Section 4.1), 'initaliztion' (Figure 3 caption). A careful proofread is needed.","section":"Throughout"},{"comment":"The method requires 'partial unlabeled test data', but Appendix E lists this only as a computational cost and never discusses the legitimacy of using test inputs during model construction. Even if the data are unlabeled, using the test distribution for fitting parameters is a benchmark-contamination concern that should be addressed explicitly in the limitations.","section":"Section 3.2.2 / Appendix E"},{"comment":"Table 4 reports λ1 and λ2 before and after training for RoBERTa, but the values are not analyzed. For example, several λ2 values after DARE 99.8% training are far below 1 (e.g., 0.1258 for MNLI), which suggests the rescaling is absorbing most of the compression error. A brief discussion of why these values are sensible would strengthen the paper.","section":"Section 4.1 and Table 4"}],"recommendation":"major_revision","confidential_remarks":"The test-set contamination is the central issue; it directly undermines every headline post-training number. The good news is that the fix is local and feasible: re-run with a validation split and report held-out accuracy, and the closed-form initialization part could still stand. I would also ask the authors to either fix or remove the Appendix A.2 variance argument, since as written it is incorrect. If the re-run shows that most of the gain disappears without test-set fitting, the contribution shrinks considerably; the paper should then be honest about that."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper proposes a simple, orthogonal plug-in for delta compression. Instead of using the pretrained model as the base, you shift the base toward a compressed average task vector and rescale the compressed delta, with two per-task scalars. The closed-form initialization for λ1 is parameter-free and is a legitimate small contribution. The experimental breadth is real: language, vision, and multimodal models, with both pruning and quantization baselines. If the evaluation were clean, this would be a useful recipe for extreme compression ratios.\n\nThe soft spot is large and central. In Section 3.2.2, λ1 and λ2 are fit on 10% of the test inputs, and the reported tables measure accuracy on those same benchmarks. The baselines never see that data. The Init-only results are often no better than the baselines (e.g., ViT-B/32 at 99.8%: 8.1 vs 8.9), so the headline gains come from scalars tuned on test samples. That is a confound, not a small detail. It does not kill the idea, but it means the paper's main claim—that dynamic base shift preserves accuracy at 99.8% compression—is not supported by the numbers as presented. The right fix is a held-out split: fit on a training or development subset, evaluate on the test set, and give the baselines the same transductive budget if you want to keep the protocol.\n\nI would also treat Appendix A.2 as intuition, not theory. The orthogonality argument does not imply the variance decomposition in Eq. 17, since zero dot product does not make two high-dimensional vectors uncorrelated across coordinates. That is a minor issue because the practical method does not rely on it.\n\nWhat the paper does well: it identifies a real gap (static base is suboptimal), proposes a concrete mechanism that is orthogonal to existing compressors, and reports storage overhead honestly. The idea deserves serious attention. But the evaluation protocol needs to be redone before any of the numbers can be trusted.\n\nMy recommendation: send it to peer review, but with a clear requirement that the post-training results be validated on held-out data or against transductive baselines. As it stands, the closed-form initialization is publishable as a small result, but the headline claims are not yet established. I would bring it to a reading group for discussion, mainly to debate the evaluation protocol.","headline":"Nice idea, real evaluation flaw: fitting two scalars on 10% of each test set makes the headline gains hard to trust until rerun on a clean split.","tokens_in":16701,"tokens_out":1688,"would_cite":false,"duration_ms":19953,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Shifting the base model toward each task keeps fine-tuned accuracy nearly intact at 99.8% delta compression.","keywords":["delta compression","task vectors","model merging","1-bit quantization","parameter pruning","base model shift","fine-tuned model storage","multi-task adaptation"],"falsifier":"On one benchmark, keep the exact DBMS protocol except optimize $\\lambda_1$ and $\\lambda_2$ on the training split, or on a separate held-out validation set, instead of on 10% of the test set, then evaluate on the test set at 99.8% sparsity. If average accuracy falls back toward the vanilla DARE baseline (near 8.9 on ViT-B/32) instead of reaching roughly 80.8, the headline gain is test-set adaptation. A complementary check: replace $\\tau_{\\text{base}}$ with a fixed random direction of the same magnitude; if the gains persist, the shared task-average direction is not the active mechanism.","tokens_in":15744,"feed_emoji":"🗜️","tokens_out":8100,"duration_ms":75888,"temperature":0.7,"pith_summary":"The paper tries to overturn a default in delta compression: instead of always compressing the difference between a fine-tuned model and the frozen pretrained weights, it shifts the base model toward the target task before compressing. The shift is controlled by two numbers per task, lambda_1 and lambda_2, chosen by low-cost optimization against the fine-tuned model's outputs. The claim is that this restores most of the fine-tuned model's accuracy at extreme compression rates, where existing methods collapse. If true, it means the storage savings of delta compression can be kept without sacrificing task quality, and the recipe can be bolted onto both pruning-based and quantization-based compressors.","feed_headline":"A shifted base rescues models at 99.8% compression","feed_subtitle":"Two per-task scalars move the compression base off the pretrained weights and restore most fine-tuned accuracy.","key_machinery":"The load-bearing object is the reconstruction identity in Eq. 9: $W'_t = W_{\\text{pre}} + \\lambda_{1,t}\\tau_{\\text{base}} + \\lambda_{2,t} C(W_t - W_{\\text{pre}} - \\lambda_{1,t}\\tau_{\\text{base}})$, with $\\tau_{\\text{base}} = \\text{BitDelta}(W_{\\text{avg}} - W_{\\text{pre}})$ shared across tasks. Here $\\tau_{\\text{base}}$ is a 1-bit encoded vector capturing the average direction in which all tasks moved away from the pretrained model, $C$ is the delta compressor (DARE or BitDelta), and $\\lambda_1, \\lambda_2$ are per-task scalars. The base shift lowers the variance of the remaining delta, making the same pruning or quantization operator produce better reconstructions, and $\\lambda_2$ then rescales the compressed per-task delta.","core_discovery":"The paper's central claim is that the pretrained model is only one possible base and usually not the best one. Its Dynamic Base Model Shift builds a shared shifted base $W_{\\text{pre}} + \\lambda_1 \\tau_{\\text{base}}$, where $\\tau_{\\text{base}}$ is the 1-bit compressed average of all task vectors, and then reconstructs each task as $W'_t = W_{\\text{pre}} + \\lambda_{1,t} \\tau_{\\text{base}} + \\lambda_{2,t} C(W_t - W_{\\text{pre}} - \\lambda_{1,t} \\tau_{\\text{base}})$. The two scalars are trained per task by minimizing the squared output difference against the fine-tuned model on a small data sample. The paper reports that this recipe rescues heavily compressed models: at 99.8% DARE sparsity, ViT-B/32 average accuracy rises from 8.9 to 80.8 and ViT-L/14 from 45.7 to 90.2, and similar gains appear with BitDelta-style 1-bit compression and on language, vision, and multi-modal models.","pith_inferences":["If the shift mechanism is real, base selection is a degree of freedom that other compression and merging methods could exploit; for example, a cluster of highly related tasks might benefit from its own shared base vector instead of one global average.","A natural extension is to make $\\lambda_1$ and $\\lambda_2$ per layer or per attention block rather than global per task; the paper's variance-reduction story predicts larger gains at 99.8% compression when the shift is applied where the delta is largest.","Because the tuning sample is drawn from the test sets, an evaluation that tunes on training data only would clarify how much of the gain generalizes; the paper does not report that experiment."],"forward_implications":["At extreme compression ratios, where vanilla DARE collapses (ViT-B/32 average 8.9 at 99.8% sparsity), DBMS restores usable accuracy (80.8), so very high delta sparsity becomes practical.","The same two-scalar recipe also improves BitDelta-style 1-bit quantization, not just pruning, so it operates at the level of the compression paradigm rather than one algorithm.","The shared base vector $\\tau_{\\text{base}}$ is stored once in 1-bit form plus two scalars per task, so the overhead is small when many fine-tuned models are stored from one pretrained checkpoint.","The pattern holds across language models (RoBERTa, GPT-2, Flan-T5), vision transformers (ViT-B/32, ViT-L/14), and a multi-modal model (BEiT3), suggesting the shift benefit is not specific to one architecture or modality."],"supporting_citations":[{"why":"BitDelta: the 1-bit quantization baseline and the compressor used to build the shared base vector $\\tau_{\\text{base}}$.","marker":"[29]"},{"why":"DARE: the random-pruning baseline that DBMS is combined with and compared against across all experiments.","marker":"[53]"},{"why":"Task arithmetic: defines delta parameters as task vectors, the object that delta compression stores, and supplies the vision evaluation setting.","marker":"[18]"},{"why":"Model soups: shows that averaging fine-tuned weights is a meaningful base, motivating the average merged model $W_{\\text{avg}}$.","marker":"[48]"},{"why":"GLUE: the benchmark whose language tasks provide the main evaluation numbers for RoBERTa, GPT-2, and Flan-T5.","marker":"[44]"}],"fun_headline_variants":["Shifted base lifts accuracy from 8.9 to 80.8 at 99.8% sparsity","Two scalars shift the base and rescue 99.8%-sparse models","Dynamic base shift keeps 80.8% accuracy at 99.8% sparsity","Move the base, not just the delta: 80.8% accuracy at 99.8% sparsity","Two learned scalars shift the compression base, rescue accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that tuning $\\lambda_1$ and $\\lambda_2$ on a random 10% slice of the very test sets used to report accuracy is a fair comparison with baselines that never see test data; if that tuning leaks signal from the test set, the reported gains are inflated.","fun_headline_variants_meta":{"raw":{"variants":["Shifted base lifts accuracy from 8.9 to 80.8 at 99.8% sparsity","Two scalars shift the base and rescue 99.8%-sparse models","Dynamic base shift keeps 80.8% accuracy at 99.8% sparsity","Move the base, not just the delta: 80.8% accuracy at 99.8% sparsity","Two learned scalars shift the compression base, rescue accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000951,"raw_usage":{"total_tokens":4100,"prompt_tokens":1031,"completion_tokens":3069,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":2952}},"tokens_in":647,"tokens_out":3069,"duration_ms":18868,"temperature":1.0,"reasoning_tokens":2952,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:53:23.541360+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On one benchmark, keep the exact DBMS protocol except optimize $\\lambda_1$ and $\\lambda_2$ on the training split, or on a separate held-out validation set, instead of on 10% of the test set, then evaluate on the test set at 99.8% sparsity. If average accuracy falls back toward the vanilla DARE baseline (near 8.9 on ViT-B/32) instead of reaching roughly 80.8, the headline gain is test-set adaptation. A complementary check: replace $\\tau_{\\text{base}}$ with a fixed random direction of the same magnitude; if the gains persist, the shared task-average direction is not the active mechanism.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BitDelta: the 1-bit quantization baseline and the compressor used to build the shared base vector $\\tau_{\\text{base}}$."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DARE: the random-pruning baseline that DBMS is combined with and compared against across all experiments."},{"cited_title":"Ilharco, M","cited_arxiv_id":null,"evidence_quote":"Task arithmetic: defines delta parameters as task vectors, the object that delta compression stores, and supplies the vision evaluation setting."},{"cited_title":"Wortsman, G","cited_arxiv_id":null,"evidence_quote":"Model soups: shows that averaging fine-tuned weights is a meaningful base, motivating the average merged model $W_{\\text{avg}}$."}],"review_version":1}