{"id":"13951f5a-50ef-4ed7-9323-f2fec22d8375","arxiv_id":"2501.15377","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning a pretrained vision model with only 2-6% of its LoRA adapters active preserves zero-shot and out-of-distribution performance while matching full-rank LoRA accuracy.","lead":"A research team adds a learned on/off switch to LoRA adapters, so only a few selected blocks change during fine-tuning. On CLIP and DINO models, this keeps most pretrained knowledge intact while matching the accuracy of full LoRA on new tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline CLIP result lacks a random-selection control: without one, the learned gate's role in preserving zero-shot knowledge is unverified, and Eq. (3)'s threshold tau is never specified.","rationale":"The reader's CONDITIONAL verdict is reasonable. I agree the unspecified tau in Eq. (3) is a genuine reproducibility blocker. My stronger concern is that the CLIP headline result is not isolated as a selection effect. Table 1 is the only ablation against random block choice, and it is on DINO/CIFAR-100 at rank 16; the paper presents no such control for the rank-128 CLIP experiments that drive the abstract and Figure 1. Without it, the 6.25%-active-blocks result may reflect only that a small LoRA budget retains more zero-shot knowledge than a large one—a capacity effect already visible in the LoRA rank sweeps of Tables 10, 12, and 14. A full LoRA at rank 8 has the same parameter count as 6.25% of rank-128 blocks and yields a similar ID/retention trade-off, so the learned gate is not shown to add value beyond parameter reduction. The conclusion's explicit statement that the method 'does not directly address catastrophic forgetting' should be weighed against the title and abstract; it does not falsify the measurements, but it does mean the headline overclaims the mechanism. The concrete test—random subsets and parameter-matched LoRA on CLIP—would settle whether the selective mechanism is essential. If random subsets match, the central claim reduces to 'sparse LoRA retains knowledge,' not 'selective LoRA.' If random subsets are worse, the method's claim is substantiated. Given the missing control and missing tau, CONDITIONAL remains the right verdict; I would not reject, because the empirical comparisons are internally consistent and the method is cheap to test.","tokens_in":22768,"tokens_out":11335,"duration_ms":100729,"concrete_test":"Re-run the CLIP ImageNet fine-tuning at rank 128 with lambda=1, using the authors' tau (which must be reported), and compare against (a) random 6.25% block subsets at rank 128 over at least 5 seeds, and (b) full LoRA at rank 8. Report IN-1K, OOD mean, zero-shot classification, and zero-shot retrieval under the paper's evaluation protocol. If random subsets match the method's headline numbers within variance, the learned selection is not load-bearing; if random subsets are substantially worse and rank-8 LoRA has lower ID accuracy, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the learned indicator function selects blocks that matter, and that this selection—not merely the reduced number of adapted parameters—produces the reported retention gains. The paper's only control for this is Table 1, on DINO with CIFAR-100 at rank 16, where learned selection beats random subsets by roughly 2% target accuracy. For the headline CLIP claim (rank 128, 6.25% active blocks), no random-subset or selection-ablated baseline is reported. Consequently, the central result could be reproduced by any 6.25% subset, making the \"selective\" mechanism non-essential. A parameter-matched full LoRA (rank 8) has essentially the same parameter count as 6.25% of rank-128 blocks and shows a similar qualitative ID/retention trade-off, suggesting a capacity effect rather than a gating effect. Section 5 also concedes the method \"does not directly address catastrophic forgetting,\" which undercuts the title and the abstract's \"without Catastrophic Forgetting\" framing. Independently, the threshold tau in Eq. (3) is never given in the main text, the appendix, or the supplementary hyperparameters, so the reported 6.25% active-block count cannot be reproduced or audited.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a PEFT method that extends Task Adaptive Parameter Sharing (TAPS) by adding an indicator function I_tau(s_i) that selectively activates LoRA blocks, with an L1 penalty on the block scores s_i. The central claim is that this gating mechanism allows fine-tuned models to match full-LoRA in-distribution accuracy while using only a small fraction of active blocks, thereby retaining zero-shot and OOD knowledge and reducing inference cost. Experiments are reported on DINO ViT-S/B-16 fine-tuned on six target datasets and on CLIP fine-tuned on ImageNet-1K, evaluated on OOD, zero-shot classification, and zero-shot retrieval benchmarks. The method is also applied to DoRA. The headline result is that CLIP at rank 128 with 6.25% active blocks reaches 81.84% ImageNet accuracy versus 81.77% for full LoRA, while zero-shot classification improves from 51.44 to 61.48 and retrieval retention is substantially better than full LoRA.","tokens_in":23002,"tokens_out":3630,"duration_ms":32724,"significance":"If the selective-activation mechanism is genuinely responsible for the observed preservation of pre-trained knowledge, the method is a simple, practical extension of LoRA-style PEFT with broad applicability. The paper’s strengths are its extensive empirical coverage: multiple backbones (DINO ViT-S/B, CLIP), multiple PEFT variants (LoRA and DoRA), many transfer datasets, several ranks and regularization strengths, and separate evaluations of OOD robustness, zero-shot classification, and zero-shot retrieval. The inference-FLOPs analysis is a useful practical contribution. However, the central scientific claim—that learned selection, rather than merely reducing the number of adapted parameters, yields the retention gains—is not fully supported for the main CLIP experiment, and the reproducibility of the reported active-block percentages is compromised by the unspecified threshold tau. The paper also overstates its contribution in the title and abstract relative to the explicit limitation stated in Section 5.","major_comments":[{"comment":"This is a load-bearing issue: the claimed efficiency gains and the claimed trade-off curves depend on where the threshold sits.","section":"Sec. 3.2, Eq. (3)"},{"comment":"The central claim 'selective activation preserves knowledge' is not distinguishable from 'fewer adapted parameters preserve knowledge' without this control.","section":"Sec. 4.2, Fig. 6 and Table 10"},{"comment":"The current framing overclaims the contribution relative to what is measured in the experiments.","section":"Sec. 5, Conclusion"},{"comment":"This concern is particularly relevant to the DINO random-selection comparison in Table 1, where the claimed advantage is about 2%.","section":"Tables 10-15"}],"minor_comments":[{"comment":"The Linear row in Table 4 lists CIFAR-100=72.07, IN-100=87.20, Mean=88.32, but the mean of 72.07 and 87.20 is 79.64. This appears to be a copy-paste error from Table 3 (where the mean 88.45 is consistent with the CIFAR-10 numbers). Please correct.","section":"Supplementary Table 4"},{"comment":"Given that AdaLoRA is cited as an adaptive rank-allocation method, a direct empirical comparison with AdaLoRA (or a brief explanation of why it is not compared) would help position the contribution. Currently the method is compared only with full LoRA/DoRA and with full fine-tuning/linear probing.","section":"Sec. 2, Related Works"},{"comment":"The FLOPs comparison is presented for the unmerged setting where inactive blocks are skipped. It would be helpful to state explicitly whether the reported speedups assume that active LoRA blocks are not merged into the base weights, since merging would eliminate the inference benefit; the current text implies but does not state this.","section":"Sec. 4.3, Fig. 9"},{"comment":"The figure caption says 'across 8 runs' for CLIP, but the text describes 7 runs at one lambda plus 6 runs at other lambdas. Please reconcile the count and clarify whether the shown activations are pooled over all runs or averaged per configuration.","section":"Sec. 4.4, Fig. 10"}],"recommendation":"major_revision","confidential_remarks":"The paper is an incremental extension of TAPS to LoRA-style PEFT, but the empirical scope is broad and the method is simple to implement. The most serious issues are the unspecified threshold tau, the missing random-selection control for the CLIP headline experiment, and the overstatement of the catastrophic-forgetting claim relative to the conclusion's caveat. These are addressable within the manuscript's scope, so I do not recommend rejection, but the paper needs substantive revision and additional experiments before it can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read arXiv:2501.15377. My take: this is a legitimate extension of TAPS-style gating to LoRA blocks for vision models, and the paper does something real—shows that plain LoRA/DoRA at high ranks on CLIP loses zero-shot classification and retrieval badly, while the gated version keeps most of that knowledge and matches ID accuracy with far fewer active blocks. Tables 10-15 tell a consistent story across ranks, and the DINO experiments add a second model family. The random-selection control in Table 1 (DINO, CIFAR-100, rank 16) is the right kind of control and shows learned selection beats random by about 2% target accuracy.\n\nThe soft spots are real but fixable. First, the headline CLIP result (rank 128, 6.25% active blocks) has no random-selection baseline. Without it, the selective mechanism is not shown to be essential: a parameter-matched rank-8 LoRA has the same parameter count and shows a similar retention-versus-ID trade-off. The DINO control suggests gating helps, but it is one dataset at one rank; the flagship claim needs the same control on CLIP. Second, the threshold tau in Eq. (3) is never specified—not in the main text, appendix, or hyperparameter section. Every active-block percentage depends on it, so the headline efficiency numbers are not reproducible as reported. That is a straightforward fix: report tau, its initialization, and how it interacts with lambda. Third, the conclusion itself concedes that the method does not directly address catastrophic forgetting, which undercuts the title and abstract. That framing should be corrected. Fourth, results are single-run with no error bars, and Table 4 has an internal inconsistency in the Linear row (the mean does not match its components). Lambda is also selected per dataset, which slightly weakens the generality claim.\n\nWho is it for: anyone working on PEFT for CLIP/DINO who wants a simple gating mechanism and a broad empirical survey. It deserves a serious referee. I would send it to review expecting the authors to add a CLIP random-selection control, report tau, and tone down the forgetting language. As submitted, conditional accept.","headline":"A broad, mostly consistent empirical study of gated LoRA for CLIP/DINO, but the headline CLIP claim needs a random-selection control and the unreported threshold tau makes the efficiency numbers unverifiable.","tokens_in":23560,"tokens_out":4240,"would_cite":false,"duration_ms":34365,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning with a small subset of LoRA blocks active can match full-LoRA target accuracy while preserving zero-shot and out-of-distribution knowledge.","keywords":["catastrophic forgetting","parameter-efficient fine-tuning","low-rank adaptation","indicator function","out-of-distribution generalization","zero-shot retention","selective block activation","CLIP"],"falsifier":"Repeat the rank-128 CLIP fine-tuning with the paper's equations but sweep the threshold $\\tau$ across a wide range, say 0, 0.01, 0.1, and 0.5, while keeping $\\lambda=1$; the claim fails if any run either leaves nearly all blocks active, drops ImageNet accuracy below LoRA's 81.77%, or loses the zero-shot retention advantage, since that would show the result depends on the unspecified threshold rather than on the gating mechanism itself.","tokens_in":22573,"feed_emoji":"🎯","tokens_out":9312,"duration_ms":70145,"temperature":0.7,"pith_summary":"The paper claims that fine-tuning pretrained vision and vision-language models does not require updating all low-rank blocks: a learned gate can switch most of them off. It adds a per-block score $s_\\ell$ and a straight-through indicator $I_\\tau(s_\\ell)$ in front of each LoRA residual, with an $\\ell^1$ penalty on the scores to push most gates to zero. On CLIP fine-tuned on ImageNet-1K at rank 128, activating only 6.25% of blocks matches LoRA's 81.77% accuracy at 81.84%, while zero-shot classification improves from 51.44 to 61.48 and zero-shot retrieval loss stays under 5.73% instead of about 28% for full fine-tuning. The same gating works when applied to DoRA and to DINO-ViT backbones, keeping target accuracy comparable while retaining far more of the pretrained knowledge. A reader should care because it suggests a cheap, model-agnostic way to adapt foundation models without erasing what they already know.","feed_headline":"6% of LoRA blocks is enough to fine-tune CLIP without forgetting","feed_subtitle":"Selective gating matches full LoRA on ImageNet and caps zero-shot retrieval loss near 6%, not 28%.","key_machinery":"The central object is a per-block gate built from a learnable scalar score $s_\\ell$ and a straight-through estimator (STE) indicator $I_\\tau(s_\\ell)$ that multiplies each low-rank residual $A_\\ell B_\\ell$, trained with an $\\ell^1$ sparsity penalty on the scores. The gate makes the set of active blocks a trainable, sparse choice: the scores decide which low-rank blocks stay on for the target task, and the $\\ell^1$ term is what produces the tiny active-block percentages (1.39–6.25% on CLIP, 2.7–44% on DINO depending on $\\lambda$, rank, and dataset). The straight-through estimator lets gradients flow through the binary decision so the selection itself is learned, not set by hand.","core_discovery":"The central discovery is that the low-rank updates learned during fine-tuning are highly redundant: a small, task-dependent subset of blocks carries almost all of the adaptation signal, and the rest can be left off without losing target-task accuracy while preserving the pretrained feature space. Formally, the paper updates each block as $W_\\ell = W_{0,\\ell} + I_\\tau(s_\\ell)A_\\ell B_\\ell$, where $I_\\tau(s_\\ell)=1$ if $s_\\ell\\ge\\tau$ and $0$ otherwise, and regularizes the gate scores with $\\lambda\\sum_\\ell |s_\\ell|$. With this update, LoRA at rank 128 on CLIP reaches 81.84% ImageNet-1K accuracy (versus 81.77% for full LoRA) using 6.25% of the blocks; zero-shot classification average rises to 61.48 versus 51.44 for LoRA, zero-shot retrieval loss falls to at most 5.73% versus about 28% for FLYP, and unmerged inference becomes up to 2.9x faster for LoRA and 5x faster for DoRA at rank 256. On DINO-ViT, activating as few as 2.7% of blocks keeps target accuracy comparable to LoRA while source-domain accuracy on ImageNet-100 is retained substantially better than under full fine-tuning.","pith_inferences":["Inference: the layer-activation maps, which concentrate in feedforward MLP blocks and in later CLIP vision layers, suggest that a fixed or pretrained subset of blocks could be enough, which would remove the need to learn gate scores at all.","Inference: the reported preservation suggests low-rank updates are redundant across transformer residuals generally, implying selective LoRA could slot into continual-learning pipelines without rehearsal buffers.","Inference: because $\\tau$ is unspecified, a normalized gate such as a hard sigmoid over scores divided by the maximum score would make the method reproducible and remove dependence on the score scale.","Inference: if sparsity is what preserves knowledge, then the method's benefit should be testable against an ablation that selects the same number of blocks by gradient magnitude or by the paper's block-activation heatmaps rather than training scores."],"forward_implications":["Fine-tuning a foundation model on a new dataset can be done with only a small fraction of its low-rank blocks active, cutting inference FLOPs and memory proportionally; the paper measures up to 2.9x (LoRA) and 5x (DoRA) faster unmerged inference at rank 256.","Because the gate applies to any LoRA-variant method (the paper demonstrates DoRA), it is a plug-in that can reduce catastrophic forgetting across PEFT techniques.","Higher ranks, which normally cause stronger forgetting, become usable: at rank 256 the gated LoRA reaches 82.31% ImageNet accuracy with 5.56% of blocks active, while plain LoRA's out-of-distribution mean drops to 53.81.","The retained zero-shot abilities mean a fine-tuned CLIP can still serve as a general image-text retriever and zero-shot classifier, not just as a specialist on the target classes."],"supporting_citations":[{"why":"Supplies the task-adaptive parameter-sharing formulation and learned indicator that the paper extends from full weight matrices to low-rank blocks.","marker":"[53]"},{"why":"Defines low-rank adaptation (LoRA), the residual update $W=W_0+AB$ that the indicator function selectively activates.","marker":"[25]"},{"why":"Provides the self-supervised ViT fine-tuning setup, the target datasets, and the K-NN source-domain evaluation used to measure catastrophic forgetting.","marker":"[3]"},{"why":"Supplies the robust-finetuning framing and the zero-shot classification plus retrieval evaluation protocol for CLIP, including the contrastive loss setup.","marker":"[19]"},{"why":"The pretrained CLIP model whose zero-shot and retrieval capabilities the paper aims to preserve during fine-tuning.","marker":"[42]"},{"why":"Defines DoRA, demonstrating that the indicator-gating extension generalizes beyond LoRA.","marker":"[61]"}],"fun_headline_variants":["Fine-tune CLIP with 6% LoRA blocks, zero forgetting","Selective LoRA: 6% active blocks, full accuracy","6% LoRA blocks match full fine-tuning on CLIP","LoRA gating saves memory and preserves zero-shot","Smallest LoRA subset keeps CLIP's knowledge intact"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the sparsity penalty and an unspecified threshold $\\tau$ together yield the reported small active-block counts; if $\\tau$ is set differently or depends on the scale of the learned scores, the headline efficiency and forgetting numbers would not reproduce.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tune CLIP with 6% LoRA blocks, zero forgetting","Selective LoRA: 6% active blocks, full accuracy","6% LoRA blocks match full fine-tuning on CLIP","LoRA gating saves memory and preserves zero-shot","Smallest LoRA subset keeps CLIP's knowledge intact"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000577,"raw_usage":{"total_tokens":2752,"prompt_tokens":1004,"completion_tokens":1748,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":1662}},"tokens_in":620,"tokens_out":1748,"duration_ms":11656,"temperature":1.0,"reasoning_tokens":1662,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:21:06.541403+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the rank-128 CLIP fine-tuning with the paper's equations but sweep the threshold $\\tau$ across a wide range, say 0, 0.01, 0.1, and 0.5, while keeping $\\lambda=1$; the claim fails if any run either leaves nearly all blocks active, drops ImageNet accuracy below LoRA's 81.77%, or loses the zero-shot retention advantage, since that would show the result depends on the unspecified threshold rather than on the gating mechanism itself.","supporting_citations":[{"cited_title":"Fowlkes, Rahul Bhotika, and Ste- fan 0 Soatto","cited_arxiv_id":null,"evidence_quote":"Supplies the task-adaptive parameter-sharing formulation and learned indicator that the paper extends from full weight matrices to low-rank blocks."},{"cited_title":"Parameter efficient fine-tuning of self-supervised vits without catastrophic forgetting","cited_arxiv_id":null,"evidence_quote":"Provides the self-supervised ViT fine-tuning setup, the target datasets, and the K-NN source-domain evaluation used to measure catastrophic forgetting."},{"cited_title":"Anchor- based robust finetuning of vision-language models","cited_arxiv_id":null,"evidence_quote":"Supplies the robust-finetuning framing and the zero-shot classification plus retrieval evaluation protocol for CLIP, including the contrastive loss setup."}],"review_version":1}