{"id":"00a4adb9-0a38-452c-afbd-c65846730797","arxiv_id":"2412.03587","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A method that gradually freezes low-importance adapters during PEFT reduces memory usage by about 40-50% with comparable or slightly better accuracy on several benchmarks.","lead":"SAFE freezes less important adapters early during fine-tuning, cutting memory and compute while roughly preserving accuracy versus standard LoRA. The method is a practical way to fit larger model fine-tuning on smaller GPUs, with a claimed regularization side effect.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported memory reduction is ill-defined: SAFE trains all adapters during its warm-up stage, so a 40.47% peak-memory saving over a full run cannot follow from the mechanism unless 'memory usage' is measured after freezing begins.","rationale":"The paper's headline contribution is the quantitative efficiency claim: 42.85% less memory, 34.59% less compute, and 11.82% less training time while matching or improving accuracy. This claim is the reason the paper would be accepted, so it must be supported by a well-defined resource measurement. SAFE's Section 4.2 design has a warm-up stage with all adapters trainable, followed by gradual freezing; under that design, the maximum memory over a complete training run cannot drop by 40% unless the memory number refers to something other than peak memory over the entire run. The text never specifies peak versus average versus final-epoch memory, and Appendix C omits profiling details. This is more directly load-bearing than the reader's chosen weakest assumption about importance scores: even if CKA perfectly identified freezeable adapters, the efficiency headline would still be ambiguous without a clear memory accounting. The proposed profiler-based check cleanly separates the two interpretations and can be run on the authors' reported settings. I keep the CONDITIONAL verdict because the ambiguity is addressable with a measurement, but I disagree with the reader's identification of the weakest assumption: the memory-accounting ambiguity is the more fragile link in the central claim.","tokens_in":99,"tokens_out":5971,"duration_ms":121636,"concrete_test":"Re-run the BERT-large/RTE setup from Table 1 with a memory profiler (e.g., torch.cuda.max_memory_allocated) that logs peak memory per step for the entire LoRA and SAFE runs, and also record the first freezing epoch t_w and the peak memory before and after t_w. If the maximum over the full SAFE run equals LoRA's peak, or if the pre-freezing peak equals LoRA's 20.35 GB, then the reported 40.47% memory reduction is not an end-to-end peak reduction and the headline numbers must be revised or qualified. If the full-run peak is genuinely lower, the memory claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central efficiency claim (abstract and Section 5.4: 42.85% memory, 34.59% compute, 11.82% time reduction) rests on the implicit assumption that freezing adapters lowers memory for the entire fine-tuning run. But SAFE's own design (Section 4.2) starts with a warm-up stage in which all adapters are trainable, and freezing begins only at epoch t_w under a 5%-stability criterion. During warm-up, backpropagation passes through every adapter, so peak activation memory is the same as LoRA. If the reported 'memory usage' is the maximum over the full run, the mechanism cannot produce the claimed reduction unless t_w is effectively zero, which contradicts the stated warm-up criterion. If the reported number is instead the post-freezing steady-state memory, it overstates the end-to-end saving because early epochs still incur the full LoRA memory footprint. The paper never defines whether Table 1 and Table 4 report peak, average, or final-epoch memory, and Appendix C gives hyperparameters but no profiling methodology. The same ambiguity affects the TFLOPs and training-time numbers: compute is reduced only after t_w, so the averages depend entirely on how early freezing starts and how resource usage is aggregated over the schedule. This is a load-bearing ambiguity in the headline quantitative claim, independent of whether the CKA-based importance score correctly identifies which adapters can be frozen.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAFE (Selective Adapter FrEezing), a fine-tuning method that first runs a warm-up stage in which all adapters are trainable, then gradually freezes adapters whose CKA-based importance score (Eq. 3) falls below a threshold. The motivation is that adapters contribute unequally to task accuracy and activation memory, so freezing low-importance adapters early should reduce memory, computation, and training time while preserving or improving accuracy. The paper evaluates SAFE against LoRA and other PEFT/resource-efficient baselines on GLUE, SQuAD, E2E NLG, WikiText-2, and eight image-classification datasets, reporting average reductions in memory, compute, and time of 42.85%, 34.59%, and 11.82%, respectively, together with a loss-landscape/Hessian analysis arguing for a regularization effect.","tokens_in":18536,"tokens_out":5454,"duration_ms":47797,"significance":"The core idea is simple and potentially useful: permanently freezing low-importance adapters shortens the backpropagation path and can release activation memory. The empirical breadth is a strength: the evaluation spans multiple model families (BERT, RoBERTa, GPT-2, LLaMA-2, ViT, SWIN), multiple tasks, and several adapter types (LoRA, Houlsby, BitFit). The importance score is a heuristic rather than a fitted derivation, so there is no circularity in the efficiency claim. However, the headline resource-reduction numbers are not yet credible because the paper never defines the memory/compute metric, and the warm-up stage logically prevents peak-memory savings of the magnitude reported. The regularization story is plausible but post hoc. If the profiling methodology and hyperparameter reporting are fixed, the contribution could be of practical value to the PEFT community.","major_comments":[{"comment":"The headline efficiency figures are internally inconsistent. The Abstract and §5.4 report average reductions of 42.85% in memory, 34.59% in computation, and 11.82% in training time, while §1 states average reductions of 46.89% in memory and 51.73% in TFLOPs against the same LoRA baseline. The paper must reconcile these numbers and provide the per-task breakdown; as written, the reader cannot tell which claim is being made.","section":"Abstract, §1, §5.4"},{"comment":"The resource-usage metric is never defined. SAFE trains all adapters during the warm-up stage, so peak memory during warm-up is the same as LoRA's. If the reported memory values (e.g., 12.11 GB in Table 1) are measured only after freezing begins, they overstate the end-to-end saving. Please state explicitly whether memory and FLOPs are peak, average, or final-epoch values, and report a per-epoch memory/compute trace or a time-weighted average over the full run. Appendix C describes the hardware but not the profiling methodology.","section":"§4.2, Tables 1 and 4, Appendix C"},{"comment":"The method depends on two free hyperparameters, tau_T and t_f, which determine how many adapters are frozen and when. The footnote in §4.2 says these are set empirically based on extensive experiments, but Appendix C.4 does not list their values for any task or model. Without these values and a sensitivity analysis, it is unclear whether the reported savings are robust or a consequence of per-task tuning.","section":"§4.2, Appendix C.4"},{"comment":"All experimental results are single runs with no error bars or significance tests. Many SAFE-versus-LoRA differences are small (e.g., GLUE average 84.99 vs. 84.66, RTE 74.10 vs. 72.66, BERTbase SQuAD F1 87.22 vs. 86.99), so the claim of 'comparable or better' performance is not statistically supported. Please report multiple seeds or confidence intervals for at least the main tables.","section":"§5.2, Tables 1–3"},{"comment":"The importance score in Eq. (3) is computed on the current partially trained model while all adapters train jointly, but the motivation in Figure 2 injects adapters one at a time. The paper does not verify that early-epoch CKA scores are stable or that they predict the effect of freezing under joint training. A control experiment that freezes the same fraction of adapters at the same schedule at random would establish that the selection mechanism, rather than freezing per se, drives the reported performance.","section":"§3, §4.1"}],"minor_comments":[{"comment":"The formula shown is not centered kernel alignment as written; the centering of the activations or Gram matrices is omitted. Please specify the exact centered computation used.","section":"Eq. (2), §4.1"},{"comment":"The model is called 'LLaMA-27B' in the main text and Figure 6, but Appendix C.1 and Table 10 refer to 'LLaMA-2 7B'. This is confusing and must be corrected.","section":"§5.1, Figure 6, Appendix C.1, Table 10"},{"comment":"The rows 'ViTlarge + LoRA' and 'SWINlarge + LoRA' appear to be typos; presumably they should read '+ SAFE' like the other rows. Please fix the table formatting and labels.","section":"Table 5, Appendix A"},{"comment":"The statement that SAFE 'can fine-tune twice as many downstream tasks under the same FLOPs budget' is not supported by the reported compute reduction of 34.59%; please rephrase or justify with a concrete calculation.","section":"§5.4"},{"comment":"The warm-up criterion ('importance score changes by less than 5% for consecutive epochs') is not defined precisely: which aggregation of importance scores is used, and what were the actual t_w values? Please report these values for each experiment.","section":"§4.2, Appendix C.4"}],"recommendation":"major_revision","confidential_remarks":"The central issue is definitional rather than purely methodological: if the authors supply memory/compute traces over the full training run and list the tau_T and t_f values used, the revision should be evaluable. I would not oppose resubmission of a revised manuscript, provided the abstract and §1 numbers are reconciled."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is genuinely new and practical: instead of dropping adapters randomly or pruning their weights, SAFE freezes entire low-importance adapters during training so their activation memory can be freed. The CKA-based importance score is a reasonable heuristic, the cubic threshold schedule is borrowed cleanly from AdaLoRA, and the evaluation across BERT, RoBERTa, GPT-2, LLaMA, ViT, and SWIN is broader than most PEFT papers bother with. The regularization story—flatter loss landscape and smaller Hessian eigenvalues—is a nice extra, though it is post hoc and not necessary for the main contribution.\n\nThe soft spot is the headline efficiency claim. The abstract says SAFE cuts memory by 42.85%, but the paper never says whether that is peak memory, average memory, or steady-state memory after freezing. This matters because the warm-up stage trains all adapters with full backpropagation, so peak memory during warm-up is identical to LoRA. If the 40% reduction is an average over the run, it depends heavily on how early freezing starts and how much of the run is in the frozen regime. The paper does not report the warm-up epoch t_w or the freezing horizon t_f for any experiment, so the number is not reproducible. The stress-test concern is valid: as written, the 40.47% reduction in Table 1 cannot follow from the mechanism unless t_w is effectively zero, which contradicts the stated 5% stability criterion.\n\nThere are also smaller issues that add up: no error bars or repeated seeds; the key thresholds τ_T and t_f are tuned per task with no sensitivity analysis; the code is not provided; and the numbers are inconsistent between the abstract (42.85%), the introduction (46.89% memory and 51.73% TFLOPs), and Table 4 (42.85%, 34.59%, 11.82%). The baseline citation \"Zhang et al.\" is ambiguous and does not resolve to a listed reference, and the paper alternates between LLaMA-27B and LLaMA-2 7B.\n\nThat said, the method itself is plausible and the direction is useful. This deserves a serious referee, not a desk rejection. A conditional acceptance is appropriate if the authors clarify the memory metric, report the actual warm-up epochs and freezing schedules, add error bars, and correct the inconsistencies. Without those, the central quantitative claim is not trustworthy.","headline":"SAFE is a sensible new trick for cutting activation memory in adapter fine-tuning, but the paper's headline efficiency numbers are under-specified to the point of being misleading, and the current draft needs a serious revision before the results can be trusted.","tokens_in":19087,"tokens_out":2554,"would_cite":false,"duration_ms":22721,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Selectively freezing low-impact adapters during fine-tuning cuts memory usage by about 43 percent while matching or improving task accuracy.","keywords":["parameter-efficient fine-tuning","adapter freezing","memory-efficient training","CKA similarity","LoRA","loss landscape regularization","large language models"],"falsifier":"Run SAFE on a GLUE task but, at the freezing stage, freeze the same number of adapters selected uniformly at random rather than by the CKA importance score; if random freezing matches SAFE's final accuracy, the importance score itself is not what preserves performance.","tokens_in":8,"feed_emoji":"🧊","tokens_out":5877,"duration_ms":109403,"temperature":0.7,"pith_summary":"The paper claims that during adapter-based fine-tuning of large language models, not all adapters deserve equal training. It proposes SAFE, which measures how much each adapter's layer representation has already diverged from the pretrained layer using CKA similarity, then gradually freezes adapters whose importance score stays low. The claim is that this freezing saves substantial memory and compute—42.85% less memory, 34.59% less compute, and 11.82% less training time on average—while matching or improving task accuracy. The paper further argues that early freezing acts as a regularization that flattens the loss landscape, and it presents Hessian-spectrum and loss-landscape evidence for that effect.","feed_headline":"Freezing unimportant adapters cuts fine-tuning memory by 43%","feed_subtitle":"SAFE monitors layer-representation similarity, then freezes low-value adapters early to save compute without losing accuracy.","key_machinery":"The central mechanism is the per-adapter importance score defined as $\\text{Imp}(\\text{Adapter}_i) = 1 - \\operatorname{CKA}_i(X_i, Y_i)$, where $X_i$ and $Y_i$ are the activations of the adapted and original layer $i$, and CKA is centered kernel alignment. A high CKA value means the adapter has barely changed the layer's representation, so the adapter is judged unimportant and becomes a candidate for freezing. Freezing is applied gradually through a cubic threshold schedule that rises from zero at the end of the warm-up epoch to a target threshold at the final freezing epoch. Because frozen adapters only participate in the forward pass, their activation memory can be de-allocated, shortening the backpropagation length and cutting memory and compute.","core_discovery":"The central discovery is that adapters attached to different transformer layers contribute unequally to downstream adaptation, and that this inequality can be exploited during training: adapters whose representations already match the target features can be frozen early without hurting accuracy. SAFE operationalises this by computing an importance score per adapter as one minus the CKA similarity between the adapted layer's activations and the original frozen layer's activations, monitoring the scores during a warm-up phase, and then freezing low-scoring adapters on a cubic threshold schedule. The paper reports that this yields accuracy equal to or better than the LoRA baseline across NLU, QA, and NLG benchmarks, with memory savings up to 79.92% on RoBERTa-large on SQuAD. It also claims the frozen-parameter constraints induce a regularization effect, evidenced by flatter loss landscapes and smaller Hessian eigenvalues.","pith_inferences":["The paper's Limitations section reports that SAFE was not evaluated in combination with quantization, gradient checkpointing, micro-batching, or weight sharding; the stated complementary benefits are a proposal, not a measured result.","A natural stress test is whether randomly freezing the same number of adapters at the same schedule matches SAFE's accuracy; if it does, the CKA importance score is not the driver of the gains.","The regularization story suggests SAFE can be viewed as a dynamic sparsity schedule over adapter parameters; comparing it against a fixed mask chosen after the warm-up phase would isolate the benefit of gradual freezing.","Because the importance score is computed on the training distribution, computing it on held-out validation batches could reduce overfitting of the freezing decisions and would be a simple extension to test."],"forward_implications":["On average across NLU, QA, and NLG benchmarks, SAFE reduces memory by 42.85%, computation by 34.59%, and training time by 11.82% relative to LoRA while matching or beating accuracy.","On SQuAD with RoBERTa-large, SAFE freezes 91.67% of adapters, cutting memory by 79.92% and compute by 88.41% while improving F1 from 93.39 to 94.13.","SAFE transfers across adapter designs: applying it to Houlsby adapters and BitFit cuts memory by about 25% with comparable GLUE scores.","SAFE works on decoder-only models and large language models: on a 7B LLaMA model it reduces memory by 48.37% without perplexity degradation.","Because frozen adapters run forward-only, SAFE can be combined with other memory-saving techniques such as gradient checkpointing and quantization."],"supporting_citations":[{"why":"Provides the LoRA baseline that SAFE is compared against in all main experiments.","marker":"(Hu et al., 2022)"},{"why":"Supplies the CKA metric used to define the per-adapter importance score.","marker":"(Kornblith et al., 2019)"},{"why":"AdapterDrop is the key prior method for reducing adapter compute, used as a comparison for accuracy and memory trade-offs.","marker":"(Rücklé et al., 2021)"},{"why":"Formalizes the optimization objective with a mask matrix, which the paper uses to explain SAFE's regularization effect.","marker":"(Fu et al., 2023)"},{"why":"AdaLoRA is a baseline and the source of the cubic threshold schedule that SAFE adopts for gradual freezing.","marker":"(Zhang et al., 2022)"},{"why":"Prior work that uses CKA-style representation similarity for layer freezing, cited as the basis for the motivational analysis.","marker":"(Li et al., 2022a)"}],"fun_headline_variants":["Freeze adapters that matter least to cut memory 43% during fine-tuning","Selective adapter freezing cuts fine-tuning memory 43% without loss","SAFE selectively freezes low-value adapters, saving memory and compute","Not all adapters matter: freeze them early for efficient fine-tuning","Memory-efficient fine-tuning: freeze unimportant adapters to save 43%"],"cache_read_input_tokens":21248,"weakest_assumption_plain":"The load-bearing premise is that the importance score measured on the partially trained model—one minus the CKA similarity between adapted and original layer activations—predicts which adapters can be permanently frozen without hurting final accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Freeze adapters that matter least to cut memory 43% during fine-tuning","Selective adapter freezing cuts fine-tuning memory 43% without loss","SAFE selectively freezes low-value adapters, saving memory and compute","Not all adapters matter: freeze them early for efficient fine-tuning","Memory-efficient fine-tuning: freeze unimportant adapters to save 43%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00071,"raw_usage":{"total_tokens":3171,"prompt_tokens":897,"completion_tokens":2274,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":2178}},"tokens_in":513,"tokens_out":2274,"duration_ms":13082,"temperature":1.0,"reasoning_tokens":2178,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:21:38.000380+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SAFE on a GLUE task but, at the freezing stage, freeze the same number of adapters selected uniformly at random rather than by the CKA importance score; if random freezing matches SAFE's final accuracy, the importance score itself is not what preserves performance.","supporting_citations":[],"review_version":1}