{"id":"713ba9ec-5bb9-44fc-91f5-312357cd1b76","arxiv_id":"2611.17191","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding a Rényi-2 effective-rank penalty to LoRA training increases the effective rank of updates and reduces forgetting on pre-training perplexity while maintaining or improving task fit.","lead":"This paper shows that LoRA adapters often use far fewer effective directions than their assigned rank, and adds a simple training penalty that pushes the adapter to use more of its rank, which reduces forgetting on pre-training tasks while keeping learning performance. If true, this gives practitioners a cheap way to improve the learning-forgetting trade-off in fine-tuning large language models without changing the adapter architecture.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"No significant objection identified","rationale":"The reader's weakest_assumption is that the observed improvements might be caused by side effects of the penalty (norm regularization, altered gradient scales) rather than by the increase in effective rank. The paper itself acknowledges this in Section 4.1 (sentence b124): 'these experiments do not separate effective-rank maximization from norm control'. This is a genuine ambiguity, but it is a limitation of the causal interpretation, not a threat to the empirical claim. The paper's central claim is that the method improves the learning–forgetting frontier, not that it does so specifically via effective-rank maximization. The empirical evidence for the method's effectiveness is strong: multiple models, ranks, seeds, and tasks, with consistent results. The authors also provide a norm-bounded route (NB-LoRA) that achieves similar results, which further supports the robustness of the approach. The concrete test I propose would help resolve the causal ambiguity, but the paper's main claim does not depend on that resolution. Therefore, the verdict should remain ACCEPT (UNCHANGED). I agree with the reader's identification of the weakest assumption.","tokens_in":43839,"tokens_out":652,"duration_ms":48159,"concrete_test":"Re-run the mixed-domain experiment on Qwen2.5-0.5B at r=896 with λ=0.01, but replace the Rényi-2 penalty with a penalty that directly maximizes the Frobenius norm ∥ΔW∥_F^2 while keeping the operator norm fixed (as in NB-LoRA but without the Stiefel–Cayley parameterization). If the resulting effective rank and retention metrics match those of the Rényi-2 penalty, then the benefit is attributable to norm control rather than spectral uniformity. If they diverge, the spectral uniformity objective is the active ingredient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that adding a Rényi-2 effective-rank penalty to the language-modeling loss reduces forgetting on pretraining proxies while maintaining competitive target-task fit—is supported by extensive, well-documented experiments across multiple models, ranks, tasks, and seeds. The authors transparently acknowledge the key ambiguity: the norm-bounded route also raises effective rank, and the experiments do not fully separate effective-rank maximization from norm control (Section 4.1, sentence b124). However, this is a limitation of the causal interpretation, not a flaw in the empirical finding. The paper does not claim to have proven a causal mechanism; it claims to have demonstrated a method that improves the learning–forgetting frontier, and that claim is backed by the data. The reader's weakest_assumption correctly identifies this ambiguity, but the paper's own discussion already covers it. No additional load-bearing concern beyond what the authors themselves flag is apparent. The method is simple, the code is released, the experiments are reproducible, and the results are consistent across settings. The paper's limitations section (b235) is honest and complete. Therefore, no significant objection is identified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper identifies that LoRA adapters often underutilize their configured rank, leading to a learning-forgetting trade-off. The authors propose a Rényi-2 effective-rank penalty that is SVD-free and parameterization-agnostic, added to the language-modeling loss to maximize adapter capacity utilization. They demonstrate across mixed-domain SFT, task-specific SFT, rank sweeps, and scheduling ablations that this penalty can reduce forgetting on pretraining proxies (Wiki, StarCoder perplexity) while maintaining competitive target-task fit. The method is compared against standard LoRA, dense fine-tuning, and several LoRA variants (PiSSA, LoRA-XS, VeRA, SVF, TinyLoRA) on Qwen2.5, Llama-3.2, and other models. The paper also connects capacity maximization to norm-bounded updates and provides spectral diagnostics.","tokens_in":48633,"tokens_out":875,"duration_ms":40441,"significance":"If the result holds, it provides a simple, practical, and theoretically motivated method to improve the learning-forgetting frontier in LoRA fine-tuning. The method is well-motivated by the observation of capacity underutilization, which is empirically demonstrated. The paper includes extensive experiments with multiple seeds, uncertainty estimates, and across multiple models and tasks, lending credibility to the findings. The code is released, enhancing reproducibility. The paper is transparent about limitations, including the ambiguity between effective-rank maximization and norm control. The work is significant for the parameter-efficient fine-tuning community as it offers a new axis for improvement orthogonal to existing methods.","major_comments":[],"minor_comments":[{"comment":"The definition of 'Train PPL' should be clarified earlier in the main text. It is described as 'training-mixture perplexity' but is actually an exponential moving average of the training loss, not held-out validation perplexity. This distinction is important for interpreting the results and should be stated in the main text, not just in Appendix F.","section":"Section 4.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The claim 'Mean training perplexity improves or increases by at most 0.02 in all but one model–rank comparison' is accurate, but the exception (Qwen2.5-3B-Instruct at r=16, where Train PPL increases by 0.313) should be explicitly mentioned for completeness.","section":"Abstract and Section 4.4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The dense FT point is a single run. Consider using different markers for single-run vs multi-run means to improve clarity, as the current legend does not distinguish them visually.","section":"Figure 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The dense baseline hyperparameter selection uses training loss rather than held-out validation. While this does not affect the main LoRA-to-LoRA comparisons, it may not yield the best possible dense baseline. This should be noted as a caveat when interpreting dense comparisons.","section":"Appendix F.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The weighted unaligned mass for penalized LoRA is slightly higher than for standard LoRA (0.972-0.973 vs 0.960), yet the penalized adapters forget less. The paper notes this but does not explain the apparent contradiction. A brief discussion of why this might be the case would strengthen the analysis.","section":"Section 5, Table 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper uses 'StarCoder PPL' and 'Wiki PPL' as forgetting probes from OLMo-Mix-1124. It would be helpful to explicitly state that these are not the actual pretraining corpora for the models used (Qwen, Llama) but are used as general pretraining distribution proxies.","section":"Section 4","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Hey,\n\nYou should know about this paper: it introduces a Rényi-2 effective-rank penalty for LoRA training and shows, across a lot of experiments, that it reduces forgetting on pretraining proxies while keeping target-task fit competitive. The core idea is simple—add an SVD-free term that encourages the adapter to use its allocated rank more evenly—and the paper does a thorough job of validating it.\n\nWhat's actually new: the observation that standard LoRA underutilizes its configured rank (effective rank far below nominal rank) is not new per se, but using effective rank as a trainable objective, specifically via the Rényi-2 surrogate, is. The paper provides a clean derivation, connects it to norm-bounded updates, and ships code. The experiments are extensive: multiple models (Qwen2.5, Llama-3.2, OLMo-2, Pythia), multiple ranks, multiple seeds, mixed-domain and single-task settings, and comparisons to several LoRA variants (PiSSA, LoRA-XS, VeRA, SVF, TinyLoRA). The results are consistent: at λ=0.01, Wiki perplexity drops at every rank ≥16 across three models, with minimal training perplexity cost. That's a real empirical finding.\n\nThe soft spots are real but not fatal. The main one is that the paper cannot separate effective-rank maximization from norm control—the norm-bounded route also raises effective rank, and the authors say so explicitly (Section 4.1). So the causal claim is ambiguous, but the paper doesn't overclaim; it says the method improves the frontier, not that it proves a mechanism. The dense baseline hyperparameter selection uses training loss rather than held-out validation, which is a minor methodological concern. The PiSSA exclusion (one run with apparent data replay) is handled transparently. The scheduling experiments are on a short budget and the mixed-domain results use early-start penalty, so the Train PPL cost at λ=0.1 might be reducible—the authors note this. The limitations section is honest.\n\nWho this is for: anyone working on LoRA-based fine-tuning, especially in continual or multi-domain settings. The method is easy to implement and the paper gives clear guidance on hyperparameters. It deserves a serious referee; the experiments are reproducible and the claims are supported. I'd cite it if I were working on PEFT for LLMs.\n\nRecommendation: accept with minor revisions. The main ask would be to add a small experiment or discussion that more directly tests the causal role of effective rank (e.g., a synthetic setting where rank is controlled independently of norm), but this is not a dealbreaker.","headline":"A solid, well-executed paper that introduces a simple, principled regularizer for LoRA and backs it with extensive experiments; the main weakness is that the causal mechanism (effective rank vs. norm control) is not fully isolated, but the authors acknowledge this and the empirical finding stands.","tokens_in":44636,"tokens_out":672,"duration_ms":10495},"pith_extraction":null,"created_at":"2026-11-13T09:24:54.873435+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":null,"supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}