{"id":"249d1799-65ef-400c-b14f-f1b5e0369b86","arxiv_id":"2411.16775","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On SuperNI and TÜLU benchmarks, LoRA and adapters match full fine-tuning on average only with large ranks and optimal learning rates, while underperforming on reasoning, coding, and long-form tasks.","lead":"This study compares parameter-efficient fine-tuning methods, such as LoRA and adapters, against full fine-tuning for instruction-tuned language models. It finds that LoRA and adapters can nearly match full fine-tuning when hyperparameters are carefully chosen, but they struggle with reasoning, coding, and long-form generation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hyperparameters for LLAMA-2/TÜLU are transferred unchanged from T5/SuperNI without retuning, so Table 3's reasoning/coding gap may be a tuning artifact rather than an intrinsic PEFT limitation.","rationale":"The reader's weakest assumption and my concern coincide: the T5/SuperNI-derived hyperparameters are carried over to LLAMA-2/TÜLU without retuning. This is load-bearing because the paper's headline claim is explicitly conditional on 'ideal training settings,' and the TÜLU results are used to show where PEFT fails even with those settings. If the ideal settings differ for LLAMA-2, the failure could be a tuning artifact. The paper itself flags this in Appendix A. I considered the unsupported long-form-generation claim and the missing error bars; both are real weaknesses but the hyperparameter-transfer issue is the one that, if resolved, could change a headline conclusion. It does not change the reader's CONDITIONAL verdict, which should stand until the retuning check is run.","tokens_in":11543,"tokens_out":6692,"duration_ms":58438,"concrete_test":"Re-run the LLAMA-2 7B / TÜLU experiments with a small grid for LoRA (r ∈ {8,64,128,256,512}; lr ∈ {1e-5,5e-5,1e-4,2e-4}) and adapter (s ∈ {8,64,128,256,512}; same lr grid), keeping full finetuning exactly as in the paper, and report best GSM8K/BBH/Codex-Eval. If no configuration brings LoRA within ~2 points of finetuning on Codex-Eval and GSM8K, Table 3's conclusion stands; if a configuration does, the 'intrinsic shortfall' claim should be retracted or reworded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. 2.2 says Setup 2 uses 'the selected PEFTs and their best hyperparameters found from the first setup' — lr=1e-4, r=512, s=512 — applied to LLAMA-2 7B on TÜLU. No hyperparameter search is reported for this second model/dataset; Appendix A concedes the optimal hyperparameters 'might not reflect latest model architecture's performance on latest instruction tuning datasets.' This matters because the abstract's qualified conclusion rests on Finding 7: LoRA/adapter 'fall short in complex reasoning, coding and long-form generation' (Table 3: FT 43.4 vs LoRA 38.78 vs adapter 35.26). Table 4 shows LoRA at r=512 is sharply peaked: on T5-3B, RougeL is 47.1 at lr=1e-4 but collapses to 23.2 at 5e-4 and 0.0 at 1e-3. A similarly misplaced lr/rank on LLAMA-2 could depress GSM8K/BBH/Codex-Eval without reflecting an intrinsic PEFT ceiling. The transferred configuration is therefore the load-bearing assumption behind the 'fall short' caveat. If retuned LoRA closes the gap, the abstract's negative claim is wrong; if it does not, the claim is supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an empirical study of parameter-efficient fine-tuning (PEFT) methods for instruction tuning. The authors evaluate five PEFT families (LoRA, adapter, prefix tuning, prompt tuning, BitFit) and full fine-tuning on T5-3B with the SuperNI dataset, searching over learning rates and PEFT-specific hyperparameters. They find that only LoRA and adapter approach full fine-tuning performance under what they call ideal training settings (large rank/adapter size, appropriate learning rate, diverse tasks), while also reporting training instability at high learning rates/ranks, weaker low-data generalization for LoRA, weaker task-level memorization, and underperformance on reasoning/coding/long-form generation in a second setup on LLAMA-2 7B with the TÜLU dataset.","tokens_in":11810,"tokens_out":4357,"duration_ms":39825,"significance":"If the central claim holds, the paper provides practical guidance for practitioners choosing between PEFT and full fine-tuning for instruction tuning, with the headline result that LoRA and adapter can nearly match full fine-tuning with favorable hyperparameters. The study has notable strengths: a broad hyperparameter grid, coverage of several PEFT families, two model/dataset settings, public benchmarks, and detailed appendix tables. The authors also honestly disclose limitations and reproducibility issues. However, the strength of the conclusions is limited by the lack of statistical uncertainty quantification and by the untested transfer of hyperparameters from the first setup to the second.","major_comments":[{"comment":"The negative finding that LoRA and adapter underperform in complex reasoning, coding, and long-form generation (Table 3: FT 43.4 vs LoRA 38.78 vs adapter 35.26) rests on hyperparameters transferred unchanged from the T5/SuperNI setup: learning rate 1e-4, LoRA rank 512, adapter size 512, with no tuning or sensitivity analysis on LLAMA-2/TÜLU. Given that Table 4 shows LoRA's RougeL collapses from 47.1 at lr=1e-4 to 23.2 at 5e-4 and 0.0 at 1e-3 on T5, the gap in Table 3 could reflect a misplaced learning rate or rank for the larger model rather than an intrinsic limitation of PEFT. The abstract's conclusion that LoRA/adapter 'fall short' in these capabilities is load-bearing and needs support from either a hyperparameter search on Setup 2 or a small sensitivity analysis (e.g., a few learning rates and ranks) demonstrating that the gap persists across reasonable configurations.","section":"Sec. 2.2, Table 3, Appendix A"},{"comment":"The quantitative claims lack statistical support. Table 1, Table 2, and Table 3 report single values without confidence intervals or error bars, and while Fig. 1a and Fig. 3a state that results are averaged over three seeds, no variance is shown. Appendix D further states that runs with unstable training that produce near-zero RougeL are included in the reported averages. This conflation of training divergence with typical performance directly affects Findings 3 and 5 (training instability and low-data generalization) and makes it difficult to judge whether the gap between LoRA/adapter and full fine-tuning in Table 1 is significant. Please report the number of runs per setting, provide variance or error bars for averaged results, and either exclude diverged runs or report them separately as a measure of instability.","section":"Tables 1-3, Fig. 1a/3a, Appendix D"}],"minor_comments":[{"comment":"The text refers to '(Fig.??)' when discussing memorization results; the figure number is missing and should be Table 2 or a proper figure reference.","section":"Sec. 3.6"},{"comment":"The statement 'we have selected seven PEFT method' does not match the five methods evaluated in the experiments; additionally, LoRA is cited as reference [10] (Lester et al., prompt tuning) rather than [8] (Hu et al., LoRA), and the citation for prompt tuning should be checked.","section":"Sec. 4 (Related Work)"},{"comment":"The caption says 'Each point represents a run with different random seeds,' which is ambiguous: does each point correspond to a single run (not averaged) or an average over seeds? Please clarify for Figs. 1b, 2a, 2b, and 3b.","section":"Fig. 2 caption"},{"comment":"The 'Average' column averages five metrics with different scales (EM, F1, P@10) across heterogeneous capabilities; this aggregation is difficult to interpret and should be justified or replaced with a per-category comparison.","section":"Table 3"},{"comment":"The description says the data comprises 707 training tasks and 50 validation tasks 'all randomly selected from SuperNI original training tasks'; it would be clearer to state whether the 50 validation tasks are disjoint from the 707 training tasks and how they were selected.","section":"Sec. 2.1"},{"comment":"The hyperparameter lists for Setup 1 do not mention LoRA alpha, which interacts with rank and learning rate; please state whether alpha was fixed or tuned, especially since alpha=512 is used in Setup 2.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful empirical study, but the two major issues—untested hyperparameter transfer to Setup 2 and lack of uncertainty quantification—need to be addressed before the claims can be taken at face value. The related-work section also contains citation errors that should be corrected during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the headline result—LoRA and adapter can nearly match full fine-tuning on SuperNI when given a large rank/size and the right learning rate—is supported by the data (Table 1: 47.1/46.7 vs 47.8). The paper does real work: a wide grid search, the low-data ramp-up comparison in Fig 3a, and the task-level memorization numbers in Table 2. These measurements are worth having, even where they align with Biderman et al. and Zhuo et al.\n\nThe soft spots are in the right places. The stress-test concern is valid and load-bearing for Finding 7: the LLAMA-2/TÜLU experiments inherit lr=1e-4, r=512, s=512 from T5/SuperNI without retuning, and Appendix A admits those hyperparameters might not suit a newer architecture. So the GSM8K gap (37.0 FT vs 29.1 LoRA) and the Codex-Eval gap (33.9 vs 19.7) could be artifacts of a misplaced lr/rank rather than an intrinsic PEFT ceiling. The paper should have retuned or explicitly run a small lr sweep for LLAMA-2.\n\nTwo smaller issues: the abstract and conclusion claim LoRA/adapter fall short in long-form generation, but no experiment in the paper measures long-form generation—Table 3 has MMLU, GSM, BBH, TydiQA, Codex-Eval, nothing open-ended. That sentence should be removed or backed by data. And checkpoint selection is ambiguous: Sec 2.1 says 'the best-performing checkpoint is selected for evaluation on the test set,' which is a form of test-set peeking; combined with single numbers in Tables 1–3 and the Appendix D note that unstable runs produce near-zero RougeL, the size of the reported gaps is uncertain.\n\nThe citation pattern is fine, and the limitations section is honest. This isn't a polished paper, but it's a serious empirical study with enough substance to merit referee time. A reviewer should ask for variance estimates, a retuned or ablated LLAMA-2 setup, and a fix for the long-form claim.","headline":"Useful empirical sweep supporting the 'LoRA/adapter can nearly match full fine-tuning' claim, but the LLAMA-2/TÜLU negative findings rest on transferred hyperparameters and an unsupported long-form claim.","tokens_in":12344,"tokens_out":3359,"would_cite":false,"duration_ms":30007,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that only LoRA and adapters can substitute for full finetuning in instruction tuning, and then only with an ideal hyperparameter recipe.","keywords":["parameter-efficient finetuning","instruction tuning","LoRA","adapters","hyperparameter search","cross-task generalization","memorization","large language models"],"falsifier":"Re-run the open instruction tuning setup with a hyperparameter search over learning rates from $10^{-5}$ to $3\\times10^{-4}$ and LoRA ranks from 128 to 1024 on the TÜLU dataset; if a tuned LoRA still lags full finetuning on GSM8K, BBH, and Codex-Eval by the same margins, the paper's boundary holds, whereas if it matches, the boundary was set by the reused hyperparameters.","tokens_in":11294,"feed_emoji":"🧠","tokens_out":11117,"duration_ms":93520,"temperature":0.7,"pith_summary":"This paper asks when parameter-efficient finetuning (PEFT) can stand in for full finetuning when teaching large language models to follow instructions. The authors run a systematic comparison of five PEFT families under a wide grid of learning rates, LoRA ranks, adapter sizes, model sizes, and numbers of training tasks. Their central claim is that only LoRA and adapters come close to full finetuning, and they do so only under an ideal training setting: a matching learning rate, the largest permissible rank or bottleneck width, and a diverse set of instruction tasks. With that recipe, LoRA reaches 47.1 RougeL and the adapter 46.7 on SuperNI with T5-3B, against 47.8 for full finetuning. The paper also establishes that these methods are unstable away from the ideal setting, that LoRA is a slower learner needing more tasks to generalize, that LoRA has weaker task-level memorization, and that both fall short on complex reasoning, coding, and long-form generation in open instruction settings.","feed_headline":"LoRA and adapters nearly match full finetuning when tuned right","feed_subtitle":"With large rank, tuned LR, and diverse tasks, they land within 0.7 RougeL of full finetuning at a fraction of the cost.","key_machinery":"The argument is carried by the contrast between two parameter-efficient mechanisms. LoRA freezes the pretrained weights and trains a low-rank update of rank $r$ applied additively to the query and key projections; the adapter inserts a trainable bottleneck layer of width $s$ between frozen transformer blocks. The study's method is a grid search that cross-products the learning rate $\\eta$ with these PEFT sizes, alongside controlled variation of model size, number of training tasks, and train/test task overlap, scored by RougeL on SuperNI and by a broader suite on TÜLU. The key observed mechanism is a stability boundary: as $r$ or $s$ grows, the optimal learning rate narrows, and too high a learning rate drives training instability that can collapse RougeL to near zero.","core_discovery":"The central discovery is an empirical boundary on when parameter-efficient tuning pays for instruction tuning. Among the methods tested, only LoRA and adapters come close to full finetuning, and they do so only when the learning rate is appropriate, the LoRA rank or adapter size is set to the largest value allowed, and the training tasks are diverse. Under those conditions, LoRA trails full finetuning by 0.7 RougeL points and the adapter by 1.1 points on the first benchmark. Away from the ideal setting, both methods show training instability, occasionally collapsing to near-zero RougeL, and the instability grows worse as rank or adapter size increases. The paper further reports that LoRA needs more tasks than adapter or full finetuning to generalize to unseen tasks, that LoRA memorizes task-level patterns less effectively, and that in open instruction tuning on a 7-billion-parameter model, both LoRA and adapter underperform full finetuning on reasoning, coding, and long-form generation, with LoRA ahead of adapter overall.","pith_inferences":["If the hyperparameters from the first setup were re-tuned for the larger model instead of transferred unchanged, the reported reasoning and coding gaps might narrow; the paper's own limitations flag that its optimal settings may not reflect the latest architectures and datasets.","Because performance kept rising with rank up to the tested maximum of 512, ranks beyond that could close more of the remaining gap, although the observed instability and growing storage cost set a practical ceiling.","LoRA's weaker task-level memorization could be reframed as a privacy-friendly property in deployments where retaining task-specific training patterns is undesirable, complementing earlier work on instance-level memorization.","A direct next experiment would be a per-model learning-rate schedule that anneals from high to low during training to avoid the found instability while preserving the expressiveness of large ranks."],"forward_implications":["Under the ideal setting, practitioners can substitute LoRA or an adapter for full finetuning while training only about 10 percent or 7 percent of the parameters, cutting memory and storage costs.","Finding the ideal setting requires searching the learning rate, because a high learning rate combined with a large rank or bottleneck width can make training collapse; the reported gains should not be read as rank-only improvements.","When only a handful of instruction tasks are available, full finetuning remains the better choice, with adapter as the safer PEFT fallback, while LoRA needs a large task pool to show its generalization strength.","For deployments that emphasize complex reasoning, coding, or long-form generation, full finetuning still outperforms both PEFT methods, and among the two, LoRA is the stronger choice overall.","LoRA's weaker task-level memorization means applications that expect to reuse a task type seen in training may lose accuracy with LoRA compared with adapter or full finetuning."],"supporting_citations":[{"why":"Defines LoRA, the low-rank adaptation method whose learning-rate sensitivity and task generalization are the paper's main subject.","marker":"[8]"},{"why":"Defines the adapter bottleneck method that serves as the main LoRA competitor throughout the study.","marker":"[7]"},{"why":"Supplies SuperNI, the instruction dataset and the Tk-Instruct finetuning baseline used in Setup 1.","marker":"[22]"},{"why":"Supplies TÜLU and the evaluation suite covering reasoning, coding, multilinguality, and open instructions in Setup 2.","marker":"[23]"},{"why":"Provides the 7-billion-parameter base model used for the open instruction tuning comparison.","marker":"[20]"},{"why":"Provides the T5 language-model-adapted base models used for the hyperparameter, scaling, and data-size experiments.","marker":"[18]"},{"why":"Cited as the theoretical result used to explain why prompt tuning and prefix tuning lack the expressiveness of finetuning.","marker":"[17]"},{"why":"Reports complementary LoRA findings on learning less and forgetting less, used to contextualize the learning-rate sensitivity and memorization observations.","marker":"[1]"}],"fun_headline_variants":["LoRA and adapters nearly match full tuning with ideal settings","Ideal settings let LoRA and adapters rival full finetuning","Only LoRA and adapters come close, given ideal conditions","Without ideal settings, LoRA and adapters become unstable","LoRA needs more training tasks to generalize to unseen tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study assumes that the hyperparameters that work best on the smaller model and the first dataset transfer unchanged to the larger model and the second dataset; if they are suboptimal there, the reported shortfalls in reasoning and coding could be a tuning artifact rather than a real limitation of LoRA and adapters.","fun_headline_variants_meta":{"raw":{"variants":["LoRA and adapters nearly match full tuning with ideal settings","Ideal settings let LoRA and adapters rival full finetuning","Only LoRA and adapters come close, given ideal conditions","Without ideal settings, LoRA and adapters become unstable","LoRA needs more training tasks to generalize to unseen tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001696,"raw_usage":{"total_tokens":6769,"prompt_tokens":1044,"completion_tokens":5725,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":5639}},"tokens_in":660,"tokens_out":5725,"duration_ms":40259,"temperature":1.0,"reasoning_tokens":5639,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:29:56.733918+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the open instruction tuning setup with a hyperparameter search over learning rates from $10^{-5}$ to $3\\times10^{-4}$ and LoRA ranks from 128 to 1024 on the TÜLU dataset; if a tuned LoRA still lags full finetuning on GSM8K, BBH, and Codex-Eval by the same margins, the paper's boundary holds, whereas if it matches, the boundary was set by the reused hyperparameters.","supporting_citations":[{"cited_title":"Houlsby, A","cited_arxiv_id":null,"evidence_quote":"Defines the adapter bottleneck method that serves as the main LoRA competitor throughout the study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies SuperNI, the instruction dataset and the Tk-Instruct finetuning baseline used in Setup 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies TÜLU and the evaluation suite covering reasoning, coding, multilinguality, and open instructions in Setup 2."},{"cited_title":"Touvron, L","cited_arxiv_id":null,"evidence_quote":"Provides the 7-billion-parameter base model used for the open instruction tuning comparison."},{"cited_title":"Petrov, P","cited_arxiv_id":null,"evidence_quote":"Cited as the theoretical result used to explain why prompt tuning and prefix tuning lack the expressiveness of finetuning."}],"review_version":1}