{"id":"1315483f-9ed0-48cc-8342-4235bb11d84f","arxiv_id":"2507.19909","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On three Java APR benchmarks, LoRA and IA3 adapters match or beat full-model fine-tuning for most tested code LLMs while training less than one percent of parameters.","lead":"This study tests whether cheap fine-tuning methods can replace retraining the whole model when adapting large language models to fix code bugs. It benchmarks 15 models on Java repair benchmarks and finds LoRA adapters often match or beat full fine-tuning while training under one percent of parameters.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table V compares FMFT at epoch 3 against PEFT at epoch 1, a favorable checkpoint choice that may inflate the reported PEFT advantage; the claimed 172%/225%/153% gains are actually ratios, not percentage gains.","rationale":"The reader identifies the asymmetric epoch choice as the weakest assumption; I agree this is the most load-bearing methodological concern for the central PEFT-vs-FMFT claim. However, my own quick calculation from the reported tables suggests that the concern may shrink rather than fully reverse the PEFT advantage in most cells: for example, using FMFT epoch-1 numbers instead of epoch-3 still leaves LoRA better for CodeGen-2B and DeepSeekCoder-1.3b on all benchmarks, and CodeT5-small is already better with FMFT even at epoch 3. The more precise problem is that the magnitude of the reported advantage is overstated, both by the epoch selection and by the incorrect interpretation of ratios as percentage gains. This does not change the disposition: the paper's empirical contribution is still valuable and plausibly correct, but the strength of the claim should be tempered with a fairer checkpoint comparison and corrected arithmetic. Hence the reader's CONDITIONAL verdict remains appropriate, and I do not propose a change to the verdict.","tokens_in":22858,"tokens_out":7580,"duration_ms":82351,"concrete_test":"Using the replication package, rerun the benchmark evaluation for LoRA and IA3 at epoch 3 (the checkpoints already exist from the 3-epoch training described in Table IV) and recompute FMFT at its best epoch per model/benchmark (take the max of epoch 1 and epoch 3 counts from Table III). Then re-tabulate Table V to compare LoRA/IA3 against this best-epoch FMFT. If PEFT still outperforms FMFT in a majority of model-benchmark cells, the qualitative claim is robust to checkpoint choice; if the majority flips, the claim is an artifact of the asymmetric epoch selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that PEFT achieves better results than FMFT rests on Table V, where FMFT is scored at epoch 3 but LoRA/IA3 are scored at epoch 1. The paper justifies epoch 1 for PEFT by noting that validation metrics plateau after epoch 1 (Table IV), but benchmark performance is not necessarily the same as validation performance. For FMFT, validation metrics generally improve through epoch 3 (Table II), so the epoch-3 checkpoint is the latest, not necessarily the best for benchmarks. Indeed, Table III shows several FMFT epoch-1 benchmark counts higher than epoch-3 counts (e.g., CodeT5-small HumanEval-Java 44 vs 39; CodeGen-350M Defects4J 67 vs 61), so scoring FMFT at epoch 3 may understate FMFT. Conversely, PEFT benchmark results at epoch 3 are not reported, so it is unknown whether the epoch-1 validation plateau corresponds to the best benchmark checkpoint or merely an early checkpoint. This asymmetry could shrink or reverse the reported PEFT advantage in multiple model-benchmark cells, and the paper's summary overstates the effect by reporting ratios as percentage gains (e.g., CodeGen-2B LoRA/FMFT counts 19/11, 81/36, 98/64 are 172%, 225%, 153% of FMFT, not gains of those sizes).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an empirical study of fine-tuning strategies for LLM-based automated program repair. Six code LLM families (15 checkpoints) are evaluated on QuixBugs, HumanEval-Java, and Defects4J under three regimens: no fine-tuning, full-model fine-tuning (FMFT), and parameter-efficient fine-tuning (PEFT) with LoRA and IA3. The authors find that zero-shot performance varies widely, that full fine-tuning helps smaller/weaker models but hurts stronger models on some benchmarks, and that PEFT often matches or beats full fine-tuning while updating under 1% of parameters. A fourth research question examines LoRA rank and scaling-factor sensitivity. The paper concludes that full fine-tuning decreases benchmarking performance and that PEFT achieves better results, and it provides a replication package.","tokens_in":23142,"tokens_out":4587,"duration_ms":53405,"significance":"If the headline comparison is fair, the paper makes a practically important contribution: adapter-based fine-tuning at <1% trainable parameters can match or exceed full fine-tuning on APR benchmarks, with clear resource implications. The study is broad (15 checkpoints, 3 benchmarks, two adapters, hyperparameter sweeps) and the replication package is a concrete strength. However, the central PEFT-vs-FMFT claim rests on an asymmetric checkpoint choice, and the abstract-level claim is stronger than the tables support for several model families. These issues are addressable, but they require substantive revision rather than copy-editing.","major_comments":[{"comment":"The headline FMFT-versus-PEFT comparison is asymmetric: FMFT is scored at epoch 3, while LoRA and IA3 are scored at epoch 1. The text justifies epoch 1 for PEFT by citing validation-plateau behavior (Table IV), but benchmark performance is not necessarily aligned with validation metrics. Table III shows several FMFT cells where epoch 1 beats epoch 3 (e.g., CodeT5-small HumanEval-Java 44 vs 39, CodeGen-350M Defects4J 67 vs 61, CodeGen-2B Defects4J 66 vs 64), so epoch 3 is not clearly the best FMFT checkpoint. Conversely, Table IV reports no benchmark results for PEFT at epochs 2 or 3, so one cannot verify that the epoch-1 validation plateau corresponds to the best benchmark checkpoint. This asymmetry could shrink or even reverse the reported PEFT advantage in several cells. Please report both methods at matched epochs, or at each method's best benchmark checkpoint, and re-evaluate the RQ3 conclusions accordingly.","section":"Section IV-C, Table V"},{"comment":"The summary states that LoRA for CodeGen-2B achieves 'performance gains of 172%, 225%, 153%' on QuixBugs, HumanEval-Java, and Defects4J. These numbers are ratios of plausible-patch counts (19/11=1.73, 81/36=2.25, 98/64=1.53), not percentage gains; a 172% gain would mean 2.72 times the baseline count. The correct phrasing is that the counts are 1.73x, 2.25x, and 1.53x the FMFT counts, or equivalently 73%, 125%, and 53% higher. This numerical misreport is part of the abstract-level claim and should be corrected.","section":"Section IV-C, RQ3 summary"},{"comment":"The abstract's blanket statement that 'full fine-tuning techniques decrease the benchmarking performance of various models' is not supported by the paper's own results in full generality. Table III shows that full fine-tuning substantially improves CodeT5, Bloom, and CodeGen-350M relative to their base models, while the deterioration is concentrated in DeepSeekCoder and StarCoder. Similarly, Table V shows that PEFT does not uniformly beat FMFT: e.g., CodeT5-small on Defects4J has FMFT=60 vs LoRA=50, and CodeT5-base on Defects4J has FMFT=75 vs LoRA=67. Moreover, FMFT results are marked X for CodeGen-6B and DeepSeekCoder-6.7B, so the PEFT-vs-FMFT claim is only tested on models up to 3B parameters. Please qualify the conclusions to state which models and configurations support the PEFT advantage, and restrict claims about the 6B models accordingly.","section":"Abstract and Section V"}],"minor_comments":[{"comment":"The text says 'we investigate the following 8 values for both rank and scaling factor' but then lists seven values: 1, 2, 4, 8, 16, 32, 64; Figures 3-6 also show seven values on each axis. Please correct the count or add the missing value.","section":"Section IV-D"},{"comment":"In the StarCoder-3b row, the Exact Match value for validation epoch 2 is printed as '.019'; this is presumably 0.19 and should be formatted consistently with the other entries.","section":"Table II"},{"comment":"The caption states that the best results on the validation dataset are highlighted in bold, but no bold highlighting is visible in the typeset table, making the best values difficult to identify.","section":"Table IV"},{"comment":"The caption marks FMFT for CodeGen-6B and DeepSeekCoder-6.7B as X due to hardware limitations; it would help to add a sentence in the main text noting that PEFT results for these two 6B models are only compared against the base model, not against FMFT.","section":"Section IV-C, Table V"},{"comment":"The text explaining discrepancies with Jiang et al. mentions that Jiang et al. did not specify the Java version and filtered out more programs by length; please add the exact Defects4J version and filtering criteria used in that reference, since some Defects4J cells differ substantially from the values in parentheses.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The asymmetric epoch selection in Table V is the central technical concern, but it is addressable: the authors could report benchmark results for all epochs and both methods, or re-frame the claim to a method-level comparison at each method's best checkpoint. The paper's breadth and replication package are genuine strengths; I would not reject, but the revision needs to be substantive rather than cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper for its breadth, not for its headline. The claim that PEFT beats full fine-tuning for APR is real but overstated, because the comparison scores FMFT at epoch 3 and PEFT at epoch 1. That asymmetry flatters PEFT, and the paper compounds it by reporting ratios as percentage gains.\n\nWhat's actually new is the scale: 15 models across QuixBugs, HumanEval-Java, and Defects4J, each under base, FMFT, LoRA, and IA3, plus a LoRA rank/alpha sweep. The replication package is a genuine asset. They also position the work honestly against RepairLLaMA and Li et al., noting the conceptual possibility of adapters was already raised, and they compare where possible. The Defects4J baselines and the RQ2 table of epoch-by-epoch validation metrics have independent value.\n\nThe soft spot is load-bearing. Table V puts FMFT at epoch 3 and LoRA/IA3 at epoch 1. The text says PEFT validation metrics plateau after epoch 1, but validation plateau is not benchmark performance. Table III shows FMFT epoch-1 counts beating epoch-3 counts in several cells (CodeT5-small HumanEval-Java 44 vs 39; CodeGen-350M Defects4J 67 vs 61), so epoch 3 is not FMFT's best checkpoint. PEFT results at epoch 3 are not reported, so we cannot see whether the epoch-1 plateau matches the best benchmark checkpoint. The honest fix is to score both at their best epoch, or report PEFT at epoch 3, and see whether the PEFT advantage survives.\n\nSecond, the RQ3 summary says CodeGen-2B LoRA gives 'performance gains of 172%, 225%, 153%'. Those are ratios (19/11, 81/36, 98/64); the gains are 72%, 125%, 53%. That misstates the effect.\n\nThere are secondary issues: no repeated runs, no statistical testing, and the PEFT models were selected based on performance for RQ3. Those are disclosed and fixable, but the checkpoint asymmetry is the one that changes the conclusion.\n\nWho this is for: practitioners choosing between full fine-tuning and adapters for APR, and empirical researchers studying PEFT for code. They will get useful numbers and a solid baseline. The weaker claim—that PEFT is competitive, sometimes better, at under 1% of trainable parameters—is supported by the data; the stronger claim that FMFT 'decreases' performance is not established at matched checkpoints.\n\nI'd send this to a serious referee. The breadth, the replication package, and the explicit comparisons to prior work justify referee time. The referee should require the epoch-asymmetry fix and corrected percentage language before acceptance.","headline":"A broad and useful empirical matrix, but the headline PEFT-over-FMFT advantage rests on an asymmetric checkpoint choice and misstated ratio-as-percentage claims.","tokens_in":23676,"tokens_out":5396,"would_cite":true,"duration_ms":50248,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning every weight of a code LLM often hurts its bug-fixing performance; freezing the model and training a tiny adapter fixes more bugs at under 1% of the training cost.","keywords":["automated program repair","large language models","parameter-efficient fine-tuning","LoRA","IA3","code models","overfitting","APR benchmarks"],"falsifier":"Re-run the same three benchmarks scoring full fine-tuning at epoch 1 rather than epoch 3 for CodeT5-small and CodeT5-base, and compare with LoRA and IA3 at epoch 1; if the fully fine-tuned models then match or exceed the adapter models on HumanEval-Java or Defects4J, the paper's central claim would not hold for those models.","tokens_in":22663,"feed_emoji":"🐞","tokens_out":9874,"duration_ms":100065,"temperature":0.7,"pith_summary":"Automated program repair (APR) is increasingly done by adapting large pretrained code models to the bug-fixing task, and the standard adaptation is to fine-tune every weight. This paper argues that for several of these models full fine-tuning actually lowers the number of bugs they fix on three benchmark suites, because the fine-tuning data come from a different distribution than the benchmarks and the models overfit to the training corpus. The alternative the paper tests is parameter-efficient fine-tuning with LoRA and IA3 adapters, which keep the pretrained weights frozen and train only a tiny additional structure. In the reported experiments the adapters match or beat full fine-tuning in a substantial number of configurations, most clearly for CodeGen and DeepSeekCoder, while training under 1% of the parameters. If the finding holds, APR teams can improve repair performance while cutting fine-tuning cost by orders of magnitude.","feed_headline":"LoRA adapters beat full model fine-tuning for bug repair","feed_subtitle":"Training under 1% of parameters fixes more Java bugs on QuixBugs, HumanEval-Java, and Defects4J.","key_machinery":"The load-bearing object is the adapter. LoRA freezes the pretrained weight matrix $W_0$ and learns a low-rank update $\\Delta W = BA$, so only the entries of the small matrices $B$ and $A$ are trainable; IA3 freezes the base model and trains three scaling vectors per layer that re-scale the attention keys, values, and feed-forward activations. Both mechanisms restrict the hypothesis space during fine-tuning. The paper argues that this restriction is exactly what preserves the pretrained model's repair ability: adapter-trained models show near-equal training and validation metrics, indicating less overfitting to the fine-tuning corpus, while full fine-tuning moves all weights toward the distribution of that corpus and away from the benchmark distributions.","core_discovery":"The paper's central claim is that full-model fine-tuning decreases the benchmarking performance of several code LLMs on APR, because the fine-tuning corpus and the benchmarks have different bug distributions and the models overfit, whereas parameter-efficient fine-tuning with LoRA or IA3 restricts the number of trainable parameters and achieves better results. The evidence is a systematic comparison on QuixBugs, HumanEval-Java, and Defects4J across six model families. In the reported configurations, CodeGen and DeepSeekCoder improved substantially with LoRA relative to full fine-tuning: for CodeGen-2B, LoRA used only 0.09% of the model's parameters while achieving improvements of 172%, 225%, and 153% over full fine-tuning on QuixBugs, HumanEval-Java, and Defects4J respectively. LoRA also beat IA3 in 21 of 24 comparisons. Models that were already strong zero-shot, notably DeepSeekCoder, lost performance after full fine-tuning, while weaker models such as CodeT5 gained from it.","pith_inferences":["The headline comparison in Table V reports full fine-tuning at epoch 3 and adapter tuning at epoch 1; if full fine-tuning were scored at its best epoch, the reported advantage for adapters would shrink for several models, especially CodeT5-small and CodeT5-base.","The paper itself flags data leakage as a concern; if leakage inflates zero-shot scores for the newer models, the observed 'full fine-tuning hurts' pattern partly reflects the model forgetting leaked benchmark solutions rather than losing general repair competence.","Everything here is Java and single-hunk bugs; an obvious extension is to test whether the adapter advantage persists on multi-hunk, multiline, or non-Java repair tasks before generalizing the recommendation."],"forward_implications":["APR systems can default to LoRA-style adapters for code LLMs, since most reported configurations show equal or better plausible-patch counts with far fewer trainable parameters.","Strong zero-shot models such as DeepSeekCoder should not be fully fine-tuned on bug-fix corpora; the paper's data show they lose benchmark performance, while adapters preserve or improve it.","Models too large for full fine-tuning on a single GPU node, around 6B parameters in this study, become trainable with adapters, widening the range of models small teams can use.","LoRA hyperparameter search is largely unnecessary for APR under the tested ranges, since rank and scaling factor had negligible effect on CodeBLEU for CodeGen-2B."],"supporting_citations":[{"why":"It supplies the evaluation framework, the HumanEval-Java benchmark, and the baseline setup that this study extends.","marker":"[8]"},{"why":"It provides the CLM bug-fix dataset used for all fine-tuning experiments.","marker":"[19]"},{"why":"It defines LoRA, the low-rank adapter method that carries the PEFT comparison.","marker":"[21]"},{"why":"It defines IA3, the second adapter method, and its scaling-vector mechanism.","marker":"[22]"},{"why":"It provides CodeBLEU, the code-structure metric used to track validation progress during fine-tuning.","marker":"[20]"},{"why":"It supplies Defects4J, one of the three APR benchmarks.","marker":"[60]"},{"why":"It supplies QuixBugs, the second APR benchmark.","marker":"[61]"}],"fun_headline_variants":["Full fine-tuning hurts code LLMs; LoRA fixes more bugs","LoRA beats full fine-tuning for automated program repair","Parameter-efficient tuning outperforms full fine-tuning in APR","Fine-tuning less is more: LoRA wins on bug repair"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison assumes that scoring full fine-tuning at epoch 3 and adapter tuning at epoch 1 is fair, because the adapter metrics plateau after the first epoch; if full fine-tuning were scored at its best epoch, the reported advantage for adapters would shrink for several models.","fun_headline_variants_meta":{"raw":{"variants":["Full fine-tuning hurts code LLMs; LoRA fixes more bugs","LoRA beats full fine-tuning for automated program repair","Parameter-efficient tuning outperforms full fine-tuning in APR","Fine-tuning less is more: LoRA wins on bug repair"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000778,"raw_usage":{"total_tokens":3492,"prompt_tokens":1054,"completion_tokens":2438,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":2370}},"tokens_in":670,"tokens_out":2438,"duration_ms":19816,"temperature":1.0,"reasoning_tokens":2370,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:50:59.024378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same three benchmarks scoring full fine-tuning at epoch 1 rather than epoch 3 for CodeT5-small and CodeT5-base, and compare with LoRA and IA3 at epoch 1; if the fully fine-tuned models then match or exceed the adapter models on HumanEval-Java or Defects4J, the paper's central claim would not hold for those models.","supporting_citations":[{"cited_title":"LoRA: Low-Rank Adaptation of Large Language Models","cited_arxiv_id":null,"evidence_quote":"It defines LoRA, the low-rank adapter method that carries the PEFT comparison."},{"cited_title":"Few-Shot Parameter-Efficient Fine-Tuning Is Better and Cheaper than in-Context Learning","cited_arxiv_id":null,"evidence_quote":"It defines IA3, the second adapter method, and its scaling-vector mechanism."},{"cited_title":"Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs","cited_arxiv_id":null,"evidence_quote":"It supplies Defects4J, one of the three APR benchmarks."},{"cited_title":"QuixBugs: A Multi-Lingual Program Repair Benchmark Set Based on the Quixey Challenge","cited_arxiv_id":null,"evidence_quote":"It supplies QuixBugs, the second APR benchmark."}],"review_version":1}