{"id":"fec15a4f-88d1-4584-9726-f1d93008e257","arxiv_id":"2505.02931","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using only 1,000 fine-tuning examples with full fine-tuning improves the number of plausible repairs by up to 78%, and iterative generation strategies help base models most on complex benchmarks like Defects4J.","lead":"This paper tests how many repair attempts and how many rounds of feedback get the best bug fixes from three instruction-tuned language models, with a cap of ten patches per bug. It finds that tiny fine-tuning sets (1,000 examples) can give large gains, and that iterative refinement helps base models most on complex bugs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Output-format compliance may confound the headline FFT-1K gain: the parser only extracts code inside triple backticks, and fine-tuning teaches that format.","rationale":"The paper's central contribution is that small-scale full fine-tuning substantially improves APR performance. That claim requires that the improvement reflects better repair, not simply better adherence to an output format the fine-tuning data happens to teach. Section 3.4's extraction logic creates exactly this risk: it requires triple backticks and never describes a fallback, while the training data are formatted to match the prompt. Because the base models are already instruction-tuned, the marginal effect of 1K FFT examples could be disproportionately about learning to wrap code in fences. This is a concrete, falsifiable threat that can be tested directly with the released patches and pipeline. If the parser change raises base-model performance materially, the headline finding and the challenge to prior FFT studies would need substantial qualification. If it does not, the finding stands. Since the concern is addressable and does not by itself invalidate the study, the existing conditional verdict remains appropriate; the condition should explicitly include this parser check in addition to the reader's suggested checks on test-overfitting and dataset overlap.","tokens_in":18372,"tokens_out":11691,"duration_ms":136602,"concrete_test":"Using the released replication package, re-run the three base instruction-tuned models with a more forgiving parser: when no code fence is found, treat the full trimmed model response as the candidate patch; also report per-condition parse-failure/uncompilable rates. If CodeLlama, DeepSeek, or Llama base plausible-patch counts rise substantially toward their FFT-1K counts, the headline gain is largely an artifact of output-format compliance rather than repair capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that FFT with 1K samples yields up to 78% improvement (Table 1: CodeLlama 60 to 107, DeepSeek 76 to 129) depends on comparing base and fine-tuned models under the same code-extraction rule. Section 3.4 states that the generated output is parsed by looking for triple backquote symbols, with no fallback described for responses that contain the fixed method without code fences. Meanwhile, the fine-tuning data are rephrased to conform with the instruction format (Section 3.3), so FFT models are explicitly trained to emit ```-wrapped code, while the base instruction-tuned models may produce correct code in plain text that the parser discards. If this happens often, a large part of the reported gain is output-format compliance rather than repair ability, and Finding 1 and Finding 3 would be overstated. The paper does not report parse-failure or uncompilable rates by model condition, so this confound is currently unexamined. This is distinct from, and potentially more severe than, the test-overfitting caveat the reader raised: a 4% overfitting rate among sampled plausible patches would not overturn a 60-to-107 improvement, but a systematic base-model undercount due to parsing could.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates an APR pipeline that limits patch generation to 10 per bug and combines instruction-tuned LLMs with fine-tuning at three dataset sizes (1K, 30K, 65K) and two methods (FFT and LoRA). The evaluation on HumanEval-Java and Defects4J compares seven generation strategies that trade off outputs per iteration against number of iterations. The central claims are that FFT with only 1K samples yields large increases in the number of plausible patches (up to 78%), that larger fine-tuning datasets produce diminishing returns, and that base models benefit from iterative refinement more than fine-tuned models do.","tokens_in":18584,"tokens_out":4581,"duration_ms":47790,"significance":"If the results hold, the finding that FFT with around 1K samples is effective and underused is practically important for APR, and it challenges prior studies that reported limited gains from full fine-tuning. The paper's strengths include the release of the full pipeline, nine fine-tuned models, generated patches, and manual assessments, which enable replication and further analysis. The systematic comparison of iterative strategies and the use of two benchmarks are also valuable. However, the headline numbers rest on a parsing assumption that has not been validated across model conditions, and the RQ2 analysis selects variants using the evaluation benchmark itself; both issues need to be addressed before the central claims can be accepted.","major_comments":[{"comment":"Section 3.4 states that the generated output is parsed by looking for the triple backquote symbol, with no fallback described for responses that contain the fixed method without code fences. The fine-tuning data are rephrased to conform with an instruction format (Section 3.3), so fine-tuned models are trained to emit fenced code, while the base models are not. The paper does not report how often each model condition produced no parsable fenced block or an empty extraction. Because Table 1 compares base and fine-tuned models under this same parser, a systematic undercounting of base-model outputs due to format non-compliance would inflate the reported improvements (e.g., CodeLlama 60 to 107, DeepSeek-Coder 76 to 129). Please report parse-failure and uncompilable rates per model condition and recompute the main tables with a fallback parser that treats the whole response as code when no fence is found.","section":"3.4, Table 1"},{"comment":"Section 4.2 states that for RQ2 the authors selected 'the base model as well as the best performing variants trained with FFT and LoRA on HumanEval-Java according to RQ1'. Because the RQ2 evaluation uses the same HumanEval-Java benchmark, the reported comparisons in Figure 4a are based on variants selected for their RQ1 performance on that benchmark, which is a form of selection bias. This is load-bearing for Finding 6 and the fine-tuned-versus-base conclusions in RQ2. The authors should either use a separate selection benchmark or a pre-registered criterion, or explicitly present the HumanEval-Java RQ2 numbers as post-selection estimates rather than unbiased evaluations.","section":"4.2"},{"comment":"The claims that diminishing returns and reduced iterative effectiveness are 'likely due to overfitting' are not supported by direct evidence in the manuscript. No training loss curves, held-out performance, output diversity measurements, or other overfitting indicators are reported. This is a plausible hypothesis, but as stated it is speculation, and the causal language is stronger than the evidence warrants. Please add supporting evidence or soften the claims.","section":"4.1.1, Findings 2 and 6"}],"minor_comments":[{"comment":"In Section 3.1, 'placement of plausible patches withing the generated outputs' should be 'within the generated outputs'.","section":"3.1"},{"comment":"In Section 4.1.1, 'Both, FTT and LoRA methods' should read 'Both FFT and LoRA methods'.","section":"4.1.1"},{"comment":"The Venn diagram in Figure 3 lists counts without explaining how the regions correspond to the four strategy/condition combinations; a legend or a short description of the circle ordering would improve readability.","section":"Figure 3"},{"comment":"The heatmaps in Figure 4 show differences relative to Strategy A, but the body text does not restate this when discussing the numeric values; consider making the baseline explicit in the text for readers who do not inspect the caption.","section":"4.2.2, Figure 4"},{"comment":"Reference [27] is missing the publication venue and year; please complete the bibliographic details.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The parser/format confound is the most serious issue, but it is directly addressable because the artifacts are released; a re-analysis with a fallback parser and per-condition parse-failure rates could settle it without new experiments. The RQ2 selection bias is also fixable through a clean selection protocol. I recommend requesting a major revision rather than rejecting, since the replication package makes the required additional analysis feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the short version: solid, useful empirical sweep with a real caveat on the headline number. The paper systematically varies fine-tuning dataset size (1K/30K/65K), FFT vs LoRA, and generation strategy under a hard 10-patch budget, across three instruction-tuned models and two Java benchmarks. That's a new and practically relevant measurement. The result that FFT with 1K samples can match or beat 30K/65K is worth knowing, and it does challenge the \"FFT needs big data\" assumption in parts of the APR literature. The replication package (code, generated patches, fine-tuned models, manual labels, sampling seed) is exactly the right level of transparency.\n\nThe biggest soft spot is the output parser. Section 3.4 says code is extracted only from triple-backtick blocks, and the fine-tuning data was reformatted to conform to that style. So fine-tuned models are explicitly taught to emit fenced code, while base models may answer in plain text that gets discarded. The paper doesn't report parse-failure rates by model condition. If that rate is nontrivial for base models, a chunk of the claimed up-to-78% improvement is format compliance, not repair ability. This is testable with the released pipeline, and I would want that analysis before trusting the headline. It's a more serious threat than the 4% overfitting found in manual inspection.\n\nOther soft spots: RQ2 chooses the \"best\" FFT/LoRA variants on HumanEval-Java and then evaluates strategies on the same benchmark—selection bias. The overfitting explanation for diminishing returns is plausible but unsupported by loss curves or a held-out split. And the authors should confirm their fine-tuning set doesn't contain Defects4J or HumanEval-Java samples; they rely on benchmark recency rather than deduplication. The lack of statistical tests is less concerning here because decoding is deterministic, though a per-problem breakdown would help.\n\nCredit where due: the paper is honest about the plausibility metric, manually inspected 3,298 plausible patches, and made the assessments public. The strategy findings (base models benefit from iteration, fine-tuned models peak early) are useful even if the exact gains shift after the parser check.\n\nWho's this for? Anyone configuring APR agents or designing APR evaluation. It deserves a serious referee, with a request to run the parse-failure analysis and verify the 1K-FFT result on an independent sample. Not a desk reject.","headline":"A genuinely useful APR empirical sweep, but the headline FFT-1K gain needs a parse-failure check before you trust it.","tokens_in":19172,"tokens_out":4829,"would_cite":true,"duration_ms":52054,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning instruction-tuned LLMs on just 1,000 examples improves program repair by up to 78%, challenging the need for large APR datasets.","keywords":["automated program repair","large language models","instruction tuning","fine-tuning","iterative refinement","execution feedback","Defects4J","HumanEval-Java"],"falsifier":"Run the full pipeline (all 12 model variants and seven strategies) and validate every plausible patch, not just a third of them, against a stronger oracle—for instance, additional hidden tests for HumanEval-Java and extra edge-case tests for Defects4J. If the overfitting rate is roughly constant across strategies, the paper's rankings survive; if iterative strategies or FFT-1K patches overfit more heavily than average, the 78% gain and the iteration-benefit findings lose their force. A second check is to rerun FFT-1K on several different random 1K subsets to see whether the reported improvement is stable across sampling seeds.","tokens_in":18118,"feed_emoji":"🛠️","tokens_out":7038,"duration_ms":69581,"temperature":0.7,"pith_summary":"This paper argues that automatic program repair with instruction-tuned large language models is best tuned as a balance between two dials: how many patch candidates are generated in one go versus how many times the model is allowed to refine an earlier attempt, all under a strict budget of ten patches per bug. The central empirical claim is that full fine-tuning on only 1,000 examples—under one percent of the available training data—raises the number of plausible patches by up to 78% over the base model, and that this small-scale configuration can beat fine-tuning on 65,000 examples. The authors also find that iteration helps base models and hurts most fine-tuned models on easy bugs, while complex bugs in Defects4J still reward iterative refinement even after fine-tuning. If these results hold, they imply that data-hungry fine-tuning and patch-flooding are both avoidable: small fine-tuning sets plus a modest number of feedback rounds are the actionable route to cost-effective LLM repair.","feed_headline":"1,000 examples can lift program repair by up to 78%","feed_subtitle":"Small fine-tuning sets plus a few feedback rounds outperform big data and patch floods.","key_machinery":"The mechanism is an iterative repair pipeline with a hard cap of ten patches per bug. The pipeline prompts a model with bug-delimited code, validates the generated patch by compiling and running the benchmark tests, and feeds the failing test code, a timeout notice, or the compilation error back into the chat context for the next attempt. Seven generation strategies (A: 10×1, B: 8-2, C: 5×2, D: 6-2-2, E: 4-3-3, F: 2×5, G: 1×10) carve the ten-patch budget into different splits of outputs per iteration and iterations, so the paper can isolate the value of iteration from the value of sampling breadth. The fine-tuning axis pairs full fine-tuning (FFT) against LoRA at dataset sizes 1K, 30K, and 65K, giving the three instruction-tuned models a factorial comparison.","core_discovery":"The paper claims that for instruction-tuned LLMs doing automatic program repair, both fine-tune data size and iteration count show a non-monotonic relationship with the number of plausible patches produced. Full fine-tuning on 1K samples lifts CodeLlama from 60 to 107 solved HumanEval-Java problems, DeepSeek-Coder from 76 to 129, and Llama3.1 from 68 to 108—gains of 59–78%—while 65K samples give no further gain and often regress, which the authors attribute to overfitting. LoRA does not match this at 1K but catches up with 30K–65K samples. The same pipeline shows that base models consistently improve when patches are refined over several rounds with execution feedback, whereas fine-tuned models peak with few iterations on simple bugs; on the harder Defects4J benchmark, iterative refinement helps even fine-tuned models, with Llama3.1 Base rising from 28 to 74 plausible patches between the least and most iterative strategy. The paper concludes that the best configuration is a balanced strategy that combines multi-output generation with iterative refinement, tuned per model and per task complexity.","pith_inferences":["If the overfitting interpretation is right, then increasing the diversity or quality of the fine-tuning set, or adding regularization, should push the 30K/65K FFT curves above the 1K peak; this is a direct, testable way to separate overfitting from model-capacity limits.","The 4% overfitting rate found in the manually-checked plausible patches (131 of 3,298) implies the absolute gains over base models are upper bounds on true fixes; ranking the strategies under a stricter correctness oracle (extra tests or human review) could change which strategy looks best, especially for iterative variants that produce many similar patches.","Because base models with iteration solve a set of problems that fine-tuned models miss (about 12% unique on HumanEval-Java for Llama3.1, 19% on Defects4J), a hybrid that routes easy bugs to a fine-tuned single-shot model and hard bugs to an iterative base model could solve more total problems than either alone.","The position analysis suggests an adaptive stopping rule: for fine-tuned models on simple benchmarks, the 10-patch budget is excessive, and compute could be reallocated to more bugs or to iterative refinement on complex ones."],"forward_implications":["APR practitioners can obtain large gains from full fine-tuning on about 1,000 curated examples, instead of collecting and training on tens of thousands of fixes.","Given a budget of ten patches, generating all ten at once is rarely optimal: base models repair more bugs with balanced iterative strategies such as 6-2-2 or 4-3-3, and only extreme iteration (1×10) shows diminishing returns.","Fine-tuned models concentrate their successes in the first few outputs, so stopping after the first five patches captures about 90% of their plausible patches on HumanEval-Java; later outputs add less than 10% to total repairs.","The choice between full fine-tuning and LoRA changes the data requirement: FFT works at 1K samples while LoRA needs 30K–65K, reversing the usual PEFT-led advice for APR.","On complex real-world bugs, iterative refinement remains valuable even for fine-tuned models, so the best strategy depends on task difficulty, not just the model."],"supporting_citations":[{"why":"Supplies the HumanEval-Java benchmark and the earlier finding that fine-tuning improves LLM repair, which this paper's small-data FFT claim extends and reinterprets.","marker":"[22]"},{"why":"Supplies Defects4J, the real-bug benchmark that serves as the complex-task counterpart to HumanEval-Java.","marker":"[59]"},{"why":"A prior APR study that reported limited gains for full fine-tuning and focused on PEFT; the paper directly challenges its conclusions about FFT data requirements.","marker":"[52]"},{"why":"RepairLLaMA, a prior APR fine-tuning study that favored LoRA over full fine-tuning, providing the comparison point for Finding 3.","marker":"[48]"},{"why":"Introduces LoRA, the parameter-efficient fine-tuning technique that the paper compares against full fine-tuning.","marker":"[26]"},{"why":"Empirical evidence that developers are unlikely to examine more than ten patches, motivating the fixed budget cap used in every experiment.","marker":"[20]"},{"why":"One of the three instruction-tuned LLMs studied; its 7B Instruct variant provides a base model and fine-tuning subject.","marker":"[14]"},{"why":"One of the three instruction-tuned LLMs studied; its 6.7B Instruct variant contributes the strongest HumanEval-Java results.","marker":"[15]"},{"why":"One of the three instruction-tuned LLMs studied; its 8B Instruct variant is the newest model and the basis for the Venn-diagram uniqueness analysis.","marker":"[24]"},{"why":"Source of the fine-tuning dataset: single-hunk fixes from GitHub Java commits, from which the 1K, 30K, and 65K subsets are sampled.","marker":"[58]"}],"fun_headline_variants":["1K fine-tune samples lift APR 78%, beating 65K","Why 65K samples underperform 1K for program repair","Iterative refinement beats patch floods in APR","Small data, big APR gains: 78% more plausible patches","APR: fewer examples, more iterations, up to 78% better"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline numbers all count a patch as a success if it compiles and passes the benchmark's test suite; if the roughly 4% overfitting rate found in the 3,298 manually inspected plausible patches extends to the other ~6,000, the reported gains and strategy comparisons shrink, though they need not reverse.","fun_headline_variants_meta":{"raw":{"variants":["1K fine-tune samples lift APR 78%, beating 65K","Why 65K samples underperform 1K for program repair","Iterative refinement beats patch floods in APR","Small data, big APR gains: 78% more plausible patches","APR: fewer examples, more iterations, up to 78% better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1544,"prompt_tokens":1131,"completion_tokens":413,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":747,"completion_tokens_details":{"reasoning_tokens":323}},"tokens_in":747,"tokens_out":413,"duration_ms":4791,"temperature":1.0,"reasoning_tokens":323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:38:52.715209+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline (all 12 model variants and seven strategies) and validate every plausible patch, not just a third of them, against a stronger oracle—for instance, additional hidden tests for HumanEval-Java and extra edge-case tests for Defects4J. If the overfitting rate is roughly constant across strategies, the paper's rankings survive; if iterative strategies or FFT-1K patches overfit more heavily than average, the 78% gain and the iteration-benefit findings lose their force. A second check is to rerun FFT-1K on several different random 1K subsets to see whether the reported improvement is stable across sampling seeds.","supporting_citations":[{"cited_title":"Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs","cited_arxiv_id":null,"evidence_quote":"Supplies Defects4J, the real-bug benchmark that serves as the complex-task counterpart to HumanEval-Java."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A prior APR study that reported limited gains for full fine-tuning and focused on PEFT; the paper directly challenges its conclusions about FFT data requirements."},{"cited_title":"Trust Enhancement Issues in Program Repair","cited_arxiv_id":"2108.13064","evidence_quote":"Empirical evidence that developers are unlikely to examine more than ten patches, motivating the fixed budget cap used in every experiment."},{"cited_title":"Rozière et al","cited_arxiv_id":null,"evidence_quote":"One of the three instruction-tuned LLMs studied; its 7B Instruct variant provides a base model and fine-tuning subject."},{"cited_title":"Dubey et al","cited_arxiv_id":null,"evidence_quote":"One of the three instruction-tuned LLMs studied; its 8B Instruct variant is the newest model and the basis for the Venn-diagram uniqueness analysis."}],"review_version":1}