{"id":"c2f29396-1a53-4103-8376-f118c580742b","arxiv_id":"2412.02789","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A replication study finds that a 7-billion-parameter open-source CodeLlama model, tuned with low temperature and specific prompts, can match ChatGPT on a code-refinement similarity metric while keeping code on local hardware.","lead":"This paper tests whether small open-source AI models, Llama 2 and CodeLlama, can handle code review refinement tasks about as well as the proprietary ChatGPT. It reports that a properly tuned CodeLlama reaches ChatGPT-level quality on one key metric, which could let companies automate code review without sending code to outside servers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Data leakage remains unchecked: if CR/CRN test samples overlap CodeLlama's pretraining data, the RQ2 BLEU-T equivalence with ChatGPT is inflated.","rationale":"I read the paper as a careful replication whose main empirical assertion is that a 7B open-source model, after prompt and temperature tuning, can produce code refinements comparable to ChatGPT as measured by BLEU-T on the CRN dataset. The most fragile condition for that assertion is that the test samples were unseen during pretraining. The paper itself identifies this in Section VII as a possible data-leakage threat and notes that no fresh dataset was used. Given that CRN is built from popular GitHub repositories and CodeLlama was trained on public code corpora, contamination is realistic rather than speculative. If contamination occurred, the reported BLEU-T and EM-T values in Table III would overstate CodeLlama's capability, and the RQ2 conclusion of equivalence would collapse. The metamorphic-transformation control is a concrete way to distinguish surface-form memorization from genuine refinement skill; it is computationally feasible with the authors' existing infrastructure and avoids requiring new labeled data. I agree with the reader's weakest-assumption identification, and because the issue is acknowledged and addressable while the rest of the study (RQ1 prompt/temperature analysis and RQ3 manual validation) is less affected, the conditional verdict remains appropriate.","tokens_in":16999,"tokens_out":6193,"duration_ms":69230,"concrete_test":"Apply a deterministic, semantics-preserving metamorphic transformation to the CRN test set: rename identifiers, reflow whitespace, and reformat code while preserving behavior and keeping the review comment unchanged. Rerun CodeLlama-7B-Instruct with the selected P5 prompt at temperature 0 on all 1,122 transformed samples and recompute EM-T and BLEU-T. If BLEU-T drops by more than 5 points or EM-T drops by more than 30% relative to Table III, the original scores are partly attributable to surface-form leakage, and the equivalence claim is not established. Stable scores would clear the model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CodeLlama achieves BLEU-T scores statistically equivalent to ChatGPT on CRN (Table III: 77.13 vs 76.44, marked as no significant difference). The load-bearing assumption, explicitly acknowledged in Section VII, is that the CR and CRN test splits were not in Llama 2's or CodeLlama's pretraining data. CRN is derived from popular GitHub repositories with samples added after January 2022, and CodeLlama was trained on large code corpora before its August 2023 release, so overlap is plausible. If the test samples, or their canonical refinements, appeared in training, CodeLlama's EM-T (13.73) and BLEU-T (77.13) could be inflated by memorization or distributional contamination rather than genuine code-refinement ability. The authors state they did not run a leakage check because this is a replication study. Since the paper's headline conclusion is precisely this equivalence, the unresolved contamination threat is the single most load-bearing weakness: it determines whether the main empirical claim survives.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper replicates Guo et al.'s empirical study of ChatGPT for automated code refinement, replacing the proprietary model with two open-source 7B models, CodeLlama and Llama 2, hosted locally in 4-bit quantized form via Ollama. Using the same CR and CRN datasets and the same seed/split configuration as the replicated study, the authors tune prompts and temperature on 250 sampled tasks (RQ1), then compare the best configurations against reported ChatGPT and CodeReviewer baselines on the full test sets of 2,106 tasks (RQ2), and break down performance by comment relevance, comment information, and change type on 400 annotated tasks (RQ3). They additionally report a preliminary Llama 3.1 experiment. The main claim is that, with proper prompting and temperature=0, CodeLlama achieves BLEU-T scores statistically equivalent to ChatGPT on CRN, and that both models perform better on refactoring/feature-modification tasks than on documentation-related tasks.","tokens_in":17121,"tokens_out":4959,"duration_ms":50868,"significance":"If the central equivalence holds, the paper provides evidence that a 7B open-source model can approach ChatGPT-3.5's n-gram similarity to canonical code refinements, with practical implications for privacy-preserving, low-cost local deployment. The study has notable strengths: it reuses the original replication package and seed, evaluates on the full test sets, uses deterministic temperature 0 for RQ2 and RQ3, applies non-parametric significance tests, and includes a manual validation of alternative solutions that partially corrects the strictness of EM-T. The release of scripts and datasets is also valuable for the community. However, the headline equivalence rests on two assumptions that are not fully verified: that the CR/CRN test splits were not part of the open models' pretraining data, and that the imported ChatGPT baseline numbers are directly comparable despite model-version drift. These weaknesses, while not invalidating the study, make the central claim less robust than the abstract suggests.","major_comments":[{"comment":"The central claim that CodeLlama achieves BLEU-T scores equivalent to ChatGPT on CRN depends on the assumption that the CRN test samples were not in CodeLlama's or Llama 2's pretraining data. The paper explicitly acknowledges this threat in Section VII but takes no further action because it is a replication study. Given that CRN is derived from popular GitHub repositories and that CodeLlama was trained on large code corpora, contamination is plausible enough to affect the reported EM-T (13.73) and BLEU-T (77.13) scores. The manuscript should at least include a lightweight leakage analysis (e.g., checking n-gram overlap between test inputs and known pretraining data, or running a small evaluation on a dataset created after the models' training cutoff) to bound this risk; without such evidence, the headline equivalence is not fully established.","section":"Section VII (Data Leakage) and RQ2 (Table III)"},{"comment":"The ChatGPT and CodeReviewer baselines are not re-measured in this study but are imported from Guo et al.'s published results. Because ChatGPT is a versioned, black-box service, the comparison is only valid if the exact model version, temperature, and prompt settings are identical to those used by Guo et al.; the paper does not report the ChatGPT version nor re-run the baseline with the current API. This is especially relevant because the paper itself cites ChatGPT's 'implicit versioning' as a motivation for open-source alternatives. The authors should either re-run the ChatGPT baseline or clearly state the version and date of the results being imported, and discuss how version drift could affect the equivalence claim.","section":"Section II.D and Table III"},{"comment":"The significance tests for BLEU-T comparisons use an unpaired Mann-Whitney U test, but the models are evaluated on the same test samples, so a paired test (e.g., Wilcoxon signed-rank) would be more appropriate and could change the conclusions. For instance, the ‡ mark on CRN BLEU-T for ChatGPT and CodeLlama is reported as 'no statistically significant difference,' but an unpaired test can miss within-sample correlation. The authors should either perform paired tests or justify the unpaired choice in the text.","section":"Section IV and Table III"},{"comment":"The Llama 2 and CodeLlama models are run in 4-bit quantized GGUF format via Ollama, which can degrade output quality compared to full-precision inference. The paper does not discuss the potential impact of quantization on the reported EM-T and BLEU-T scores, nor does it validate the quantized models against their full-precision versions on a subset. Since the claim is about the capabilities of the Llama models themselves, the authors should acknowledge this limitation and, if feasible, provide a small-scale comparison with full precision.","section":"Section II.I and Section IV"},{"comment":"In RQ1, the best prompt for CodeLlama is selected as P5 based on BLEU-T, even though P4 yields a higher EM-T (15.2 vs 13.2 at temperature 0). The paper justifies this by saying BLEU-T is the more reliable indicator, but the choice directly affects the RQ2 results where EM-T is also reported. If EM-T is considered a key metric for exact matches, selecting P5 may understate CodeLlama's EM-T in the main comparison. The authors should either justify the selection criterion more rigorously or report RQ2 results for both P4 and P5 for completeness.","section":"Section III (RQ1)"}],"minor_comments":[{"comment":"There are several typos and stylistic issues, e.g., 'can stale the development' should be 'can slow down the development,' and 'complicity' in the Related Work section appears to be a typo for 'complexity.'","section":"Abstract and Introduction"},{"comment":"The description of the CRN dataset states that it adds 5,451 samples from 240 new repositories, but the relationship between the total 9,117 samples and the 5,451 additions is not fully explained; clarifying the arithmetic would help readers assess dataset composition.","section":"Section II.C"},{"comment":"The note under Table II says 'All BLEU/BLEU-T results are statistically different vs the best prompt,' but no significance test results are shown in the table or the surrounding text; please provide the test statistics or a reference to supplementary material.","section":"Table II"},{"comment":"The Llama 3.1 experiment is presented without prompt tuning for that model, and the claim that 'our statistical analysis shows no significant difference' should specify which model pairs were tested and report the corresponding p-values or test statistics.","section":"Section VI.D and Table VII"},{"comment":"Reference [24], the replication package URL, contains a space ('code refinement llama') and appears to be malformed; please provide a working URL.","section":"References"},{"comment":"The manual validation of alternative solutions is a valuable addition, but the paper does not describe whether more than one annotator participated or how disagreements were resolved; adding this information would strengthen the validity of the classification.","section":"Section VI.A"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid replication study with a useful practical message, but the central equivalence claim on CRN BLEU-T is vulnerable to data leakage and baseline version drift. I would encourage the editor to request the authors to address the leakage concern with a quantitative overlap analysis or a fresh-dataset sanity check, and to re-run or clearly qualify the ChatGPT baseline. These are fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a clean, honest replication that gives the first real numbers for CodeLlama and Llama 2 on the CR/CRN code-refinement benchmarks. The headline result is narrower than the abstract suggests, but the body mostly earns it: CodeLlama gets BLEU-T scores statistically indistinguishable from ChatGPT on CRN, while clearly lagging on exact match. That is a genuinely new empirical datapoint for teams wondering whether a local 7B model is good enough for this task.\n\nWhat the paper does well: it reuses the same seed and train/validation/test splits as Guo et al., which makes the comparison direct. The RQ1 prompt/temperature sweep is useful and gives a clear practical recipe (temperature 0, prompts P4/P5). The Llama 3.1 add-on is a nice extra, and the manual check of alternative solutions is a thoughtful attempt to interpret the EM-T gap. Running everything on consumer hardware with 4-bit quantization makes the deployment claim concrete.\n\nThe soft spots are real but not fatal. Data leakage is the load-bearing worry: CRN is derived from popular GitHub repos and CodeLlama was pretrained on code before its release. The authors acknowledge this in Section VII and chose not to run a leakage check because this is a replication. I think a reviewer should press on this, but it is not disqualifying. If CodeLlama were memorizing exact refinements, its EM-T would be much higher than 13.7. Distributional overlap could still inflate BLEU-T, so the \"equivalent BLEU-T\" claim needs a bit more support.\n\nSecond, the ChatGPT and CodeReviewer baselines are imported from the replicated study, meaning version drift is uncontrolled. That is acceptable in a replication, but it weakens the comparison. Third, the replication package link in reference [24] is malformed (space in the URL), and RQ3 has some very small cells (n=6, 8) that the authors themselves flag. These are minor, fixable issues.\n\nThe abstract's \"often comparable\" overstates things a little; the data support \"comparable on one similarity metric on one dataset, plus some task categories.\" But the paper is measured and transparent, and the limitations are stated upfront. I would send this to peer review. The main thing I would ask the authors to add is a leakage check on a fresh dataset or a metamorphic transformation, even if it means going beyond the replication protocol.","headline":"Solid, honest replication: CodeLlama matches ChatGPT on BLEU-T for CRN, though leakage and imported baselines deserve scrutiny.","tokens_in":17740,"tokens_out":2357,"would_cite":true,"duration_ms":26953,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CodeLlama, a 7B open-source model, matches ChatGPT's BLEU-T score on CRN code-refinement tasks when tuned with temperature 0 and concise prompts.","keywords":["code refinement","code review automation","CodeLlama","Llama 2","ChatGPT comparison","BLEU-T","replication study","prompt engineering"],"falsifier":"Run the identical tuned configurations (temperature 0, P4/P5 prompts) on a freshly collected set of code-refinement tasks from repositories whose commits postdate the models' training cutoff, or on metamorphic variants of the existing test samples that change syntax but not semantics, and compare EM-T and BLEU-T to the values in Table III; a substantial drop would confirm data leakage and invalidate the ChatGPT comparison.","tokens_in":16754,"feed_emoji":"🤖","tokens_out":5584,"duration_ms":52259,"temperature":0.7,"pith_summary":"This replication study claims that a properly tuned 7-billion-parameter open-source model, CodeLlama, can perform automated code refinement at a quality comparable to ChatGPT, measured by n-gram similarity to the canonical refined code. The authors rerun the methodology of Guo et al.'s ChatGPT study on two datasets of code-review tasks, using Llama 2 and CodeLlama hosted locally in 4-bit quantized form on consumer machines. They find that temperature 0 with prompts that combine concise requirements and a scenario description gives the best results, and that on the CRN dataset CodeLlama's BLEU-T score is statistically indistinguishable from ChatGPT's, while ChatGPT still finds exact-match solutions more often. They also find that both models do well on tasks that modify existing code, such as refactoring and feature changes, and poorly on tasks that add new documentation or mix code and documentation changes. If correct, the result matters because it suggests that code-review assistance can be delivered locally, at low cost, without sending proprietary code to a third-party API.","feed_headline":"Open-source CodeLlama ties ChatGPT on code refinement","feed_subtitle":"A 7B model running locally reaches ChatGPT-level BLEU-T scores on review fixes, with prompt tuning at temperature 0.","key_machinery":"The load-bearing machinery is the definition of a code refinement task as a mapping $D: (C_1 + R) \\to C_2$, where $C_1$ is the initial code, $R$ is the review comment, and $C_2$ is the refined code, together with the trimmed metrics EM-T and BLEU-T introduced in the replicated study. BLEU-T locates the core code in a model's output by matching the first and last lines of the ground truth, so verbose or explanatory replies from a model do not unfairly depress the score. On top of this, the paper uses five prompt templates (basic, scenario description, detailed requirements, concise requirements, and concise requirements plus scenario description) and temperatures 0, 0.5, and 1, with temperature 0 and the concise-requirement prompts (P4 for Llama 2, P5 for CodeLlama) selected as best. The models are hosted locally through Ollama in 4-bit quantized GGUF format, which is the enabler for running a 7B parameter model on ordinary laptops.","core_discovery":"On its own terms, the paper's central discovery is that scale and closed-source status are not required for useful code refinement: CodeLlama-7B, a code-specialized model, produces refined code that is token-wise as close to the dataset's canonical solution as ChatGPT 3.5 produces, on the CRN dataset, with no statistically significant difference in BLEU-T under a Mann-Whitney U test at 95% confidence. ChatGPT retains a clear edge in exact-match (EM-T) scores, and the paper attributes this to ChatGPT's much larger parameter count and better ability to infer the original developer's writing style. A manual check of 39 tasks where one model found the canonical answer and the other did not found that 14 of 29 CodeLlama alternatives and 4 of 10 ChatGPT alternatives were semantically valid, showing that exact-match failures often reflect superficial formatting differences rather than wrong code. The paper also establishes that task type and comment quality dominate model choice: concrete suggestions and changes to existing code are far easier for all models than vague comments or additions of new documentation.","pith_inferences":["An immediate testable extension would be to run the same tuned models on repositories updated after the models' training cutoff; a sharp performance drop would indicate that the reported scores partly reflect memorization rather than a reusable refinement skill.","The manual validity analysis hints that a semantics-aware evaluation, for example compiling or running unit tests on the generated code, could shrink the apparent gap between ChatGPT and CodeLlama further than the EM-T metric suggests.","The finding that documentation tasks are nearly impossible for all tested models suggests a targeted opportunity: fine-tuning or prompt design specifically for doc-and-code changes may yield larger gains than pushing general-purpose performance."],"forward_implications":["If the central claim is correct, teams can deploy a 7B open-source model on local hardware and receive code-refinement suggestions at ChatGPT-comparable n-gram quality without sending code to an external API.","Exact-match parity is not achieved, so teams that require identical canonical solutions will still need larger models or additional fine-tuning.","Code-centric fine-tuned models should be preferred over general-purpose small models for code-related review automation.","Prompts that specify concise requirements and a scenario description, run at temperature 0, are the configuration a practitioner should adopt.","Evaluation on code-refinement tasks should be stratified by change type, because aggregate scores conceal large gaps between refactoring and documentation tasks."],"supporting_citations":[{"why":"The replicated study; supplies the methodology, prompt templates, dataset split seed, ChatGPT baseline results, and the EM-T/BLEU-T trimming code.","marker":"[5]"},{"why":"Defines the code refinement task (C1 + R) to C2, provides the CR dataset, and supplies the CodeReviewer baseline.","marker":"[17]"},{"why":"The Llama 2 7B model used as one of the two open-source models under evaluation.","marker":"[22]"},{"why":"The CodeLlama-Instruct 7B model used as the code-specialized open-source model.","marker":"[23]"},{"why":"Provides the taxonomy of code-change categories used in RQ3 to break down performance by task type.","marker":"[32]"},{"why":"The Mann-Whitney U test used to claim that CodeLlama's BLEU-T score is statistically indistinguishable from ChatGPT's on CRN.","marker":"[34]"}],"fun_headline_variants":["CodeLlama matches ChatGPT on code refinement scores","Open-source CodeLlama ties ChatGPT at code fixes","Small open-source model rivals ChatGPT in code review","Local 7B Llama matches ChatGPT for code refinement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim stands or falls on the assumption that the Llama models had not already seen the CR and CRN code-refinement examples during their own training, because the models' training window overlaps the datasets' creation; if the tests are memorized, the reported scores overstate genuine code-refinement ability.","fun_headline_variants_meta":{"raw":{"variants":["CodeLlama matches ChatGPT on code refinement scores","Open-source CodeLlama ties ChatGPT at code fixes","Small open-source model rivals ChatGPT in code review","Local 7B Llama matches ChatGPT for code refinement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000568,"raw_usage":{"total_tokens":2724,"prompt_tokens":1017,"completion_tokens":1707,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":1645}},"tokens_in":633,"tokens_out":1707,"duration_ms":11544,"temperature":1.0,"reasoning_tokens":1645,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:06:16.437737+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical tuned configurations (temperature 0, P4/P5 prompts) on a freshly collected set of code-refinement tasks from repositories whose commits postdate the models' training cutoff, or on metamorphic variants of the existing test samples that change syntax but not semantics, and compare EM-T and BLEU-T to the values in Table III; a substantial drop would confirm data leakage and invalidate the ChatGPT comparison.","supporting_citations":[],"review_version":1}