{"id":"a40ff928-5444-4b34-b0f4-301e7d3bf056","arxiv_id":"2411.13323","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Some base LLMs, particularly codegen-multi, show strong memorization of the Defects4J bug benchmark, while newer models like LLaMa 3.1 show weaker leakage signals.","lead":"This study tests whether large language models have memorized popular bug-fixing benchmark datasets instead of learning general repair skills. It finds that some older models, especially codegen-multi, reproduce Defects4J solutions almost verbatim, while newer models show fewer signs of memorization.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The quantitative leakage signal depends on a 2024 GitHub control that is not distributionally matched to Defects4J files; without a matched control, the NLL and 5-gram gaps may reflect era or project-style differences rather than memorization.","rationale":"The reader identified the control-set comparability as the weakest assumption. I agree and argue it is the most load-bearing because the paper's central quantitative claim is a gap measured against this control. The paper has other supporting evidence (repository membership in TheStack, a verbatim example), but the aggregate NLL and 5-gram numbers are what make the claim about 'significant evidence of memorization' and 'consistently highest rate' across all models. The authors flag the limitation in Section V, but flagging a limitation does not resolve it; without a matched control, the observed gap is confounded by code vintage and project selection. The proposed within-repository control test would settle this by holding repository and commit constant. If the test passes, the central claim is substantially strengthened; if it fails, the quantitative evidence collapses and only the anecdotal verbatim example remains. Since the paper is otherwise methodical and the concern is addressable, the appropriate verdict is CONDITIONAL pending this validation.","tokens_in":15031,"tokens_out":15875,"duration_ms":182318,"concrete_test":"For each Defects4J fixed file in the evaluation, sample 5-10 control Java files from the same repository at the same commit or the closest preceding commit, excluding any file touched by the fix, and recompute NLL and 5-gram accuracy using the identical procedure (same strides, same seed, same model). Then compute the mean within-repository delta between fixed files and these controls. If the fixed files do not show a substantially lower NLL / higher 5-gram accuracy than their same-repository controls, the reported Defects4J signal is project familiarity rather than benchmark memorization. This test controls for vintage, project style, repository selection, and file-type distribution simultaneously.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section II-A constructs the 'new and unseen' control by sampling 250 files per language from 2024 GitHub repositories with >100 stars. All of the paper's headline quantitative evidence (e.g., codegen-multi NLL 0.15 on Defects4J vs 0.85 on new Java repos; 82% vs 48% 5-gram accuracy) is computed as a gap against this control. Section V explicitly states that the authors 'cannot guarantee that the new repositories are identically distributed compared to those in the benchmarks.' The control differs systematically from Defects4J fixed files in at least three ways: (1) vintage (2024 code vs 2014-era Java), (2) project maturity and style (highly starred Apache-era projects vs contemporary repos), and (3) file selection (curated bug-fix files vs random repository files). Any of these differences can lower NLL and raise n-gram accuracy on benchmark files for reasons unrelated to memorization. Because the central claim's quantitative support is the gap between Defects4J and this control, the claim is vulnerable if the control is not shown to be comparable. The within-benchmark comparisons (e.g., Defects4J vs GitBug-Java) are weaker because GitBug-Java also differs in project age and composition, so they cannot substitute for a matched control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether open-source base LLMs memorize bug benchmarks, focusing on Defects4J, BugsInPy, BugsC++, GitBug-Java, and SWEBench-Lite. The authors compute TheStack membership, negative log-likelihood (NLL), and 5-gram accuracy for nine models, and compare these benchmarks against a newly curated set of 2024 GitHub repositories presented as likely-unseen control data. They report that Defects4J consistently has the lowest NLL and highest 5-gram accuracy across all models, with codegen-multi reaching 82% 5-gram accuracy and a verbatim reproduction of a Defects4J fix; newer models such as Llama 3.1 show smaller gaps. The paper recommends using newer benchmarks and regular leakage monitoring.","tokens_in":15262,"tokens_out":4071,"duration_ms":45491,"significance":"The question is timely and important: if the results hold, they quantify a widely suspected threat to the validity of a large body of SE evaluations that rely on Defects4J and similar benchmarks. The study has clear strengths: it uses multiple independent indicators (membership, NLL, n-gram accuracy), evaluates a broad set of open models, constructs an external control corpus rather than relying only on internal comparisons, and explicitly discusses many threats to validity. The consistent ordering across all model families, combined with high TheStack membership and a verbatim reproduction example, makes the memorization signal for Defects4J genuinely suggestive. The main weakness is that the quantitative magnitude of the leakage signal depends on a control corpus that is not distributionally matched to the benchmark files, a limitation the authors themselves acknowledge in Section V.","major_comments":[{"comment":"The quantitative leakage signal is computed as a gap relative to the 2024 GitHub control corpus, but this control is not matched to the benchmark files. Section V states that the authors 'cannot guarantee that the new repositories are identically distributed compared to those in the benchmarks,' and the control differs in at least three systematic ways: vintage (2024 code versus 2014-era Java), project maturity (recent >100-star repositories versus long-established Apache-era projects), and file selection (randomly sampled files versus curated bug-fix files). The headline numbers, such as codegen-multi's NLL of 0.15 on Defects4J versus 0.85 on new Java repos, or 82% versus 48% 5-gram accuracy, are all measured against this control. These gaps could be inflated by genre, era, or boilerplate differences rather than memorization. I recommend either constructing a more matched control (e.g., non-buggy files from the same or contemporaneous projects, or pre-bug versions of the same files) or clearly re-labeling the reported quantities as relative differences whose interpretation is limited by this confound.","section":"Section II-A and Section V"},{"comment":"The key results are reported as single average values per model and dataset, with no uncertainty quantification. With roughly 250 files per dataset, the claim that Defects4J is 'consistently' the most memorized benchmark should be accompanied by standard errors, bootstrap confidence intervals, or per-file distributions. For example, Table VI reports codegen-multi 5-gram accuracy of 0.82 on Defects4J versus 0.55 on GitBug-Java; without variance estimates it is impossible to tell whether this 27-point gap is driven by the whole distribution or by a small number of verbatim-reproduced files. Adding confidence intervals or a per-file scatter plot would make the central comparison robust.","section":"Tables V, VI and Figures 3, 5"},{"comment":"The regression analysis is presented as evidence that 'models with more parameters tend to memorize more,' but it is based on only nine models from five families, with parameters and training budget as the only fixed predictors and dataset and tokenizer as random effects. The specification is not described in enough detail to assess convergence or identifiability, and the causal-sounding interpretation is not justified by an observational, small-N design. This analysis is secondary to the main claim, but the current phrasing overstates what can be concluded and should be softened or accompanied by a robustness check (e.g., family-level grouping, or a bootstrap over the 63 model-dataset observations).","section":"Section III-D and Table IV"}],"minor_comments":[{"comment":"The description of 5-gram accuracy computation says 'five uniformly distributed starting points per stride' but does not specify the stride length for the n-gram accuracy computation (as opposed to the 512-token stride mentioned for NLL) or how partial final strides are handled; please clarify so the metric is reproducible.","section":"Section II-C"},{"comment":"The LOC column reports values of 132 and 65 for New Java Repos and New Python Repos, while the text says 250 files were randomly sampled; please specify units (kLOC) and explain why the totals appear much smaller than the benchmark LOC values.","section":"Table I"},{"comment":"The heatmap color scales are not intuitive: the captions say darker colors correspond to higher ratios, but the numerical orientation (row versus column) is hard to read. A direct table or a labeled color bar would improve interpretability.","section":"Figure 4 and Figure 9"},{"comment":"The paper does not provide an artifact link or repository with the scripts, sampled file lists, and deduplication code. Given the paper's emphasis on leakage measurement, making these artifacts available would strengthen reproducibility.","section":"Throughout"},{"comment":"Minor typo: 'TheStack, a widely-used used pretraining code dataset' should read 'widely used pretraining code dataset.'","section":"Section I"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses an important and timely question, and the multi-metric design is a genuine contribution. The main issue is not internal circularity but the dependence of the headline quantitative magnitudes on a control corpus whose comparability the authors explicitly disclaim. Because the evidence also includes within-benchmark comparisons, TheStack membership, and a verbatim reproduction, I do not think the paper should be rejected; however, the control-corpus limitation is load-bearing for the stated effect sizes and for the practical guidance about which benchmarks to trust. A revised version that reports matched-control analyses or clearly reframes the claims as relative signals, and that adds uncertainty quantification, would be acceptable. I would also encourage the authors to consider adding an artifact link for the deduplication and evaluation scripts, as this is a reproducibility-oriented community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is the first systematic application of established leakage metrics (TheStack membership, NLL, 5-gram accuracy) to the bug benchmarks the SE community actually uses: Defects4J, BugsInPy, BugsC++, GitBug-Java, SWEBench-Lite. The headline finding, that Defects4J consistently looks most memorized across all nine base models, is well supported. The strongest single piece of evidence is the verbatim reproduction of a Defects4J patch by codegen-multi—comments and all—which no distribution-shift story can explain away. That alone justifies the paper's central caution about using Defects4J for LLM evaluation.\n\nThe paper does several things well. The methodology is clearly described and follows prior work rather than inventing ad hoc detectors. The authors include a purpose-built 2024 GitHub control, which is a reasonable attempt at unseen data. They also honestly hedge in Section V, acknowledging that the control may not be identically distributed to benchmark files. The membership analysis in TheStack adds independent weight: 80% of Defects4J repositories are present. The comparison across model families—older codegen-multi showing strong leakage signals while LLaMa 3.1 looks cleaner—is a useful practical data point.\n\nWhere are the soft spots? First, the stress-test concern about the 2024 control is real but not fatal. The NLL and 5-gram gaps between Defects4J and the new repos could partly reflect vintage or project-style differences, as the authors admit. But the within-benchmark comparison to GitBug-Java and the verbatim example reduce the chance that the entire signal is an artifact. Second, no code or data is released; for an audit paper, that is a genuine reproducibility gap. Third, the headline tables lack confidence intervals or variance information, so we cannot tell whether the small differences between, say, BugsInPy and GitBug-Java are meaningful. Fourth, the regression analysis interprets higher parameter count as leading to more memorization, but larger models also just predict code better in general; the coefficients are statistically significant but extremely small. Last, the paper sometimes says \"memorization\" where \"familiarity\" would be more precise—not a flaw, just an overreach in wording.\n\nNone of these sink the central claim. The paper deserves a serious referee: it is important for anyone evaluating LLMs on bug benchmarks and offers a reusable recipe. I would want the authors to release artifacts and add uncertainty quantification, but the core result stands. Recommend engaging with it, and I'd bring it to our reading group.","headline":"A solid first systematic leak audit of bug benchmarks; the Defects4J memorization evidence is convincing, though the 2024 control is imperfect and artifacts are missing.","tokens_in":15856,"tokens_out":1612,"would_cite":true,"duration_ms":19337,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Defects4J is memorized by open-source LLMs, so program-repair scores on it overstate real capability.","keywords":["data leakage","benchmark contamination","Defects4J","large language models","memorization","negative log-likelihood","n-gram accuracy","automated program repair"],"falsifier":"Take a strong open-weight model and inspect its training corpus directly: if the exact patched Defects4J files appear in the pretraining data with timestamps before the benchmark fixes were committed, the memorization account is confirmed. Conversely, if a newly pretrained model with Defects4J deliberately excluded still shows the same NLL and 5-gram gap on Defects4J relative to control repositories, the gap is not evidence of memorization.","tokens_in":14795,"feed_emoji":"🐛","tokens_out":4970,"duration_ms":46213,"temperature":0.7,"pith_summary":"Bug benchmarks used to score code-generating LLMs may be leaking into training data, and this paper quantifies that risk for widely used software-engineering bug datasets. The authors compare how open-source base models score on five established bug benchmarks (Defects4J, BugsInPy, BugsC++, GitBug-Java, SWEBench-Lite) against freshly mined 2024 GitHub repositories, using membership in a public pretraining corpus, negative log-likelihood, and 5-gram accuracy. They find that Defects4J stands out for every model as the most familiar benchmark, and that the older model codegen-multi reproduces Defects4J patched files at 82% 5-gram accuracy with near-zero surprise. The implication is that evaluations on Defects4J, and to a lesser extent older bug data, can overstate a model's real ability to fix unseen bugs. The authors recommend pairing legacy benchmarks with recent ones like GitBug-Java and monitoring contamination with these metrics.","feed_headline":"LLMs memorize Defects4J fixes, one model hits 82 percent","feed_subtitle":"Legacy bug benchmarks inflate fix rates: codegen-multi reproduces patched files near-verbatim, so scores are not pure skill.","key_machinery":"The detection machinery is a tripartite comparison. Membership checking asks whether each benchmark repository appears in publicly known pretraining corpora, specifically versions of TheStack. Negative log-likelihood (NLL) measures how surprised a model is by a file: unusually low NLL relative to control code indicates prior exposure. 5-gram accuracy measures whether the model, given preceding context, emits exact five-token runs from the reference; near-1.0 accuracy on patched files signals verbatim retention. The benchmark files in question are the fixed, patched files, and the control is a deduplicated sample of 2024 GitHub repositories with more than 100 stars.","core_discovery":"Defects4J, the most widely used bug benchmark in program repair, is measurably memorized by open-source base LLMs. Concretely, Defects4J has the lowest negative log-likelihood and the highest 5-gram accuracy of every benchmark tested across all nine models; codegen-multi achieves 82% 5-gram accuracy on Defects4J patched files with an NLL of 0.15, and in at least one case reproduced a full fix verbatim from a short prompt. Membership analysis shows 80% of Defects4J repositories appear in TheStack pretraining data. Newer models such as the LLaMa 3.1 family show smaller gaps between Defects4J and recent controls, consistent with lower memorization, but the paper argues that no model is completely clean. The larger point is that high scores on legacy bug benchmarks are not reliable evidence of generalization.","pith_inferences":["A natural next step the paper leaves implicit is to report NLL and 5-gram deltas alongside every leaderboard entry, so readers can see whether a model's high fix rate coincides with low surprise on the benchmark's patched files.","The same methodology could be applied to instruction-tuned and agentic systems by feeding prompts drawn from patched files and measuring token-level reproduction; the paper consciously restricts itself to base models.","If contamination is compounding, old benchmarks may become useless for model comparisons even between new models, since a model trained longer on the same public corpus may simply memorize more; the regression hints at this trajectory.","A practical extension would be to build leakage-controlled splits by taking recent bugs and withholding them from all training corpora, then measuring the gap between per-benchmark and per-split scores."],"forward_implications":["Scores on Defects4J should not be read as measuring a model's ability to repair unseen Java bugs; for older models a large part of the score may reflect memorized solutions.","Researchers should combine legacy benchmarks with recent, low-leakage benchmarks such as GitBug-Java before drawing conclusions about model capability.","The regression result suggests that larger models and models trained on more tokens tend to have higher n-gram accuracy and lower NLL, so simple 'bigger is better' evaluations will increasingly reward memorization unless benchmarks are refreshed.","SWEBench-Lite, though recently published, contains older bug fixes; codegen-multi shows low NLL on it, meaning recency of benchmark release does not guarantee absence of leakage.","Membership in a pretraining corpus alone does not prove memorization, but together with NLL and 5-gram evidence it forms a practical contamination signal."],"supporting_citations":[{"why":"Defects4J: the primary benchmark under study, providing the patched Java files whose memorization is measured.","marker":"[4]"},{"why":"Xu et al. 2024: supplies the NLL and n-gram accuracy methodology used for leakage detection.","marker":"[9]"},{"why":"Li 2023: establishes NLL-based contamination estimation, adapted here to bug benchmarks.","marker":"[10]"},{"why":"TheStack: the pretraining corpus checked for benchmark repository membership.","marker":"[28]"},{"why":"CodeGen: the model family found to show the strongest memorization of Defects4J.","marker":"[18]"},{"why":"Llama 3.1: the model family whose small benchmark-to-control gaps support the low-leakage contrast.","marker":"[20]"},{"why":"StarCoder 2 and The Stack v2: connect the membership dataset to a model that trained on it.","marker":"[21]"},{"why":"SWE-bench: recently published benchmark whose older underlying fixes explain unexpectedly low NLL in older models.","marker":"[6]"},{"why":"GitBug-Java: recent low-leakage Java benchmark used as the comparison point for benchmark selection.","marker":"[12]"}],"fun_headline_variants":["LLMs memorize bug benchmarks, scores inflated","Defects4J leaks into LLM training data, codegen-multi hits 82% exact","Bug benchmark Defects4J is memorized: codegen-multi reproduces fixes verbatim","Defects4J benchmark in training data overstates LLM abilities","Codegen-multi memorizes Defects4J fixes; LLaMa 3.1 looks cleaner"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The leakage signal is defined as the gap between benchmark files and freshly collected 2024 GitHub repositories; the authors concede they cannot guarantee the new repositories are distributionally identical to benchmark code, so the gap could partly reflect genre or boilerplate differences rather than memorization.","fun_headline_variants_meta":{"raw":{"variants":["LLMs memorize bug benchmarks, scores inflated","Defects4J leaks into LLM training data, codegen-multi hits 82% exact","Bug benchmark Defects4J is memorized: codegen-multi reproduces fixes verbatim","Defects4J benchmark in training data overstates LLM abilities","Codegen-multi memorizes Defects4J fixes; LLaMa 3.1 looks cleaner"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001241,"raw_usage":{"total_tokens":5090,"prompt_tokens":936,"completion_tokens":4154,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":4047}},"tokens_in":552,"tokens_out":4154,"duration_ms":29236,"temperature":1.0,"reasoning_tokens":4047,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:33:05.973762+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a strong open-weight model and inspect its training corpus directly: if the exact patched Defects4J files appear in the pretraining data with timestamps before the benchmark fixes were committed, the memorization account is confirmed. Conversely, if a newly pretrained model with Defects4J deliberately excluded still shows the same NLL and 5-gram gap on Defects4J relative to control repositories, the gap is not evidence of memorization.","supporting_citations":[{"cited_title":"Defects4j: a database of existing faults to enable controlled testing studies for java programs,","cited_arxiv_id":null,"evidence_quote":"Defects4J: the primary benchmark under study, providing the patched Java files whose memorization is measured."},{"cited_title":"Large language models for test-free fault localization,","cited_arxiv_id":null,"evidence_quote":"TheStack: the pretraining corpus checked for benchmark repository membership."},{"cited_title":"Codegen: An open large language model for code with multi-turn program synthesis,","cited_arxiv_id":null,"evidence_quote":"CodeGen: the model family found to show the strongest memorization of Defects4J."},{"cited_title":"SWE-bench: Can language models resolve real-world github issues?","cited_arxiv_id":null,"evidence_quote":"SWE-bench: recently published benchmark whose older underlying fixes explain unexpectedly low NLL in older models."},{"cited_title":"Gitbug-java: A reproducible benchmark of recent java bugs,","cited_arxiv_id":null,"evidence_quote":"GitBug-Java: recent low-leakage Java benchmark used as the comparison point for benchmark selection."}],"review_version":1}