Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Are Large Language Models Memorizing Bug Benchmarks?

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Defects4J is memorized by open-source LLMs, so program-repair scores on it overstate real capability.

desk verdict A solid first systematic leak audit of bug benchmarks; the Defects4J memorization evidence is convincing, though the 2024 control is imperfect and artifacts are missing. read the letter →

arxiv 2411.13323 v3 pith:QVS4Q726 submitted 2024-11-20 cs.SE cs.AIcs.LG

Daniel Ramos , Claudia Mamede , Kush Jain , Paulo Canelas , Catarina Gamboa , Claire Le Goues This is my paper · ORCID
classification cs.SEcs.AIcs.LG
keywords dataleakagebenchmarkcontaminationDefects4Jlargelanguagemodelsmemorizationnegativelog-likelihoodn-gramaccuracyautomatedprogramrepair
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Bug benchmarks used to score code-generating LLMs may be leaking into training data, and this paper quantifies that risk for widely used software-engineering bug datasets. The authors compare how open-source base models score on five established bug benchmarks (Defects4J, BugsInPy, BugsC++, GitBug-Java, SWEBench-Lite) against freshly mined 2024 GitHub repositories, using membership in a public pretraining corpus, negative log-likelihood, and 5-gram accuracy. They find that Defects4J stands out for every model as the most familiar benchmark, and that the older model codegen-multi reproduces Defects4J patched files at 82% 5-gram accuracy with near-zero surprise. The implication is that evaluations on Defects4J, and to a lesser extent older bug data, can overstate a model's real ability to fix unseen bugs. The authors recommend pairing legacy benchmarks with recent ones like GitBug-Java and monitoring contamination with these metrics.

What carries the argument

The detection machinery is a tripartite comparison. Membership checking asks whether each benchmark repository appears in publicly known pretraining corpora, specifically versions of TheStack. Negative log-likelihood (NLL) measures how surprised a model is by a file: unusually low NLL relative to control code indicates prior exposure. 5-gram accuracy measures whether the model, given preceding context, emits exact five-token runs from the reference; near-1.0 accuracy on patched files signals verbatim retention. The benchmark files in question are the fixed, patched files, and the control is a deduplicated sample of 2024 GitHub repositories with more than 100 stars.

What would settle it

Take a strong open-weight model and inspect its training corpus directly: if the exact patched Defects4J files appear in the pretraining data with timestamps before the benchmark fixes were committed, the memorization account is confirmed. Conversely, if a newly pretrained model with Defects4J deliberately excluded still shows the same NLL and 5-gram gap on Defects4J relative to control repositories, the gap is not evidence of memorization.

Watch

Extended reading notes

Core claim

Defects4J, the most widely used bug benchmark in program repair, is measurably memorized by open-source base LLMs. Concretely, Defects4J has the lowest negative log-likelihood and the highest 5-gram accuracy of every benchmark tested across all nine models; codegen-multi achieves 82% 5-gram accuracy on Defects4J patched files with an NLL of 0.15, and in at least one case reproduced a full fix verbatim from a short prompt. Membership analysis shows 80% of Defects4J repositories appear in TheStack pretraining data. Newer models such as the LLaMa 3.1 family show smaller gaps between Defects4J and recent controls, consistent with lower memorization, but the paper argues that no model is completely clean. The larger point is that high scores on legacy bug benchmarks are not reliable evidence of generalization.

Load-bearing premise

The leakage signal is defined as the gap between benchmark files and freshly collected 2024 GitHub repositories; the authors concede they cannot guarantee the new repositories are distributionally identical to benchmark code, so the gap could partly reflect genre or boilerplate differences rather than memorization.

Editorial extensions

If this is right

  • Scores on Defects4J should not be read as measuring a model's ability to repair unseen Java bugs; for older models a large part of the score may reflect memorized solutions.
  • Researchers should combine legacy benchmarks with recent, low-leakage benchmarks such as GitBug-Java before drawing conclusions about model capability.
  • The regression result suggests that larger models and models trained on more tokens tend to have higher n-gram accuracy and lower NLL, so simple 'bigger is better' evaluations will increasingly reward memorization unless benchmarks are refreshed.
  • SWEBench-Lite, though recently published, contains older bug fixes; codegen-multi shows low NLL on it, meaning recency of benchmark release does not guarantee absence of leakage.
  • Membership in a pretraining corpus alone does not prove memorization, but together with NLL and 5-gram evidence it forms a practical contamination signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the paper leaves implicit is to report NLL and 5-gram deltas alongside every leaderboard entry, so readers can see whether a model's high fix rate coincides with low surprise on the benchmark's patched files.
  • The same methodology could be applied to instruction-tuned and agentic systems by feeding prompts drawn from patched files and measuring token-level reproduction; the paper consciously restricts itself to base models.
  • If contamination is compounding, old benchmarks may become useless for model comparisons even between new models, since a model trained longer on the same public corpus may simply memorize more; the regression hints at this trajectory.
  • A practical extension would be to build leakage-controlled splits by taking recent bugs and withholding them from all training corpora, then measuring the gap between per-benchmark and per-split scores.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates whether open-source base LLMs memorize bug benchmarks, focusing on Defects4J, BugsInPy, BugsC++, GitBug-Java, and SWEBench-Lite. The authors compute TheStack membership, negative log-likelihood (NLL), and 5-gram accuracy for nine models, and compare these benchmarks against a newly curated set of 2024 GitHub repositories presented as likely-unseen control data. They report that Defects4J consistently has the lowest NLL and highest 5-gram accuracy across all models, with codegen-multi reaching 82% 5-gram accuracy and a verbatim reproduction of a Defects4J fix; newer models such as Llama 3.1 show smaller gaps. The paper recommends using newer benchmarks and regular leakage monitoring.

Significance. The question is timely and important: if the results hold, they quantify a widely suspected threat to the validity of a large body of SE evaluations that rely on Defects4J and similar benchmarks. The study has clear strengths: it uses multiple independent indicators (membership, NLL, n-gram accuracy), evaluates a broad set of open models, constructs an external control corpus rather than relying only on internal comparisons, and explicitly discusses many threats to validity. The consistent ordering across all model families, combined with high TheStack membership and a verbatim reproduction example, makes the memorization signal for Defects4J genuinely suggestive. The main weakness is that the quantitative magnitude of the leakage signal depends on a control corpus that is not distributionally matched to the benchmark files, a limitation the authors themselves acknowledge in Section V.

major comments (3)
  1. [Section II-A and Section V] The quantitative leakage signal is computed as a gap relative to the 2024 GitHub control corpus, but this control is not matched to the benchmark files. Section V states that the authors 'cannot guarantee that the new repositories are identically distributed compared to those in the benchmarks,' and the control differs in at least three systematic ways: vintage (2024 code versus 2014-era Java), project maturity (recent >100-star repositories versus long-established Apache-era projects), and file selection (randomly sampled files versus curated bug-fix files). The headline numbers, such as codegen-multi's NLL of 0.15 on Defects4J versus 0.85 on new Java repos, or 82% versus 48% 5-gram accuracy, are all measured against this control. These gaps could be inflated by genre, era, or boilerplate differences rather than memorization. I recommend either constructing a more matched control (e.g., non-buggy files from the same or contemporaneous projects, or pre-bug versions of the same files) or clearly re-labeling the reported quantities as relative differences whose interpretation is limited by this confound.
  2. [Tables V, VI and Figures 3, 5] The key results are reported as single average values per model and dataset, with no uncertainty quantification. With roughly 250 files per dataset, the claim that Defects4J is 'consistently' the most memorized benchmark should be accompanied by standard errors, bootstrap confidence intervals, or per-file distributions. For example, Table VI reports codegen-multi 5-gram accuracy of 0.82 on Defects4J versus 0.55 on GitBug-Java; without variance estimates it is impossible to tell whether this 27-point gap is driven by the whole distribution or by a small number of verbatim-reproduced files. Adding confidence intervals or a per-file scatter plot would make the central comparison robust.
  3. [Section III-D and Table IV] The regression analysis is presented as evidence that 'models with more parameters tend to memorize more,' but it is based on only nine models from five families, with parameters and training budget as the only fixed predictors and dataset and tokenizer as random effects. The specification is not described in enough detail to assess convergence or identifiability, and the causal-sounding interpretation is not justified by an observational, small-N design. This analysis is secondary to the main claim, but the current phrasing overstates what can be concluded and should be softened or accompanied by a robustness check (e.g., family-level grouping, or a bootstrap over the 63 model-dataset observations).
minor comments (5)
  1. [Section II-C] The description of 5-gram accuracy computation says 'five uniformly distributed starting points per stride' but does not specify the stride length for the n-gram accuracy computation (as opposed to the 512-token stride mentioned for NLL) or how partial final strides are handled; please clarify so the metric is reproducible.
  2. [Table I] The LOC column reports values of 132 and 65 for New Java Repos and New Python Repos, while the text says 250 files were randomly sampled; please specify units (kLOC) and explain why the totals appear much smaller than the benchmark LOC values.
  3. [Figure 4 and Figure 9] The heatmap color scales are not intuitive: the captions say darker colors correspond to higher ratios, but the numerical orientation (row versus column) is hard to read. A direct table or a labeled color bar would improve interpretability.
  4. [Throughout] The paper does not provide an artifact link or repository with the scripts, sampled file lists, and deduplication code. Given the paper's emphasis on leakage measurement, making these artifacts available would strengthen reproducibility.
  5. [Section I] Minor typo: 'TheStack, a widely-used used pretraining code dataset' should read 'widely used pretraining code dataset.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the leakage signal is computed from external controls and metrics defined in prior work, with no parameter fitted to the conclusion.

full rationale

The paper's analysis chain is empirical and self-contained against external baselines. Benchmark files are compared to a separately mined control corpus of 2024 GitHub repositories; membership is measured against TheStack using an external tool; NLL and 5-gram accuracy are computed via reference implementations and definitions taken from prior work [9], [10]. No equation defines leakage in terms of the paper's own conclusion, and no fitted parameter is subsequently renamed as a prediction. The regression in Section III-D uses model parameters and training budget as predictors of NLL and 5-gram accuracy, which are measured quantities rather than constructed outputs, so it does not reduce to its inputs. The paper explicitly concedes in Section V that it 'cannot guarantee that the new repositories are identically distributed compared to those in the benchmarks'; this is a validity threat about confounding, not a circular step, because the control mismatch does not make the benchmark scores equal to the paper's assumptions by construction. The only self-citations are contextual references to prior APR and fault-localization work (e.g., [3], [25]), and they are not load-bearing for the leakage claim. No self-citation chain, imported uniqueness theorem, or ansatz-via-citation pattern appears. Accordingly, the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no invented entities and no fitted physical parameters. The central comparisons rest on hand-chosen methodological thresholds (n-gram order, stride, dedup overlap, star filter, sample size) and on assumptions that membership, NLL, and 5-gram accuracy are valid leakage indicators and that the 2024 control repos are unseen and comparable. These are reasonable but imperfectly validated assumptions.

free parameters (5)
  • n-gram order = 5
    Hand-chosen from prior work (Xu et al., 2024) as a balance between compute and accuracy; reported 5-gram accuracy values depend directly on this choice.
  • stride length = 512 tokens
    Used when a file exceeds a model's context window; affects how chunks overlap and therefore the NLL and n-gram estimates.
  • dedup overlap threshold = 85%
    Files with more than 85% overlap are removed to avoid duplicate-file bias; the threshold is arbitrary and changes the unique sample set.
  • minimum GitHub stars = 100
    Proxy for repository quality and community interest in the 2024 control set; determines the size and distribution of the 'unseen' comparison corpus.
  • random sample size = 250 files per language
    Computational cost control for evaluating NLL and 5-gram accuracy; sample size affects the variance of the reported averages.
assumptions (4)
  • domain assumption Ground-truth fixed (patched) files are representative of the code a model would be evaluated on and are the right unit for detecting benchmark memorization.
    Section II-A states 'We collected only fixed files since these correspond to the solutions a model may have memorized'; this ignores the possibility that only buggy versions were seen, which the authors partially address in Section V.
  • domain assumption Membership in TheStack is a meaningful proxy for training-data exposure.
    Section II-C uses the Am I in the Stack tool to infer leakage risk, while noting that membership does not guarantee a specific model trained on it; only some evaluated models report training on TheStack.
  • domain assumption The 2024 GitHub repositories are both unseen by the models and distributionally comparable to the benchmark files.
    Section II-A applies MinHash filtering and a >100-star threshold to construct a control set; Section V concedes 'we cannot guarantee that the new repositories are identically distributed compared to those in the benchmarks.'
  • domain assumption Lower NLL and higher 5-gram accuracy relative to control data indicate memorization rather than common-code idioms or tokenizer/formatting artifacts.
    Section III-C interprets 5-gram gaps as memorization signals while acknowledging that common coding patterns and formatting (Figure 6b) can suppress or inflate n-gram matches; the assumption is partially tested by the new-repo control but not fully controlled.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are Large Language Models Memorizing Bug Benchmarks?." pith.science (2026). https://pith.science/paper/QVS4Q726

@misc{pith2026241113323,
  author       = {Pith},
  title        = {Pith review of: Are Large Language Models Memorizing Bug Benchmarks?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QVS4Q726}},
  note         = {Machine review of arXiv:2411.13323}
}
read the original abstract

Large Language Models (LLMs) have become integral to various software engineering tasks, including code generation, bug detection, and repair. To evaluate model performance in these domains, numerous bug benchmarks containing real-world bugs from software projects have been developed. However, a growing concern within the software engineering community is that these benchmarks may not reliably reflect true LLM performance due to the risk of data leakage. Despite this concern, limited research has been conducted to quantify the impact of potential leakage. In this paper, we systematically evaluate popular LLMs to assess their susceptibility to data leakage from widely used bug benchmarks. To identify potential leakage, we use multiple metrics, including a study of benchmark membership within commonly used training datasets, as well as analyses of negative log-likelihood and n-gram accuracy. Our findings show that certain models, in particular codegen-multi, exhibit significant evidence of memorization in widely used benchmarks like Defects4J, while newer models trained on larger datasets like LLaMa 3.1 exhibit limited signs of leakage. These results highlight the need for careful benchmark selection and the adoption of robust metrics to adequately assess models capabilities.

Figures

Figures reproduced from arXiv: 2411.13323 by the authors.

Figure 1
Figure 1. Excerpt from Defects4J (Lang:Bug 39). Given the first lines of the function until line 11, codegen-multi generated lines 12 to 23, matching the benchmark solution. bug benchmarks and raise awareness of the risks associated with using established benchmarks, which may inadvertently inflate performance by testing memorized data. We use multiple metrics to detect potential leakage. First, we investigate whether benchma… view at source ↗
Figure 2
Figure 2. Overview of our methodology for detecting leakage. We collected bug benchmarks and unseen repositories from 2024. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. NLL by model and dataset. NLL is not comparable across models in different families, only across benchmarks within a family. NLL for other models are consistent with the results displayed. TABLE III: Percentage of repositories in each benchmark leaked in TheStack versions 1.0, 2.0 and 2.1. Benchmark v1.0 (%) v2.0 (%) v2.1 (%) GitBug-Java 61.1 42.6 38.9 BugsInPy 94.1 64.7 64.7 BugsC++ 60.9 60.9 65.2 Defects4J 80.0 80… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Heatmap illustrating the relative NLL ratios across datasets for the codegen-multi, LLaMa 3.1, and Gemma 2. Each cell represents the ratio of the NLL for the dataset in the column to that of the dataset in the row. For example, the NLL for new Java repos is 5.63× highe…
Figure 5
Figure 5. Figure 5: 5-gram accuracy by model and dataset. Due to space constraints, we selected a sample of the most relevant models. 5-gram accuracy for other models are consistent with the results displayed. TABLE IV: Summary of regressions tests for negative log like￾lihood (NLL) and 5…
Figure 6
Figure 6. Figure 6: Example patch of a bug from BugsInPy (fastapi:Bug 12). We prompt each model with the 30 lines prior to the patch. We highlight the extra code generated not in the ground truth and ground truth code not generated by the model. The remaining lines of the examples are omi…
Figure 7
Figure 7. Figure 7: NLL results for all the models and benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: 5-gram results for all the models and benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Heatmaps for NLL ratios 0.40 0.45 0.50 0.55 0.60 0.65 −0.15 −0.05 0.00 0.05 0.10 0.15 Residuals vs. Fitted (5−gram accuracy) Fitted Values Residuals (a) Residual vs. Fitted Plot for average 5-gram accuracy. 0.4 0.5 0.6 0.7 0.8 0.9 1.0 −0.2 −0.1 0.0 0.1 0.2 Residuals vs…
Figure 10
Figure 10. Figure 10: Residuals for the linear regressions [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CveBinarySheet: A Comprehensive Pre-built Binaries Database for IoT Vulnerability Analysis

    cs.CR 2025-01 reject novelty 5.0 of 10

    A dataset paper that proposes prebuilt vulnerable binaries for 1,033 CVEs across 16 components and 5 architectures, but supplies no way to access or verify the data.

Reference graph

Works this paper leans on

45 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [2]

    A survey on software fault localization,

    W. E. Wong et al. , “A survey on software fault localization,” IEEE Transactions on Software Engineering, vol. 42, no. 8, pp. 707–740, 2016

    work page 2016

  2. [3]

    Automated Program Repair,

    C. Le Goues, M. Pradel, and A. Roychoudhury, “Automated Program Repair,” Communications of the ACM , vol. 62, no. 12, pp. 56–65, 2019

    work page 2019

  3. [4]

    Defects4j: a database of existing faults to enable controlled testing studies for java programs,

    R. Just, D. Jalali, and M. D. Ernst, “Defects4j: a database of existing faults to enable controlled testing studies for java programs,” in Interna- tional Symposium on Software Testing and Analysis, 2014, pp. 437–440

    work page 2014

  4. [5]

    Bugsinpy: a database of existing bugs in python programs to enable controlled testing and debugging studies,

    R. Widyasari et al. , “Bugsinpy: a database of existing bugs in python programs to enable controlled testing and debugging studies,” in Foun- dations of Software Engineering , 2020

    work page 2020

  5. [6]

    SWE-bench: Can language models resolve real-world github issues?

    C. E. Jimenez et al. , “SWE-bench: Can language models resolve real-world github issues?” in International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/ forum?id=VTF8yNQM66

    work page 2024

  6. [7]

    Concerned with data contamination? assessing countermeasures in code language model,

    J. Cao, W. Zhang, and S. Cheung, “Concerned with data contamination? assessing countermeasures in code language model,” CoRR, vol. abs/2403.16898, 2024. [Online]. Available: https://doi.org/10.48550/ arXiv.2403.16898

    arXiv doi:10.48550/arxiv.2403.16898 2024

  7. [8]

    Leakage and the reproducibility crisis in ml-based science,

    S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in ml-based science,” CoRR, vol. abs/2207.07048, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2207.07048

    arXiv doi:10.48550/arxiv.2207.07048 2022

  8. [9]

    Benchmarking benchmark leakage in large language models,

    R. Xu et al. , “Benchmarking benchmark leakage in large language models,” CoRR, vol. abs/2404.18824, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2404.18824

    arXiv doi:10.48550/arxiv.2404.18824 2024

Show all 45 references
  1. [10]

    Estimating contamination via perplexity: Quantifying memo- risation in language model evaluation,

    Y . Li, “Estimating contamination via perplexity: Quantifying memo- risation in language model evaluation,” CoRR, vol. abs/2309.10677,

    arXiv

  2. [11]

    Bugsc++: A highly usable real world defect benchmark for c/c++,

    G. An et al., “Bugsc++: A highly usable real world defect benchmark for c/c++,” in International Conference on Automated Software Engineering. IEEE Press, 2024, p. 2034–2037. [Online]. Available: https://doi.org/10.1109/ASE56229.2023.00208

    2024

  3. [12]

    Gitbug-java: A reproducible benchmark of recent java bugs,

    A. Silva, N. Saavedra, and M. Monperrus, “Gitbug-java: A reproducible benchmark of recent java bugs,” in International Conference on Mining Software Repositories, ser. MSR ’24. ACM, 2024, p. 118–122

    2024

  4. [13]

    Agentless: Demystifying llm-based software engineering agents,

    C. S. Xia et al. , “Agentless: Demystifying llm-based software engineering agents,” CoRR, vol. abs/2407.01489, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2407.01489

    2024 doi arXiv

  5. [14]

    Openhands: An open platform for ai software developers as generalist agents,

    X. Wang et al. , “Openhands: An open platform for ai software developers as generalist agents,” 2024. [Online]. Available: https: //arxiv.org/abs/2407.16741

    2024 arXiv

  6. [15]

    Swe-bench+: Enhanced coding benchmark for llms,

    R. Aleithan et al. , “Swe-bench+: Enhanced coding benchmark for llms,” 2024. [Online]. Available: https://arxiv.org/abs/2410.06992

    2024 arXiv

  7. [16]

    On the resemblance and containment of documents,

    A. Broder, “On the resemblance and containment of documents,” in Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171), 1997, pp. 21–29

    1997

  8. [17]

    Similarity search in high dimensions via hashing,

    A. Gionis, P. Indyk, and R. Motwani, “Similarity search in high dimensions via hashing,” in International Conference on Very Large Data Bases, ser. VLDB ’99, 1999, p. 518–529

    1999

  9. [18]

    Codegen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp et al. , “Codegen: An open large language model for code with multi-turn program synthesis,” in International Conference on Learning Representations. OpenReview.net, 2023

    2023

  10. [19]

    Code llama: Open foundation models for code,

    B. Rozière et al. , “Code llama: Open foundation models for code,”

  11. [20]

    Llama: Open and efficient foundation language models,

    H. Touvron et al. , “Llama: Open and efficient foundation language models,” 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2302. 13971

    2023 doi

  12. [21]

    Available: https://doi.org/10.48550/arXiv.2308.12950

    [Online]. Available: https://doi.org/10.48550/arXiv.2308.12950

    doi arXiv

  13. [22]

    Gemma 2: Improving open language models at a practical size,

    G. Team et al. , “Gemma 2: Improving open language models at a practical size,” 2024. [Online]. Available: https://doi.org/10.48550/ arXiv.2408.00118

    2024 doi arXiv

  14. [23]

    Starcoder 2 and the stack v2: The next generation,

    A. Lozhkov et al., “Starcoder 2 and the stack v2: The next generation,” 2024

    2024

  15. [24]

    Mistral 7b,

    A. Q. Jiang et al. , “Mistral 7b,” 2023. [Online]. Available: https://arxiv.org/abs/2310.06825

    2023 arXiv

  16. [25]

    Codegemma: Open code models based on gemma,

    C. Team et al. , “Codegemma: Open code models based on gemma,”

  17. [26]

    Repairllama: Efficient representations and fine-tuned adapters for program repair,

    A. Silva, S. Fang, and M. Monperrus, “Repairllama: Efficient representations and fine-tuned adapters for program repair,” CoRR,

  18. [27]

    Large language model for vulnerability detection: Emerging results and future directions,

    X. Zhou, T. Zhang, and D. Lo, “Large language model for vulnerability detection: Emerging results and future directions,” in International Conference on Software Engineering: New Ideas and Emerging Results . ACM, 2024, pp. 47–51

    2024

  19. [28]

    Large language models for test-free fault localization,

    A. Z. H. Yang, C. Le Goues, R. Martins, and V . J. Hellendoorn, “Large language models for test-free fault localization,” in International Conference on Software Engineering . ACM, 2024, pp. 17:1–17:12

    2024

  20. [29]

    A study of the uniqueness of source code,

    M. Gabel and Z. Su, “A study of the uniqueness of source code,” in International Symposium on Foundations of Software Engineering , G. Roman and A. van der Hoek, Eds. ACM, 2010, pp. 147–156

    2010

  21. [30]

    Available: https://doi.org/10.48550/arXiv.2312.15698

    [Online]. Available: https://doi.org/10.48550/arXiv.2312.15698

    doi arXiv

  22. [31]

    Memorization without overfitting: Analyzing the training dynamics of large language models,

    K. Tirumala, A. H. Markosyan, L. Zettlemoyer, and A. Aghajanyan, “Memorization without overfitting: Analyzing the training dynamics of large language models,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems, 2022

    2022

  23. [32]

    The stack: 3 TB of permissively licensed source code,

    D. Kocetkov et al. , “The stack: 3 TB of permissively licensed source code,” Trans. Mach. Learn. Res. , 2023. [Online]. Available: https://openreview.net/forum?id=pxpbTdUEpD

    2023

  24. [33]

    Measuring massive multitask language understanding,

    D. Hendrycks et al. , “Measuring massive multitask language understanding,” in International Conference on Learning Representations1. OpenReview.net, 2021. [Online]. Available: https://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  25. [34]

    An empirical study of catastrophic forgetting in large language models during continual fine-tuning,

    Y . Luo et al. , “An empirical study of catastrophic forgetting in large language models during continual fine-tuning,” CoRR, vol. abs/2308.08747, 2023. [Online]. Available: https://doi.org/10.48550/ arXiv.2308.08747

    2023 doi arXiv

  26. [35]

    Program synthesis with large language models,

    J. Austin et al. , “Program synthesis with large language models,” CoRR, 2021. [Online]. Available: https://arxiv.org/abs/2108.07732

    2021 arXiv

  27. [36]

    Squad: 100, 000+ questions for machine comprehension of text,

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100, 000+ questions for machine comprehension of text,” in Conference on Empirical Methods in Natural Language Processing , 2016, pp. 2383–2392. [Online]. Available: https://doi.org/10.18653/v1/d16-1264

    2016 doi

  28. [37]

    Keep the conversation going: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,

    C. S. Xia and L. Zhang, “Keep the conversation going: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,” CoRR, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2304.00385

    2023 doi arXiv

  29. [38]

    Evaluating large language models trained on code,

    M. Chen et al. , “Evaluating large language models trained on code,”

  30. [39]

    Livecodebench: Holistic and contamination free evalua- tion of large language models for code,

    N. Jain et al., “Livecodebench: Holistic and contamination free evalua- tion of large language models for code,” CoRR, vol. abs/2403.07974,

    arXiv

  31. [40]

    Why does your data leak? uncovering the data leakage in cloud from mobile apps,

    C. Zuo, Z. Lin, and Y . Zhang, “Why does your data leak? uncovering the data leakage in cloud from mobile apps,” in Symposium on Security and Privacy, 2019, pp. 1296–1310. APPENDIX A Fig. 7: NLL results for all the models and benchmarks. TABLE V: Average NLL across benchmarks ...

    2019

  32. [41]

    Measuring coding challenge competence with APPS,

    D. Hendrycks et al. , “Measuring coding challenge competence with APPS,” in Neural Information Processing Systems Track on Datasets and Benchmarks, J. Vanschoren and S. Yeung, Eds., 2021

    2021

  33. [43]

    Detecting pretraining data from large language models,

    W. Shi et al. , “Detecting pretraining data from large language models,” in International Conference on Learning Representations . OpenReview.net, 2024. [Online]. Available: https://openreview.net/ forum?id=zWqr3MQuNs

    2024

  34. [45]

    Available: https://doi.org/10.48550/arXiv.2403.07974

    [Online]. Available: https://doi.org/10.48550/arXiv.2403.07974

    doi arXiv

  35. [2021]

    Available: https://arxiv.org/abs/2107.03374

    [Online]. Available: https://arxiv.org/abs/2107.03374

    arXiv

  36. [2023]

    Available: https://doi.org/10.48550/arXiv.2309.10677

    [Online]. Available: https://doi.org/10.48550/arXiv.2309.10677

    doi arXiv

  37. [2024]

    Available: https://doi.org/10.48550/arXiv.2406.11409

    [Online]. Available: https://doi.org/10.48550/arXiv.2406.11409

    doi arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.