Pith. sign in

REVIEW 3 major objections 6 minor 48 references

SnipGen: A Mining Repository Framework for Evaluating LLMs for Code

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SnipGen is a framework for mining recent GitHub commits into method-level, prompt-annotated testbeds that aim to keep code-generation benchmarks ahead of LLM memorization.

desk verdict SnipGen ships a real, useful method-level code dataset and mining tool, but its central decontamination claim is asserted, not measured; worth refereeing with conditions. read the letter →

arxiv 2502.07046 v2 pith:TNBEHQGN submitted 2025-02-10 cs.SE cs.AIcs.LG

Daniel Rodriguez-Cardenas , Alejandro Velasco , Denys Poshyvanyk This is my paper · ORCID
classification cs.SEcs.AIcs.LG
keywords LLMevaluationcodegenerationdatacontaminationrepositoryminingpromptengineeringmethod-levelsnippetsGitHubcommitssoftware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces SnipGen, a framework that mines recent GitHub commits to build evaluation testbeds for large language models that write code. The goal is to reduce data contamination: because the snippets are drawn from commits dated after 2021, and are deduplicated by a token-based similarity threshold, they are less likely to have been memorized by models whose training data predates those commits. SnipGen extracts individual Python methods, attaches features such as AST shape, docstrings, complexity metrics, and vulnerability spans, and wraps them in eight composable prompt templates for code completion, commit generation, and code summarization. The paper reports mining roughly 227K data points from 338K recent code changes and releasing the mining tool, the dataset, and the prompts. If the contamination-mitigation premise holds, SnipGen gives researchers a configurable, low-cost way to evaluate code LLMs on data that postdates model training.

What carries the argument

The load-bearing device is the pair of a configurable recent-commit window and a near-duplicate filter: snippets are pulled only from commits within a date range chosen relative to a model's release, then deduplicated at the token level so training-era copies do not dominate. The other central piece is RandomCut, a mutation that truncates a method right after its signature, letting the same snippet serve as a completion task, a code-generation task, or, together with the original code, a commit-message task. The eight prompt templates (P1-P8) combine these pieces, with P6-P8 allowing multi-step chains that refine or restrict the output.

What would settle it

Take a random sample of SnipGen snippets and compare model accuracy on methods whose code existed in the repository before the 2022-2023 window against methods first appearing in that window; if accuracy is equally high on old code, the recent-window filter is not the reason performance is fresh. A more direct check would measure token overlap between SnipGen snippets and a target model's training set, and show that the 0.7 Jaccard threshold removes snippets that appear there.

Watch

Extended reading notes

Core claim

The central claim is that evaluation data for code LLMs can be produced on demand in a way that meaningfully reduces overlap with training data, by sampling method-level snippets from recent, practitioner-specified commit windows rather than relying on static benchmarks. SnipGen operationalizes this by curating only code that changed in the window, removing exact and near duplicates using a BPE-tokenized Jaccard similarity threshold of 0.7, and generating task-specific prompts from the snippet's docstring, signature, and randomly truncated body. The paper also claims that these data points are rich enough to support fine-grained analysis: each snippet carries AST-level features, standard code metrics, vulnerability locations, and the prompts themselves, so researchers can study how code properties influence generation success. In support, it reports six curated testbeds (code completion, docstring-to-code generation, commit-message generation, summarization, and vulnerability-span completion) totaling roughly 5K sampled snippets per task, and lists three prior benchmark studies built on the framework.

Load-bearing premise

The framework's value rests on the assumption that filtering snippets to a recent commit window and removing near-duplicates within that window produces data that current LLMs have not already memorized—an assumption the paper explicitly notes is not checked against entire project histories or actual training corpora.

Editorial extensions

If this is right

  • Researchers can regenerate a testbed after any LLM release by shifting the commit window, so the evaluation data can always postdate the model's training cutoff.
  • The same 5K snippet pool can be assembled into six different task testbeds by changing filters and prompt templates, so task comparisons share the same underlying code.
  • The linked AST, metric, docstring, and vulnerability features allow evaluation results to be stratified by code property, not just averaged.
  • Because the near-duplicate filter is token-based and the time window is configurable, the framework can be adapted to new programming languages and new model release dates without discarding the methodology.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper does not run is a direct memory probe: hold a model fixed and compare its accuracy on SnipGen snippets from before versus after its training cutoff; a drop on post-cutoff snippets would quantify how much of benchmark performance is memorization.
  • The feature set would support causal studies of prompt design, such as whether adding a docstring changes completion quality independently of code length, since the same snippet can be prompted multiple ways.
  • Because the deduplication only compares snippets within the mined set, a strict contamination check would align SnipGen with a target model's actual training corpus; this is a testable extension, not something the paper claims to have done.
  • The vulnerability-labeled snippets could be used to probe whether code LLMs reproduce known vulnerable patterns at higher rates than safe ones; the paper stores those spans but does not evaluate that downstream question.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces SnipGen, a repository mining framework that extracts Python method-level snippets from GitHub commits, computes AST/feature/documentation/vulnerability metadata, and generates prompt-augmented testbeds for code completion, commit generation, and summarization. The authors report mining approximately 227K data points from 338K code changes in 200 popular Python repositories, deduplicating with a BPE-based Jaccard similarity threshold of 0.7, manually validating 960 points, and constructing six task-specific testbeds. Three use cases (Galeras, SyntaxEval, ASTxplainer) are presented as successful applications of SnipGen. The paper's stated goal is to mitigate data contamination in LLM evaluation while providing the mining tool, methodology, and dataset to the community.

Significance. If the framework and dataset are reliable, SnipGen could provide a useful complement to static benchmarks such as HumanEval and CodeXGLUE, offering richer per-snippet metadata (AST structure, complexity, docstring statistics, vulnerability spans) and configurable prompt templates. Concrete strengths include the released open-source code and dataset, the explicit pipeline with reproducible tools, and the multi-dimensional feature schema. However, the primary selling point—contamination mitigation—is not quantitatively validated, and the prompt-template descriptions contain an internal inconsistency that affects reproducibility. The framework is potentially valuable, but the current paper overstates its decontamination guarantees and needs either validation or careful scoping before it can serve as a trusted testbed.

major comments (3)
  1. [III-A, V] The central claim that SnipGen 'mitigates data contamination' is not substantiated by the evidence presented. The deduplication step (BPE-token Jaccard at 0.7) removes near-duplicates only within the newly mined set, and the 2022–2023 commit window does not prevent older lines of code or reused fragments from appearing, as the paper itself concedes in Section V ('Assumption Regarding Snippet Exposure'). Since the abstract and introduction motivate SnipGen specifically by contamination relative to existing benchmarks, the authors should either (a) provide a quantitative overlap analysis against public pretraining corpora (e.g., The Stack, GitHub Code Clean) or against the full git history of the selected repositories, or (b) explicitly scope the claim to 'reduces overlap with recent commits' and remove the stronger wording. Without this, the primary advantage over existing benchmarks is unverified.
  2. [II-C, Table I] There is a direct inconsistency between Table I and the prose in Section II-C. Table I lists P5 as 'I need a summary for the following code:<code>' (summarization) and P6 as 'Change the method signature by<signature>' (processing), but Section II-C states that 'P5 is designed to ask the model to generate the commit message from the mutated code and the actual code' and that 'in code summarization, P6 provides only the code.' Additionally, the text says P1–P5 support single-step tasks while P6–P8 enable multi-step processing, yet the table assigns P6 to a different role. This makes the prompt-generation procedure ambiguous and hampers reproducibility of the testbeds.
  3. [III-A] The manual validation of 960 out of approximately 227K data points (about 0.4%) is too small to support the statement that the extracted features and docstring-code meaningfulness were 'confirmed.' No confidence intervals, per-category breakdown, or inter-rater agreement are reported. Since several testbeds (WithDocString, FromDocString, FromCommit) rely on docstring meaningfulness filters, the authors should either expand the validation sample or present the current effort as a pilot study with corresponding caveats to avoid overstating the quality assurance of the dataset.
minor comments (6)
  1. [Abstract, II-C] The phrase 'Chain-of-Thought-like sequence of prompts' is not demonstrated: no worked example or evaluation of a multi-step combination such as P6–P8 is provided. Suggest adding a concrete example or removing the 'Chain-of-Thought-like' characterization from the abstract.
  2. [III-A] The repository filter 'size: >= 30000' is ambiguous; the unit (bytes, KB, or something else) and the meaning of 'size' should be specified.
  3. [II-A] In the data schema, the type 'Tuple' for span_position is not a concrete database type; clarify how the span is represented (e.g., JSON array or two integer columns).
  4. [III-B] The three use cases cited are all prior works from the same research group. While this is acceptable, the paper would be more persuasive if it included an independent or third-party application of SnipGen to demonstrate general usability.
  5. [Table II, II-B] The testbed names in Table II (e.g., 'SummarizationGen', 'VulnerabilitySpan') are not all defined consistently in Section II-B; aligning the nomenclature would improve readability.
  6. [IV] The comparison with LiveCodeBench is brief; given that LiveCodeBench also targets contamination, a more direct contrast on methodology (temporal commit filtering and BPE deduplication versus continuously updated contest problems) would help position SnipGen.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor self-citation in use-case evidence, but no circular derivation; SnipGen's central claims stand on the released framework and dataset.

  1. other [Section II step 5 and Section III-B (Successful Use Cases), citing [31]-[33]]
    "SnipGen enables the model evaluation and benchmarking as used in [31]–[33]."

    The presented evidence that SnipGen has been successfully used consists of three prior papers by the same research group that consumed the SnipGen dataset. This is self-referential validation and therefore a minor self-citation. It is not load-bearing: the curation pipeline, feature schema, prompt templates, and released code/dataset are described independently, the cited works are externally published, and no prediction or fitted parameter reduces to this citation. The Section V admission that project history is not checked is a limitation on the contamination-mitigation claim, not a circular step.

full rationale

SnipGen does not derive a predicted quantity from fitted parameters or rename an input as an output. Its pipeline (PyDriller commit mining, AST feature extraction, BPE-Jaccard deduplication at a 0.7 threshold, CodeQL vulnerability tagging, and the P1-P8 prompt templates) is concrete and self-contained, and the artifacts are released for independent use. The decontamination goal is explicitly scoped in Section V, 'Assumption Regarding Snippet Exposure,' which concedes that older lines or reused fragments may remain; this is an evidentiary weakness, not circularity. The only self-referential element is the use of the authors' own prior works as successful use cases, which is a minor self-citation and does not force the framework's design or conclusions. Overall circularity burden is low.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No fitted scientific parameters appear; however, several threshold choices shape the dataset and the paper's workload. The key unvalidated premise is the contamination-avoidance assumption, which the authors acknowledge as a limitation.

free parameters (4)
  • Jaccard similarity deduplication threshold = 0.7
    Set manually from cited prior work [35,36]; determines how aggressively near-duplicate snippets are removed and directly shapes testbed uniqueness.
  • Minimum snippet length for RandomCut = 10 tokens or 100 characters
    Manual filter in Section III-A for code completion candidate snippets; affects sample composition and difficulty.
  • Minimum docstring length = 10 words or 50 characters
    Manual filter used for text-to-code and summarization testbeds; affects quality and size.
  • Manual validation sample size = 960 of ~227K data points
    Hand-checked subsample for docstring-code meaningfulness and feature correctness; a practical choice, not a statistically justified coverage.
assumptions (4)
  • domain assumption Recent commit time (2022-2023) and popularity filters produce code snippets unlikely to have been memorized by the evaluated LLMs.
    Underlies the entire contamination mitigation goal; the paper admits in Section V that it does not trace full project history or older reused fragments.
  • ad hoc to paper BPE-tokenizer Jaccard similarity with threshold 0.7 effectively removes duplicate and near-duplicate snippets.
    No ground-truth evaluation of deduplication quality; threshold borrowed from the literature.
  • domain assumption CodeQL vulnerability annotations are correct.
    Section V: detection relies solely on CodeQL and its updates, with no alternative tool validation.
  • domain assumption Manual validation of 960 random samples is adequate to certify docstring-code meaningfulness for sampled testbeds.
    Manual evaluation is not automated and no inter-rater agreement is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SnipGen: A Mining Repository Framework for Evaluating LLMs for Code." pith.science (2026). https://pith.science/paper/TNBEHQGN

@misc{pith2026250207046,
  author       = {Pith},
  title        = {Pith review of: SnipGen: A Mining Repository Framework for Evaluating LLMs for Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TNBEHQGN}},
  note         = {Machine review of arXiv:2502.07046}
}
read the original abstract

Language Models (LLMs), such as transformer-based neural networks trained on billions of parameters, have become increasingly prevalent in software engineering (SE). These models, trained on extensive datasets that include code repositories, exhibit remarkable capabilities for SE tasks. However, evaluating their effectiveness poses significant challenges, primarily due to the potential overlap between the datasets used for training and those employed for evaluation. To address this issue, we introduce SnipGen, a comprehensive repository mining framework designed to leverage prompt engineering across various downstream tasks for code generation. SnipGen aims to mitigate data contamination by generating robust testbeds and crafting tailored data points to assist researchers and practitioners in evaluating LLMs for code-related tasks. In our exploratory study, SnipGen mined approximately 227K data points from 338K recent code changes in GitHub commits, focusing on method-level granularity. SnipGen features a collection of prompt templates that can be combined to create a Chain-of-Thought-like sequence of prompts, enabling a nuanced assessment of LLMs' code generation quality. By providing the mining tool, the methodology, and the dataset, SnipGen empowers researchers and practitioners to rigorously evaluate and interpret LLMs' performance in software engineering contexts.

Figures

Figures reproduced from arXiv: 2502.07046 by the authors.

Figure 2
Figure 2. SnipGen data schema. The snippet represents the core commit collected with documentation. Linked tables contain calculated features. SnipGen can collect a set of Python methods that serve as evaluative data points. Each data point has associated features at seven dimensions as observed at [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. SnipGen Dataset and use. A describes the SnipGen data collection and steps until prompt generation. B describes the canonical path for training and evaluate LLMs The SnipGen framework is designed to select a SE task and evaluate a LLM using the testbed with a given context with a designed prompt [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 32 canonical work pages

  1. [1]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye et al. , “Program synthesis with large language models,” 2021

    work page 2021

  2. [2]

    Measuring coding chal- lenge competence with APPS,

    D. Hendrycks, S. Basart, S. Kadavath et al. , “Measuring coding chal- lenge competence with APPS,” CoRR, vol. abs/2105.09938, 2021

    arXiv 2021

  3. [3]

    Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions,

    M. Chen, J. Tworek, H. Jun et al. , “Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions,” 2021, publisher: arXiv Version Number: 2

    work page 2021

  4. [4]

    Toward deep learn- ing software repositories,

    M. White, C. Vendome, M. Linares-Vasquez et al., “Toward deep learn- ing software repositories,” in 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories , 2015, pp. 334–345

    work page 2015

  5. [5]

    An empirical study on the usage of transformer models for code completion,

    M. Ciniselli, N. Cooper, L. Pascarella et al., “An empirical study on the usage of transformer models for code completion,” IEEE Transactions on Software Engineering , vol. 48, no. 12, pp. 4818–4837, 2022

    work page 2022

  6. [6]

    Ensemble Models for Neural Source Code Summarization of Subroutines,

    A. LeClair, A. Bansal, and C. McMillan, “Ensemble Models for Neural Source Code Summarization of Subroutines,” Jul. 2021, arXiv:2107.11423 [cs]

    arXiv 2021

  7. [7]

    An empirical investigation into the use of image captioning for automated software documentation,

    K. Moran, A. Yachnes, G. Purnell et al., “An empirical investigation into the use of image captioning for automated software documentation,” in 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) , 2022, pp. 514–525

    work page 2022

  8. [8]

    Towards automating code review activities,

    R. Tufano, L. Pascarella, M. Tufano et al. , “Towards automating code review activities,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) , 2021, pp. 163–174

    work page 2021

Show all 48 references
  1. [9]

    Using pre-trained models to boost code review automation,

    R. Tufano, S. Masiero, A. Mastropaolo et al., “Using pre-trained models to boost code review automation,” in2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE) , 2022, pp. 2291–2302

    2022

  2. [10]

    Graph-based statistical language model for code,

    A. T. Nguyen and T. N. Nguyen, “Graph-based statistical language model for code,” in ICSE’15. IEEE Press, 2015, p. 858–868

    2015

  3. [11]

    Deep learning code frag- ments for code clone detection,

    M. White, M. Tufano, C. Vendome et al. , “Deep learning code frag- ments for code clone detection,” in 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), 2016, pp. 87–98

    2016

  4. [12]

    Deep learning similarities from different representations of source code,

    M. Tufano, C. Watson, G. Bavota et al. , “Deep learning similarities from different representations of source code,” in 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR), 2018, pp. 542–553

    2018

  5. [13]

    Learning How to Mutate Source Code from Bug-Fixes,

    ——, “Learning How to Mutate Source Code from Bug-Fixes,” ICSME 2019, pp. 301–312, 2019

    2019

  6. [14]

    Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks

    Y . Zhou, S. Liu, J. Siow et al. , “Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks.”

  7. [15]

    Sorting and transforming program repair ingredients via deep learning code similarities,

    M. White, M. Tufano, M. Mart ´ınez et al. , “Sorting and transforming program repair ingredients via deep learning code similarities,” in 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER) , 2019, pp. 479–490

    2019

  8. [16]

    On learning meaningful code changes via neural machine translation,

    M. Tufano, J. Pantiuchina, C. Watson et al. , “On learning meaningful code changes via neural machine translation,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) , 2019, pp. 25–36

    2019

  9. [17]

    An empirical investigation into learning bug-fixing patches in the wild via neural machine transla- tion,

    M. Tufano, C. Watson, G. Bavota et al. , “An empirical investigation into learning bug-fixing patches in the wild via neural machine transla- tion,” in 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE) , 2018, pp. 832–837

    2018

  10. [18]

    Sequencer: Sequence-to- sequence learning for end-to-end program repair,

    Z. Chen, S. Kommrusch, M. Tufano et al. , “Sequencer: Sequence-to- sequence learning for end-to-end program repair,” IEEE Transactions on Software Engineering , vol. 47, no. 9, pp. 1943–1959, 2021

    1943

  11. [19]

    Can we automatically fix bugs by learning edit operations?

    A. Connor, A. Harris, N. Cooper et al., “Can we automatically fix bugs by learning edit operations?” in 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . Los Alamitos, CA, USA: IEEE Computer Society, mar 2022, pp. 782–792

    2022

  12. [20]

    LiveCodeBench: Holistic and Contami- nation Free Evaluation of Large Language Models for Code,

    N. Jain, K. Han, A. Gu et al., “LiveCodeBench: Holistic and Contami- nation Free Evaluation of Large Language Models for Code,” Jun. 2024, arXiv:2403.07974 [cs]

    2024 arXiv

  13. [21]

    Emergent Abilities of Large Language Models,

    J. Wei, Y . Tay, R. Bommasani et al. , “Emergent Abilities of Large Language Models,” Oct. 2022, arXiv:2206.07682 [cs]

    2022 arXiv

  14. [22]

    Large language models are human-level prompt engineers,

    Y . Zhou, A. I. Muresanu, Z. Han et al. , “Large language models are human-level prompt engineers,” ArXiv, vol. abs/2211.01910, 2022

    2022 arXiv

  15. [23]

    Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design,

    J. White, S. Hays, Q. Fu et al., “Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design,” ArXiv, vol. abs/2303.07839, 2023

    2023 arXiv

  16. [24]

    Prompting is pro- gramming: A query language for large language models,

    L. Beurer-Kellner, M. Fischer, and M. Vechev, “Prompting is pro- gramming: A query language for large language models,” Proc. ACM Program. Lang., vol. 7, no. PLDI, jun 2023

    2023

  17. [25]

    Chain-of-Thought Prompt- ing Elicits Reasoning in Large Language Models,

    J. Wei, X. Wang, D. Schuurmans et al. , “Chain-of-Thought Prompt- ing Elicits Reasoning in Large Language Models,” Jan. 2023, arXiv:2201.11903 [cs]

    2023 arXiv

  18. [26]

    Snipgen tesbed to evaluate llms for code,

    D. Rodriguez-Cardenas, “Snipgen tesbed to evaluate llms for code,” https://doi.org/10.5281/zenodo.14279563, January 2025

    2025 doi

  19. [27]

    Snipgen: A code snippet generation tool,

    S. R. Group, “Snipgen: A code snippet generation tool,” https://github. com/WM-SEMERU/snipgen, 2025, accessed: 2025-01-30

    2025

  20. [28]

    Pydriller documentation,

    PyDriller Contributors, “Pydriller documentation,” https://pydriller. readthedocs.io/en/latest/, n.d., accessed: 2024-11-29

    2024

  21. [29]

    Tree-sitter documentation,

    Tree-Sitter Contributors, “Tree-sitter documentation,” https://tree-sitter. github.io/tree-sitter/, n.d., accessed: 2024-11-29

    2024

  22. [30]

    About codeql,

    GitHub, “About codeql,” https://codeql.github.com/docs/ codeql-overview/about-codeql/, n.d., accessed: 2024-11-29

    2024

  23. [31]

    Benchmarking Causal Study to Interpret Large Language Models for Source Code ,

    D. Rodriguez-Cardenas, D. N. Palacio, D. Khati et al., “ Benchmarking Causal Study to Interpret Large Language Models for Source Code ,” in 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME). Los Alamitos, CA, USA: IEEE Computer Society, Oct. 2023...

    2023

  24. [32]

    Evaluating and Explaining Large Language Models for Code Using Syntactic Structures,

    D. N. Palacio, A. Velasco, D. Rodriguez-Cardenas et al. , “Evaluating and Explaining Large Language Models for Code Using Syntactic Structures,” Aug. 2023, arXiv:2308.03873

    2023 arXiv

  25. [33]

    Which syn- tactic capabilities are statistically learned by masked language models for code?

    A. Velasco, D. N. Palacio, D. Rodriguez-Cardenas et al., “Which syn- tactic capabilities are statistically learned by masked language models for code?” in Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results , s...

    2024

  26. [34]

    Improving ChatGPT Prompt for Code Generation,

    C. Liu, X. Bao, H. Zhang et al., “Improving ChatGPT Prompt for Code Generation,” May 2023, arXiv:2305.08360 [cs]

    2023 arXiv

  27. [35]

    The adverse effects of code duplication in machine learning models of code,

    M. Allamanis, “The adverse effects of code duplication in machine learning models of code,” in OOPLSA, 2019, pp. 143–153

    2019

  28. [36]

    Neural Machine Translation with Byte- Level Subwords,

    C. Wang, K. Cho, and J. Gu, “Neural Machine Translation with Byte- Level Subwords,” Dec. 2019, arXiv:1909.03341

    2019 arXiv

  29. [37]

    DeepFix: Fixing Common C Lan- guage Errors by Deep Learning,

    R. Gupta, S. Pal, A. Kanade et al., “DeepFix: Fixing Common C Lan- guage Errors by Deep Learning,” Proceedings of the AAAI Conference on Artificial Intelligence , vol. 31, no. 1, Feb. 2017, number: 1

    2017

  30. [38]

    Competition-Level Code Generation with AlphaCode,

    Y . Li, D. Choi, J. Chung et al. , “Competition-Level Code Generation with AlphaCode,” Feb. 2022, arXiv:2203.07814

    2022 arXiv

  31. [39]

    Learning to mine aligned code and natural language pairs from stack overflow,

    P. Yin, B. Deng, E. Chen et al. , “Learning to mine aligned code and natural language pairs from stack overflow,” in International Conference on Mining Software Repositories, ser. MSR. ACM, 2018, pp. 476–486

    2018

  32. [40]

    CodeSearchNet chal- lenge: Evaluating the state of semantic code search,

    H. Husain, H.-H. Wu, T. Gazit et al. , “CodeSearchNet chal- lenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436, 2019

    1909 arXiv

  33. [41]

    CodeXGLUE: A machine learning benchmark dataset for code understanding and generation

    S. Lu, D. Guo, S. Ren et al. , “CodeXGLUE: A machine learning benchmark dataset for code understanding and generation.”

  34. [42]

    xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval,

    M. A. M. Khan, M. S. Bari, X. L. Do et al., “xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval,” Nov. 2023, arXiv:2303.03004

    2023 arXiv

  35. [43]

    Evaluating Large Language Models Trained on Code,

    M. Chen, J. Tworek, H. Jun et al., “Evaluating Large Language Models Trained on Code,” Jul. 2021, arXiv:2107.03374 [cs]

    2021 arXiv

  36. [44]

    Securityeval dataset: Mining vul- nerability examples to evaluate machine learning-based code generation techniques,

    M. L. Siddiq and J. C. S. Santos, “Securityeval dataset: Mining vul- nerability examples to evaluate machine learning-based code generation techniques,” in Proceedings of the 1st International Workshop on Min- ing Software Repositories Applications for Privacy and Security , s...

    2022

  37. [45]

    PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs,

    A. Yadav, H. Beniwal, and M. Singh, “PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs,” in Findings of the Association for Computational Linguistics: EMNLP 2024 . Miami, Florida, USA: Association for Computational Linguistics, 2024, pp. 17 113–17 126

    2024

  38. [46]

    Are large language models memorizing bug benchmarks?

    D. Ramos, C. Mamede, K. Jain et al. , “Are large language models memorizing bug benchmarks?” 2024

    2024

  39. [47]

    Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers,

    Q. Guo, R. Wang, J. Guo et al., “Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers,” Feb. 2024, arXiv:2309.08532 [cs]

    2024 arXiv

  40. [48]

    WizardLM: Empowering Large Language Models to Follow Complex Instructions,

    C. Xu, Q. Sun, K. Zheng et al. , “WizardLM: Empowering Large Language Models to Follow Complex Instructions,” Jun. 2023, arXiv:2304.12244

    2023 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.