{"id":"69958010-752e-4595-98fe-86a2c1e26dd5","arxiv_id":"2502.07046","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new framework mines recent GitHub commits into about 227K method-level code snippets with prompt templates for testing LLMs on code generation while reducing training-data contamination.","lead":"SnipGen is a tool and dataset for evaluating code-generating AI models. It mines recent GitHub commits to build test questions with prompts, aiming to avoid overlap between what models were trained on and what they are tested on.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Contamination mitigation is asserted but never measured: the 2022-2023 commit window plus internal BPE-Jaccard dedup at 0.7 does not prevent near-duplicates in LLM pretraining corpora, and Section V concedes project history is not checked.","rationale":"The reader's verdict is CONDITIONAL with high confidence, and I agree. I checked the central claim rather than peripheral issues. The prompt-template inconsistency in Table I (P5 labeled 'Summarization' but described in Section II-C as commit generation) is real but does not invalidate the framework; it is a reproducibility issue. The unvalidated contamination claim is more load-bearing because it is the stated reason the testbed should be preferred over existing benchmarks. The paper is honest about the limitation in Section V, which is a point in its favor, but honesty about a limitation does not remove the need for validation. A corpus-overlap experiment is the one direct check that would settle the matter. My recommendation is to keep the CONDITIONAL verdict and make the corpus-overlap analysis one of the explicit conditions. This is not an objection to the artifact's existence; it is a request to verify the paper's central motivation.","tokens_in":8697,"tokens_out":3688,"duration_ms":34750,"concrete_test":"Download a public pretraining corpus that is a plausible training source for open code LLMs (e.g., The Stack v1, or the StarCoderData snapshot), filter to files with timestamps before 2022-01-01, and for every snippet in the RandomCut and WithDocString testbeds compute the maximum exact-substring overlap and BPE-Jaccard similarity (using the same tokenizer and threshold of 0.7 as in Section III-A) against that corpus. Report the fraction of testbed snippets with at least one near-duplicate. If that fraction is above a small prespecified bound (e.g., 5%), the claimed contamination mitigation for the released dataset is not supported; if it is near zero, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's motivating claim is that SnipGen 'aims to mitigate data contamination by generating robust testbeds.' The actual mitigation consists of (i) restricting commit collection to 2022-2023 and (ii) removing duplicates within the newly mined data using a BPE-token Jaccard threshold of 0.7 (Section III-A). Neither step compares the final snippets against any LLM training corpus or against the full git history of the selected repositories. Since the authors deliberately select the 200 most popular Python repositories (keras, numpy, pandas, sentry), large portions of those repositories are very likely present in pretraining corpora such as The Stack or GitHub Code Clean, including in earlier commits or copies in other repositories. The paper's own Section V, 'Assumption Regarding Snippet Exposure,' admits that extracted code changes 'might include older lines of code or reused code fragments' and that the evaluation does not encompass the entire project history. That admission directly undercuts the headline motivation: if a nontrivial fraction of RandomCut prefixes or full methods has a near-duplicate in pretraining data, then performance on SnipGen can reflect memorization rather than generalization. No quantitative overlap analysis is provided, so the central claim is currently unverified rather than disproven; the framework and dataset can still be useful if the decontamination claim is scoped down or validated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SnipGen, a repository mining framework that extracts Python method-level snippets from GitHub commits, computes AST/feature/documentation/vulnerability metadata, and generates prompt-augmented testbeds for code completion, commit generation, and summarization. The authors report mining approximately 227K data points from 338K code changes in 200 popular Python repositories, deduplicating with a BPE-based Jaccard similarity threshold of 0.7, manually validating 960 points, and constructing six task-specific testbeds. Three use cases (Galeras, SyntaxEval, ASTxplainer) are presented as successful applications of SnipGen. The paper's stated goal is to mitigate data contamination in LLM evaluation while providing the mining tool, methodology, and dataset to the community.","tokens_in":9006,"tokens_out":5164,"duration_ms":44696,"significance":"If the framework and dataset are reliable, SnipGen could provide a useful complement to static benchmarks such as HumanEval and CodeXGLUE, offering richer per-snippet metadata (AST structure, complexity, docstring statistics, vulnerability spans) and configurable prompt templates. Concrete strengths include the released open-source code and dataset, the explicit pipeline with reproducible tools, and the multi-dimensional feature schema. However, the primary selling point—contamination mitigation—is not quantitatively validated, and the prompt-template descriptions contain an internal inconsistency that affects reproducibility. The framework is potentially valuable, but the current paper overstates its decontamination guarantees and needs either validation or careful scoping before it can serve as a trusted testbed.","major_comments":[{"comment":"The central claim that SnipGen 'mitigates data contamination' is not substantiated by the evidence presented. The deduplication step (BPE-token Jaccard at 0.7) removes near-duplicates only within the newly mined set, and the 2022–2023 commit window does not prevent older lines of code or reused fragments from appearing, as the paper itself concedes in Section V ('Assumption Regarding Snippet Exposure'). Since the abstract and introduction motivate SnipGen specifically by contamination relative to existing benchmarks, the authors should either (a) provide a quantitative overlap analysis against public pretraining corpora (e.g., The Stack, GitHub Code Clean) or against the full git history of the selected repositories, or (b) explicitly scope the claim to 'reduces overlap with recent commits' and remove the stronger wording. Without this, the primary advantage over existing benchmarks is unverified.","section":"III-A, V"},{"comment":"There is a direct inconsistency between Table I and the prose in Section II-C. Table I lists P5 as 'I need a summary for the following code:<code>' (summarization) and P6 as 'Change the method signature by<signature>' (processing), but Section II-C states that 'P5 is designed to ask the model to generate the commit message from the mutated code and the actual code' and that 'in code summarization, P6 provides only the code.' Additionally, the text says P1–P5 support single-step tasks while P6–P8 enable multi-step processing, yet the table assigns P6 to a different role. This makes the prompt-generation procedure ambiguous and hampers reproducibility of the testbeds.","section":"II-C, Table I"},{"comment":"The manual validation of 960 out of approximately 227K data points (about 0.4%) is too small to support the statement that the extracted features and docstring-code meaningfulness were 'confirmed.' No confidence intervals, per-category breakdown, or inter-rater agreement are reported. Since several testbeds (WithDocString, FromDocString, FromCommit) rely on docstring meaningfulness filters, the authors should either expand the validation sample or present the current effort as a pilot study with corresponding caveats to avoid overstating the quality assurance of the dataset.","section":"III-A"}],"minor_comments":[{"comment":"The phrase 'Chain-of-Thought-like sequence of prompts' is not demonstrated: no worked example or evaluation of a multi-step combination such as P6–P8 is provided. Suggest adding a concrete example or removing the 'Chain-of-Thought-like' characterization from the abstract.","section":"Abstract, II-C"},{"comment":"The repository filter 'size: >= 30000' is ambiguous; the unit (bytes, KB, or something else) and the meaning of 'size' should be specified.","section":"III-A"},{"comment":"In the data schema, the type 'Tuple' for span_position is not a concrete database type; clarify how the span is represented (e.g., JSON array or two integer columns).","section":"II-A"},{"comment":"The three use cases cited are all prior works from the same research group. While this is acceptable, the paper would be more persuasive if it included an independent or third-party application of SnipGen to demonstrate general usability.","section":"III-B"},{"comment":"The testbed names in Table II (e.g., 'SummarizationGen', 'VulnerabilitySpan') are not all defined consistently in Section II-B; aligning the nomenclature would improve readability.","section":"Table II, II-B"},{"comment":"The comparison with LiveCodeBench is brief; given that LiveCodeBench also targets contamination, a more direct contrast on methodology (temporal commit filtering and BPE deduplication versus continuously updated contest problems) would help position SnipGen.","section":"IV"}],"recommendation":"major_revision","confidential_remarks":"The paper is essentially an artifact/dataset paper: its main contribution is the mining tool, the methodology, and the released dataset, not an empirical evaluation of LLMs. The load-bearing issue is the contamination-mitigation claim, which is currently unverified and partially contradicted by the authors' own limitation statement. The prompt-template inconsistency in Table I versus Section II-C is a concrete reproducibility defect that must be fixed. The self-citation pattern is acceptable given that the cited works are prior published uses of the dataset, but an independent use case would strengthen the paper's claims of general utility. In my view, the artifact is valuable enough to warrant a major revision rather than rejection, provided the authors either provide a quantitative contamination analysis or carefully scope their claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on SnipGen. The paper gives the code-LLM community a genuinely new artifact: roughly 227K method-level Python snippets mined from 2022-2023 commits in top-200 repositories, with docstrings, AST features, CodeQL vulnerability spans, and prompt templates for completion, commit generation, and summarization. The mining pipeline is described in enough detail to reproduce, and the code and dataset links are provided. That is real value. The three use cases show the dataset has been exercised in practice, though all three are the authors' own prior works, so they are evidence of usability, not independent validation.\n\nThe soft spots are real but mostly non-fatal. The biggest one is exactly what the stress test flags: the contamination-mitigation claim is never measured. Selecting 2022-2023 commits and deduplicating internally at a BPE-Jaccard threshold of 0.7 does not tell you whether the final snippets have near-duplicates in pretraining corpora like The Stack. To the authors' credit, Section V says this outright: the evaluation does not check the full project history and extracted changes may include older or reused code fragments. So the paper is honest about the limitation, but the abstract still promises that SnipGen 'aims to mitigate data contamination.' That mismatch needs fixing in revision—either add a quantitative overlap analysis or scope the claim down.\n\nSecond, the manual validation of 960 out of ~227K points (0.4%) is thin, though random sampling does give some confidence. Third, I noticed an inconsistency between Table I and the prose: the table labels P5 as summarization and P6 as signature processing, while the text assigns P5 to commit generation and P6 to summarization. Sloppy, easily fixed, but confusing.\n\nThe circularity worry is minor. The dataset is independent of the outcomes in the three use cases; self-citation is expected in a resource paper.\n\nOverall: this is an infrastructure report, not a scientific claim that must be true to be useful. If you treat it as 'here is a fresh, method-level testbed plus tooling,' it holds up. If you treat the decontamination claim as established, it doesn't. I'd send this to peer review with a request for the overlap analysis or a revised claim, a fixed prompt table, and a pinned release of the code. A serious referee would get value out of it.","headline":"SnipGen ships a real, useful method-level code dataset and mining tool, but its central decontamination claim is asserted, not measured; worth refereeing with conditions.","tokens_in":9508,"tokens_out":2338,"would_cite":true,"duration_ms":19799,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SnipGen is a framework for mining recent GitHub commits into method-level, prompt-annotated testbeds that aim to keep code-generation benchmarks ahead of LLM memorization.","keywords":["LLM evaluation","code generation","data contamination","repository mining","prompt engineering","method-level snippets","GitHub commits","software engineering"],"falsifier":"Take a random sample of SnipGen snippets and compare model accuracy on methods whose code existed in the repository before the 2022-2023 window against methods first appearing in that window; if accuracy is equally high on old code, the recent-window filter is not the reason performance is fresh. A more direct check would measure token overlap between SnipGen snippets and a target model's training set, and show that the 0.7 Jaccard threshold removes snippets that appear there.","tokens_in":8507,"feed_emoji":"💻","tokens_out":6483,"duration_ms":56222,"temperature":0.7,"pith_summary":"The paper introduces SnipGen, a framework that mines recent GitHub commits to build evaluation testbeds for large language models that write code. The goal is to reduce data contamination: because the snippets are drawn from commits dated after 2021, and are deduplicated by a token-based similarity threshold, they are less likely to have been memorized by models whose training data predates those commits. SnipGen extracts individual Python methods, attaches features such as AST shape, docstrings, complexity metrics, and vulnerability spans, and wraps them in eight composable prompt templates for code completion, commit generation, and code summarization. The paper reports mining roughly 227K data points from 338K recent code changes and releasing the mining tool, the dataset, and the prompts. If the contamination-mitigation premise holds, SnipGen gives researchers a configurable, low-cost way to evaluate code LLMs on data that postdates model training.","feed_headline":"Recent-commit mining yields 227K fresh code tests for LLMs","feed_subtitle":"SnipGen mines code written after 2021 into method-level prompts for completion, commit messages, and summaries.","key_machinery":"The load-bearing device is the pair of a configurable recent-commit window and a near-duplicate filter: snippets are pulled only from commits within a date range chosen relative to a model's release, then deduplicated at the token level so training-era copies do not dominate. The other central piece is RandomCut, a mutation that truncates a method right after its signature, letting the same snippet serve as a completion task, a code-generation task, or, together with the original code, a commit-message task. The eight prompt templates (P1-P8) combine these pieces, with P6-P8 allowing multi-step chains that refine or restrict the output.","core_discovery":"The central claim is that evaluation data for code LLMs can be produced on demand in a way that meaningfully reduces overlap with training data, by sampling method-level snippets from recent, practitioner-specified commit windows rather than relying on static benchmarks. SnipGen operationalizes this by curating only code that changed in the window, removing exact and near duplicates using a BPE-tokenized Jaccard similarity threshold of 0.7, and generating task-specific prompts from the snippet's docstring, signature, and randomly truncated body. The paper also claims that these data points are rich enough to support fine-grained analysis: each snippet carries AST-level features, standard code metrics, vulnerability locations, and the prompts themselves, so researchers can study how code properties influence generation success. In support, it reports six curated testbeds (code completion, docstring-to-code generation, commit-message generation, summarization, and vulnerability-span completion) totaling roughly 5K sampled snippets per task, and lists three prior benchmark studies built on the framework.","pith_inferences":["An extension the paper does not run is a direct memory probe: hold a model fixed and compare its accuracy on SnipGen snippets from before versus after its training cutoff; a drop on post-cutoff snippets would quantify how much of benchmark performance is memorization.","The feature set would support causal studies of prompt design, such as whether adding a docstring changes completion quality independently of code length, since the same snippet can be prompted multiple ways.","Because the deduplication only compares snippets within the mined set, a strict contamination check would align SnipGen with a target model's actual training corpus; this is a testable extension, not something the paper claims to have done.","The vulnerability-labeled snippets could be used to probe whether code LLMs reproduce known vulnerable patterns at higher rates than safe ones; the paper stores those spans but does not evaluate that downstream question."],"forward_implications":["Researchers can regenerate a testbed after any LLM release by shifting the commit window, so the evaluation data can always postdate the model's training cutoff.","The same 5K snippet pool can be assembled into six different task testbeds by changing filters and prompt templates, so task comparisons share the same underlying code.","The linked AST, metric, docstring, and vulnerability features allow evaluation results to be stratified by code property, not just averaged.","Because the near-duplicate filter is token-based and the time window is configurable, the framework can be adapted to new programming languages and new model release dates without discarding the methodology."],"supporting_citations":[{"why":"Provides the motivating contrast: a continuously updated, contamination-aware benchmark for code LLMs.","marker":"[20]"},{"why":"Supplies the Chain-of-Thought prompting technique that SnipGen's composable prompt sequences build on.","marker":"[25]"},{"why":"The commit-mining library used to extract snippets from repository histories.","marker":"[28]"},{"why":"The AST parser used to compute syntactic features and identify method boundaries.","marker":"[29]"},{"why":"The static-analysis engine used to detect and locate vulnerability spans in snippets.","marker":"[30]"},{"why":"Supports the multi-step prompt configuration pattern used by the processing prompt templates.","marker":"[34]"},{"why":"Establishes that code duplication harms learned code models, motivating the near-duplicate filter.","marker":"[35]"},{"why":"Supplies the byte-level subword (BPE) tokenization used in the Jaccard similarity deduplication.","marker":"[36]"}],"fun_headline_variants":["SnipGen mines 227K fresh code tasks from recent commits","Fight data contamination with fresh commit-derived code tests","Code LLM evaluation gets fresh on-demand data from commits","SnipGen: Fresh benchmark mining for less contaminated LLM tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's value rests on the assumption that filtering snippets to a recent commit window and removing near-duplicates within that window produces data that current LLMs have not already memorized—an assumption the paper explicitly notes is not checked against entire project histories or actual training corpora.","fun_headline_variants_meta":{"raw":{"variants":["SnipGen mines 227K fresh code tasks from recent commits","Fight data contamination with fresh commit-derived code tests","Code LLM evaluation gets fresh on-demand data from commits","SnipGen: Fresh benchmark mining for less contaminated LLM tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001223,"raw_usage":{"total_tokens":5037,"prompt_tokens":963,"completion_tokens":4074,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":4004}},"tokens_in":579,"tokens_out":4074,"duration_ms":25134,"temperature":1.0,"reasoning_tokens":4004,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T13:56:44.212383+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of SnipGen snippets and compare model accuracy on methods whose code existed in the repository before the 2022-2023 window against methods first appearing in that window; if accuracy is equally high on old code, the recent-window filter is not the reason performance is fresh. A more direct check would measure token overlap between SnipGen snippets and a target model's training set, and show that the 0.7 Jaccard threshold removes snippets that appear there.","supporting_citations":[{"cited_title":"Pydriller documentation,","cited_arxiv_id":null,"evidence_quote":"The commit-mining library used to extract snippets from repository histories."},{"cited_title":"Tree-sitter documentation,","cited_arxiv_id":null,"evidence_quote":"The AST parser used to compute syntactic features and identify method boundaries."},{"cited_title":"About codeql,","cited_arxiv_id":null,"evidence_quote":"The static-analysis engine used to detect and locate vulnerability spans in snippets."},{"cited_title":"The adverse effects of code duplication in machine learning models of code,","cited_arxiv_id":null,"evidence_quote":"Establishes that code duplication harms learned code models, motivating the near-duplicate filter."}],"review_version":1}