{"id":"6cd58a65-60c9-450b-8863-382e5cfd2f24","arxiv_id":"2502.05023","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ChatGPT's generated Java methods are more likely to closely match copyleft code when given class or access-method context (up to 5X), less likely at temperature 2, and unaffected by an explicit anti-copy prompt.","lead":"This study measured how often ChatGPT reproduces Java code from copyleft-licensed GitHub projects when asked to implement a method. It found that adding matching class context raises the copy rate several-fold, while hotter temperature settings reduce it.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training-data membership is unverified; the paper's causal language about 'reproducing copyleft code' depends on it, though the relative context effect remains a valid measurement.","rationale":"The reader's weakest assumption is the training-data inclusion, and I agree that it is the main structural dependency. The paper's internal comparisons (RQ2 full class context vs RQ1 minimal context, RQ4 temperature) are statistically solid and the 5X relative increase is robust across similarity thresholds, so the central empirical effect is not in doubt. However, the interpretation of that effect as 'reproducing copyleft code from training data' requires knowing whether the model actually saw these specific methods. The proposed test—membership inference via signature-only prompting plus a post-cutoff control—would settle this without requiring access to OpenAI's training corpus. If the concern lands, it weakens the RQ1 absolute rate and the RQ5 'cannot be asked' conclusion, but it does not overturn the practical guidance that larger matching contexts increase similarity. The RQ3 overgeneralization and the CODEIPPROMPT discrepancy are real but secondary; they warrant caution rather than rejection. The reader's CONDITIONAL verdict already incorporates these caveats, so no verdict change is needed. The paper deserves credit for a large, reproducible study with clear methodology and publicly released data; the concern is about interpretation and external validity, not about the integrity of the measurements.","tokens_in":14025,"tokens_out":13717,"duration_ms":150655,"concrete_test":"Stratified random sample of 150 methods with max similarity >0.7 in RQ1. For each, generate code using only the method signature (no JavaDoc, no class context) with temperature 1; compute JPlag max similarity to the original. If a substantial fraction remain >0.7, memorization is indicated. As a control, select methods from the same 146 repositories from commits dated after 2024 (after the model's knowledge cutoff) and run the RQ1 prompt (signature + JavaDoc). If post-cutoff methods also yield comparable >0.7 rates, the training-data assumption is not necessary; if they yield near-zero rates, the pre-2020 results likely reflect memorization. Report both rates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's core narrative is that ChatGPT 'reproduces' or 'mirrors' copyleft code from its training data, and that a larger context increases this reproduction. The load-bearing assumption, acknowledged in the threats section, is that the selected methods (repositories existing before December 2020) were actually present in GPT-4-turbo's training data. This is unverifiable from public information. If many of the 7,347 methods were not in training, the 19.50% max-similarity rate under full class context could be driven by the model performing plausible in-context completion from the provided class code (which is the original copyleft class with the target method removed), rather than by memorization of the specific method. The RQ3 access-methods condition was designed to mitigate this, but it was run only on 631 methods already flagged as plagiarized under RQ2, so its '2X' result does not generalize to all methods. The near-identical outputs (61 methods at similarity 1.0 with full context) are hard to explain without memorization, but the lower end of the >0.7 range may include generic implementations. The practical risk claim—that accepting copyleft-like recommendations and then providing them as context increases the chance of receiving more similar code—survives regardless of mechanism, but the paper's stronger claim about 'reproducing' training code and its 'cannot be asked' conclusion hinge on the unverified training-data assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically investigates whether ChatGPT (gpt-4-turbo) generates Java methods that are similar to copyleft-licensed implementations, and how context, temperature, and explicit prompts affect this behavior. The authors build a dataset of 7,347 methods from 146 copyleft-licensed GitHub repositories, generate more than 70,000 method implementations across five research questions, and measure similarity using JPlag and a character-level fuzzy ratio. They report that, with only signature and JavaDoc context, 3.35% of methods have a max similarity above 0.7 (RQ1); adding the full class context raises this to 19.50% (RQ2, ~5.8X); adding only access methods raises it about 2X on a subpopulation (RQ3); higher temperature (2) reduces similarity (RQ4); and an explicit request not to copy has no significant effect (RQ5).","tokens_in":14306,"tokens_out":6047,"duration_ms":65582,"significance":"The paper has notable strengths: a large stratified sample, five responses per request, multiple similarity metrics, exploration of the full threshold range, Wilcoxon signed-rank tests with effect sizes, and a public replication package. The RQ2 finding that full-class context multiplies the rate of high-similarity outputs by a factor of five is practically relevant for developers and for designers of AI-assisted coding tools, and the RQ4 result on temperature is a concrete mitigation hypothesis. If the results are interpreted carefully, the study advances the empirical understanding of license-related risks in LLM code generation. The main caveats are that training-data membership of the selected methods is unverified, and the RQ3 result is computed on a non-representative subset, limiting the generality of the headline 2X factor.","major_comments":[{"comment":"The RQ3 analysis is performed only on the 631 methods that were already flagged as likely plagiarized under the full-class context of RQ2. This selection makes the reported 2X increase (from 9.19% to 18.70% for max similarity) conditional on prior flagging, and it does not establish that access methods increase plagiarism risk for the general population of methods. The Answer to RQ3 and the Conclusion state that 'a matching set of access methods ... increases the chance ... by a 2X factor' without this qualification. Please either re-run RQ3 on a random sample of methods not pre-filtered by RQ2, or explicitly frame the result as applying only to methods already at high risk under full-class context.","section":"III-B (Method Selection for RQ3) and Table IV"},{"comment":"The paper states that methods from repositories existing before December 2020 are 'almost certainly' processed by GPT-4-turbo, but no verification is provided. This assumption is load-bearing for the claim that high similarity indicates 'reproducing' copyleft code from training data. In particular, for RQ2 the prompt includes the rest of the class with the target method removed, so the model could generate a similar method by in-context completion from the provided code alone, without any memorization of the training data. Please add a training-membership check or, alternatively, consistently soften the language from 'reproduces' to 'produces code similar to' and explicitly discuss in-context completion as a possible mechanism for the RQ2 effect.","section":"IV-F (Threats to Validity) and IV-A (RQ1)"},{"comment":"The Conclusion and the Answer to RQ5 claim that ChatGPT 'cannot be asked, through the prompt, to avoid reusing existing code.' This is too broad: the experiment tested a single prompt variant ('not copy any known implementation') on a small selected subset (239 methods with max similarity above 0.9 in RQ1). The result shows that this particular instruction has no significant effect on that subset, but it does not warrant the general conclusion that no prompt can reduce reuse. Please limit the claim to the specific instruction tested, or add additional prompt variants to support the stronger claim.","section":"IV-E (RQ5)"}],"minor_comments":[{"comment":"The text says that results are discussed for thresholds above 0.7 and that every threshold value with a step of 0.05 was considered, but the paper only presents tables and figures for the 0.7 threshold. It would be helpful to show how the conclusions change across the threshold sweep, even in an appendix.","section":"III-B (Plagiarism detection)"},{"comment":"The description of the repository selection criteria says 'by the end of 2020' and later 'before December 2020.' Please make the cutoff phrasing consistent.","section":"III-A (Repository Selection)"},{"comment":"The text describes the difference distributions as 'skewed on the left,' but does not explicitly state that negative differences correspond to higher similarity with the broader context. Adding axis labels and a sentence explaining the sign convention would improve readability.","section":"IV-B (RQ2 Results)"},{"comment":"For the temperature 0 versus 1 comparison, the paper reports p-values but not effect sizes, while effect sizes are reported for the temperature 2 comparison. Please report effect sizes for all statistically significant comparisons, including the fuzzy ratio results.","section":"IV-D (RQ4)"},{"comment":"The sentence 'we do expect the observed degree of reuse of existing code to be an intrinsic characteristic of the architecture of the model' is speculative and not supported by the single-model design. It would be more accurate to say that the findings are limited to GPT-4-turbo and should be replicated on other models.","section":"IV-F (Threats to Validity)"},{"comment":"There are a few typos: '2,3X increase' in Section IV-C should be '2.3X'; reference [2] has 'EEE/ACM' instead of 'IEEE/ACM'; and in reference [8] the author name appears as 'Y . V orobeychik' with a stray space.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is empirically solid in its design and execution, and the RQ2 context effect is a valuable contribution. The main issues are (1) the RQ3 2X claim is computed on a pre-filtered subset and needs to be either re-analyzed on a representative sample or explicitly conditioned on the subset; and (2) the training-data membership assumption is asserted rather than verified, and the causal wording in the abstract and conclusions should be softened accordingly. The RQ5 conclusion should also be narrowed to the specific tested prompt. These are fixable with additional analyses or careful rewording; the underlying methodology and data collection are sound enough for a major-revision recommendation rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one if you care about LLM code generation and licensing. The headline result is believable: with only a method signature and Javadoc, GPT-4-turbo returns code with max similarity above 0.7 to a copyleft implementation 3.35% of the time; add the rest of the class as context and it jumps to 19.50%. That's a 5.8X increase, with large effect sizes, and it survives the threshold sweep. The access-methods condition also raises similarity, but note it's run only on 631 methods already flagged under full context, so the 2X factor is a conditional result, not a population estimate. Temperature 2 does reduce similarity; temperature 0 vs 1 is a wash. Explicitly asking not to copy doesn't move the needle.\n\nWhat's new compared to CODEIPPROMPT is the systematic mapping of context and temperature effects on a frontier model, plus the practical scenario: once a developer has accepted some copyleft-like code, the context itself increases the chance of getting more. That's actionable. The design is solid: stratified sampling, five responses per request, JPlag plus character-level fuzzy ratio, all thresholds reported, Wilcoxon with effect sizes. They also release the material.\n\nSoft spots. The load-bearing assumption that the selected methods were in GPT-4-turbo's training data is unverifiable, and the paper relies on it for the word 'reproduce.' The relative context effect doesn't depend on that assumption—it's a behavioral measurement—but the framing does. The RQ5 'cannot be asked' conclusion is stated more strongly than the evidence: it's tested only on 239 methods with similarity above 0.9, so the null result may not generalize to less extreme cases. And the paper never reconciles its low baseline (3.35%) with CODEIPPROMPT's 50%+ figure; a sentence or two on prompt differences would help.\n\nWho it's for: developers and legal teams wanting numbers, and researchers studying LLM memorization. The central argument holds up. It deserves peer review—the methodology is careful and the context effect is a genuine contribution, even if the training-data assumption caps how far the causal story can go. I'd accept it with requests to soften the overgeneralizations.","headline":"Solid measurement study: the context effect on reproducing copyleft code is real and new, but a couple of secondary claims are overgeneralized.","tokens_in":14830,"tokens_out":1957,"would_cite":true,"duration_ms":19457,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ChatGPT reproduces copyleft code more than five times as often when given the full class as context.","keywords":["AI-assisted coding","code generation","copyleft licenses","intellectual property","ChatGPT","GPT-4-turbo","code memorization","plagiarism detection"],"falsifier":"Replicate the experiment with a set of copyleft-licensed Java methods published after GPT-4-turbo's training cutoff (for example, created in 2024). If the full-class context still raises the fraction of generated methods with similarity above 0.7 to roughly 19%, then the amplification effect does not require the methods to have been in training data, contradicting the memorization interpretation; if the rate stays near the baseline 3.35%, then the context effect is indeed tied to training set exposure.","tokens_in":13847,"feed_emoji":"⚖️","tokens_out":3068,"duration_ms":30734,"temperature":0.7,"pith_summary":"This paper asks whether code generated by ChatGPT can accidentally reproduce copyleft-licensed code, and what makes that more or less likely. Using more than 70,000 generated Java method implementations, the authors find that while individual requests rarely match copyleft code, providing the rest of the class as context raises the worst-case similarity rate from 3.35% to 19.50%—more than a fivefold increase. Even just including access methods roughly doubles the risk. Raising the temperature to 2 reduces similarity, but explicitly instructing the model not to copy known implementations has no meaningful effect. The results matter because developers who accept one suspicious recommendation may unwittingly increase the chance that subsequent suggestions also carry copyleft restrictions.","feed_headline":"ChatGPT copies copyleft code 5x more with class context","feed_subtitle":"Providing a fuller class as context raises risky similarity from 3.35% to 19.50%; high temperature helps, but asking it not to copy does…","key_machinery":"The study centers on a dataset of 7,347 Java methods extracted from 146 copyleft-licensed GitHub repositories that existed before December 2020, guaranteeing the code predates GPT-4-turbo's training cutoff. Similarity between generated and original methods is measured with JPlag's token-level similarity and a character-level fuzzy ratio based on Levenshtein distance, with suspicious cases defined as similarity above 0.7. The experimental machinery systematically varies prompts: baseline uses signature plus JavaDoc, the full-class condition embeds the rest of the class, the access-method condition adds only getters and setters, temperature is set to 0, 1, or 2, and one prompt explicitly asks the model not to copy. Wilcoxon signed-rank tests with effect sizes distinguish real factor effects from noise.","core_discovery":"The central claim is that the likelihood ChatGPT returns code closely resembling copyleft-licensed implementations depends strongly on the context supplied in the prompt. When the prompt contains only the method signature and JavaDoc, 3.35% of methods exceed a maximum similarity of 0.7 to the original copyleft code. When the prompt includes the rest of the class, this rises to 19.50%; with only access methods as context, it rises to 18.70% on the restricted sample. Differences are statistically significant with large effect sizes for the class context. A temperature of 2 lowers similarity compared to temperature 1, while temperature 0 has a mild effect. An explicit request to avoid copying does not shift similarity in a statistically significant way.","pith_inferences":["A direct test of whether high similarity truly reflects training-data memorization would be to repeat the protocol on methods published after GPT-4-turbo's training cutoff; if the context effect persists for post-cutoff methods, the mechanism would be in-context pattern completion rather than memorization.","The cascade effect implies a kind of licensing ratchet: the more copyleft-like code a developer incorporates, the harder it becomes to keep later suggestions clean, which raises the practical importance of checking every generated method before acceptance.","The same measurement protocol could be applied to open-weight models with inspectable training data to separate memorization from other causes, and to estimate how much variability across models is due to training-set overlap rather than architecture.","One could design a lightweight run-time detector that flags generated methods whose surrounding class context matches a copyleft repository, using the paper's observed 5x amplification as motivation for checking context similarity before inserting a generated method."],"forward_implications":["Developers who accept one code recommendation that matches copyleft code are substantially more likely to receive further such recommendations when they later ask for other methods in the same class.","Using a high temperature setting (for example 2) reduces the chance of reproducing copyleft code, though the paper notes this must be balanced against evidence that high temperatures degrade code correctness.","Telling ChatGPT not to copy known implementations is not an effective safeguard, so technical and process-level defenses are needed rather than prompt-based ones.","The measured risk is likely an under-approximation because the dataset only covers GitHub repositories selected before December 2020, and other code sources are not considered.","The observed context amplification is expected to transfer beyond GPT-4-turbo and beyond Java, since the authors attribute it to the memorization properties of large language models rather than to a model-specific quirk."],"supporting_citations":[{"why":"Shows that large code models memorize training data and reproduce snippets, establishing the memorization phenomenon this study extends.","marker":"[6]"},{"why":"Reports that training code appears in model outputs across diverse prompts, providing further prior evidence for the memorization mechanism.","marker":"[7]"},{"why":"Assesses intellectual-property infringement in code LLMs, the baseline context for this paper's study of GPT-4 with varying prompts.","marker":"[8]"},{"why":"Supplies the GitHub repository source from which the copyleft Java methods dataset is built.","marker":"[19]"},{"why":"JPlag is the plagiarism-detection tool used to compute token-level similarity between generated and original methods.","marker":"[21]"},{"why":"Provides the fuzzy-ratio metric based on Levenshtein distance that complements JPlag similarity for near-identical copies.","marker":"[22]"},{"why":"The Wilcoxon signed-rank test is the statistical test used to determine whether factor changes produce significant differences.","marker":"[23]"},{"why":"Supplies the effect-size measure (Cohen's r) used to interpret the practical significance of the observed differences.","marker":"[24]"}],"fun_headline_variants":["Class context makes ChatGPT 5x likelier to echo copyleft code","ChatGPT copyleft code reuse jumps 5x with full class context","Full class context ups ChatGPT's copyleft-code mimicry to 19.5%","ChatGPT copies copyleft code 5x more when given class context","Asking doesn't help: class context boosts ChatGPT copyleft copying 5x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study assumes that the GitHub methods selected from repositories existing before December 2020 were actually part of GPT-4-turbo's training data, an assumption it cannot verify directly; if many were not, the observed similarity increases could be due to in-context pattern completion rather than memorization of copyleft code.","fun_headline_variants_meta":{"raw":{"variants":["Class context makes ChatGPT 5x likelier to echo copyleft code","ChatGPT copyleft code reuse jumps 5x with full class context","Full class context ups ChatGPT's copyleft-code mimicry to 19.5%","ChatGPT copies copyleft code 5x more when given class context","Asking doesn't help: class context boosts ChatGPT copyleft copying 5x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000854,"raw_usage":{"total_tokens":3649,"prompt_tokens":821,"completion_tokens":2828,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":437,"completion_tokens_details":{"reasoning_tokens":2722}},"tokens_in":437,"tokens_out":2828,"duration_ms":21631,"temperature":1.0,"reasoning_tokens":2722,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T20:32:53.744624+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replicate the experiment with a set of copyleft-licensed Java methods published after GPT-4-turbo's training cutoff (for example, created in 2024). If the full-class context still raises the fraction of generated methods with similarity above 0.7 to roughly 19%, then the amplification effect does not require the methods to have been in training data, contradicting the memorization interpretation; if the rate stays near the baseline 3.35%, then the context effect is indeed tied to training set exposure.","supporting_citations":[{"cited_title":"Traces of memorisation in large language models for code,","cited_arxiv_id":null,"evidence_quote":"Shows that large code models memorize training data and reproduce snippets, establishing the memorization phenomenon this study extends."},{"cited_title":"Unveil- ing memorization in code models,","cited_arxiv_id":null,"evidence_quote":"Reports that training code appears in model outputs across diverse prompts, providing further prior evidence for the memorization mechanism."},{"cited_title":"Codeipprompt: Intellectual property infringement assessment of code language models,","cited_arxiv_id":null,"evidence_quote":"Assesses intellectual-property infringement in code LLMs, the baseline context for this paper's study of GPT-4 with varying prompts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GitHub repository source from which the copyleft Java methods dataset is built."},{"cited_title":"Jplag: State-of-the-art software plagiarism & collusion detec- tion,","cited_arxiv_id":null,"evidence_quote":"JPlag is the plagiarism-detection tool used to compute token-level similarity between generated and original methods."},{"cited_title":"(2024) Thefuzz","cited_arxiv_id":null,"evidence_quote":"Provides the fuzzy-ratio metric based on Levenshtein distance that complements JPlag similarity for near-identical copies."}],"review_version":1}