{"id":"9973f488-0b04-4bb6-a9cb-cefbcb008471","arxiv_id":"2607.07481","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"A competition report showing that white-box membership inference attacks on code LLMs mostly fail (AUC ~0.56–0.61) except for one structure-aware method (SERSEM, AUC ~0.77) that generalizes to a held-out model.","lead":"This paper reports results of a competition where participants tried to detect whether code files were used to train large language models, using only model internals. Most methods barely outperformed random guessing, but one approach using structural code signals achieved AUC ~0.77 and generalized to a held-out model. The work matters because contamination detection is essential for trustworthy LLM benchmarks in software engineering.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Member and non-member sets come from different source datasets (The Stack Edu vs The Heap), creating a distributional confound that SERSEM's feature design may exploit independent of genuine membership signals.","rationale":"The reader identified a ground-truth quality concern (The Heap's deduplication false negatives), which is legitimate but somewhat speculative — The Heap was purpose-built for this task and deduplicated against The Stack v2 using LSH. The more load-bearing concern is the systematic distributional confound between the two source datasets. Even with perfect deduplication, members come from The Stack Edu and non-members from The Heap, and these datasets likely differ in code style, comment density, and structural features — exactly the features SERSEM is designed to detect. The BoW curation addresses lexical artifacts but not deeper distributional shifts. This concern does not invalidate the competition design or the paper's contribution as a benchmark, but it does mean the specific interpretation that SERSEM's advantage reflects 'structure-aware membership inference' (rather than source-dataset detection) is not fully established by the current evaluation. The verdict remains CONDITIONAL: the paper is a valuable community contribution, but the central claim about why SERSEM outperforms baselines requires the additional control described above to be fully convincing. The reader's other points (no error bars, PAC sample size mismatch, only 2 submissions) are valid but secondary — with n=5000, the SERSEM vs baseline gap (0.773 vs 0.61) is large enough to be statistically meaningful, and the small number of submissions is a limitation of a first-edition competition rather than a flaw in the evaluation design.","tokens_in":7890,"tokens_out":3942,"duration_ms":105779,"concrete_test":"Construct a within-source-dataset control: sample non-members from The Stack v2 files known to be excluded from a specific model's training (e.g., files added after the training cutoff, or files from repositories not included in The Stack Edu subset). Evaluate SERSEM on this control where both members and non-members originate from the same source dataset. If SERSEM's AUC drops substantially (e.g., below 0.65) compared to the reported 0.773, the current performance is inflated by source-dataset distributional artifacts rather than genuine membership signals. If AUC remains comparable, the membership inference interpretation is strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SERSEM outperforms probability-based methods because its structure-aware features detect genuine memorization. However, the member set is drawn from The Stack Edu and the non-member set from The Heap (§2.2) — these are fundamentally different data sources with potentially different code distributions. The BoW curation step only removes files distinguishable by keyword presence; it does not address deeper distributional shifts in comment density, identifier length, formatting conventions, or code structure. Critically, SERSEM specifically targets 'comments, long identifiers, string literals, formatting anomalies, and developer markers like TODO' (§3.1) — precisely the features most likely to differ systematically between educational/curated code (The Stack Edu) and general deduplicated code (The Heap). If these features differ between source datasets, SERSEM could achieve high AUC by detecting source-dataset origin rather than model training membership. The held-out Mellum evaluation does not resolve this confound: both StarCoder2 and Mellum were trained on The Stack v2 (which includes The Stack Edu), so the same source-dataset distinction exists for both models. A method that detects 'is this file from The Stack Edu?' would generalize across both models and achieve high AUC without performing genuine membership inference. This is distinct from the reader's concern about deduplication false negatives — even with perfect deduplication, the cross-dataset confound remains.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper reports the setup and results of the 'Poisoned Chalice of LLM Evaluation Competition,' which frames code contamination detection as a white-box membership inference task. The competition provides curated member/non-member datasets (derived from The Stack Edu and The Heap), target models (StarCoder2-3B/7B), baselines, and a held-out evaluation on Mellum-4B. Two submitted methods are described: SERSEM, which combines entropy-weighted token scoring with hidden-layer probing, and CalibratedProbs, which adjusts log-probability by token diversity. SERSEM achieves AUC-ROC of 0.773 on StarCoder2-3B and 0.753 on held-out Mellum, substantially outperforming probability-based baselines that cluster near random guessing (AUC 0.56–0.61). The paper is concise and the competition design is well-motivated.","tokens_in":8501,"tokens_out":1139,"duration_ms":2058000,"significance":"The paper makes a useful community contribution by organizing a competition with a held-out model and dataset, uniform execution environment, and curated data designed to remove keyword artifacts. The finding that simple likelihood-based MIA signals are insufficient for code contamination detection (AUC near 0.56–0.61) while structure-aware probing achieves substantially higher AUC is a meaningful empirical result. The reproducible evaluation setup and public dataset release are strengths. However, the central claim that SERSEM detects genuine membership rather than dataset-origin artifacts is not fully established by the current experimental design.","major_comments":[{"comment":"§2.2: The member set is drawn from The Stack Edu and the non-member set from The Heap. These are fundamentally different data sources with potentially different code distributions (comment density, identifier length, formatting conventions). The BoW curation step (§2.2) removes files distinguishable by keyword presence but does not address deeper distributional shifts. SERSEM specifically targets 'comments, long identifiers, string literals, formatting anomalies, and developer markers like TODO' (§3.1)—precisely the features most likely to differ systematically between educational/curated code (The Stack Edu) and general deduplicated code (The Heap). A method that detects 'is this file from The Stack Edu?' would generalize across both StarCoder2 and Mellum (both trained on The Stack v2) and achieve high AUC without performing genuine membership inference. The held-out Mellum evaluation (","section":null},{"comment":"Footnote 3: PAC was evaluated on only 1,000 samples while all other methods used 5,000. This 5× difference in evaluation set size creates non-comparable AUC estimates with potentially higher variance for PAC. While PAC is a baseline rather than the central submission, the paper presents all five methods side-by-side in Figure 1 and §4 without flagging this caveat in the discussion. A note in the results section (not just the footnote) and confidence intervals or standard errors would allow readers to assess whether the AUC differences between PAC (0.563–0.568) and the other baselines (0.591–0.613) are meaningful.","section":null}],"minor_comments":[{"comment":"§5: The conclusion states that 'limited generalizability and sensitivity to keyword drift identified in prior work become less problematic when such confounds are explicitly addressed during dataset construction.' Given that the dataset-origin confound (Major Comment 1) is not addressed, this claim is overstated. Consider softening.","section":null},{"comment":"§3.1 and §3.2: The AUC values reported in the method summaries (0.79 for SERSEM on both StarCoder2-3B and 7B) differ from the final evaluation values in §4 (0.773 on StarCoder2-3B). Clarify whether the §3 values are from participant-side experiments on the development set, while §4 values are from the held-out evaluation set.","section":null},{"comment":"§2.2: The paper states that The Heap was 'deduplicated against The Stack v2' using locality-sensitive hashes. No false negative rate or sensitivity analysis for the deduplication is reported. While this may be outside the scope of the competition report, a brief acknowledgment of this assumption's importance would strengthen the paper.","section":null},{"comment":"Figure 1: The ROC curves would benefit from confidence intervals or shaded regions, particularly given the different sample sizes across methods.","section":null},{"comment":"§2.4: The baselines are described briefly but methodological details are deferred to external references. A sentence or two on the perturbation strategy for PAC would improve readability.","section":null},{"comment":"The paper does not report per-language breakdowns of AUC, despite selecting five languages (Go, Java, Python, Ruby, Rust) 'to provide a broader overview of the generalizability of the submitted approaches across languages' (§2.2). A table or figure showing per-language performance should be included.","section":null}],"recommendation":"major_revision","confidential_remarks":"The dataset-origin confound is the most serious concern. The competition organizers are also the authors of The Heap (self-citation [11]), which is the sole source of non-member labels. While The Heap's construction is independently verifiable in principle, the paper would benefit from an explicit discussion of this potential conflict and why the BoW curation is insufficient to address distributional shifts. The competition format is valuable and the paper is appropriate for the FSE Competition Track, but the central claim about SERSEM's effectiveness needs to be disentangled from the source-dataset confound before publication."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the careful review and agree that the dataset-origin confound is a serious concern warranting explicit discussion in the revision. We accept the PAC sample-size caveat and will surface it more prominently. On the core confound issue, we agree the current design cannot fully rule it out and will revise the manuscript's claims accordingly; a complete fix requires additional experiments that are beyond what we can conduct within the revision window.","responses":[{"response":"The referee raises a valid and important concern. We acknowledge that the current experimental design cannot definitively distinguish genuine membership inference from detection of dataset-origin artifacts. The referee is correct that The Stack Edu (educational/curated code) and The Heap (general deduplicated code) may differ in distributional properties—comment density, identifier length, formatting conventions—that align precisely with the features SERSEM targets. The BoW classifier we employed removes files distinguishable by keyword presence, but it does not address deeper stylistic or structural distributional shifts. Furthermore, the referee's observation that both StarCoder2 and Mellum were trained on The Stack v2 (which contains The Stack Edu) means that a method detecting 'is this file from The Stack Edu?' would indeed generalize across both models and achieve high AUC without performing genuine membership inference. The held-out Mellum evaluation, which we presented as evidence of generalization, does not rule out this alternative explanation. We will revise the manuscript to explicitly acknowledge this limitation. Specifically: (1) We will add a dedicated discussion in §4 or §5 noting that the dataset construction creates a potential confound between membership status and dataset origin, and that SERSEM's feature design (targeting comments, long identifiers, developer markers) is precisely the type of approach that could exploit this confound. (2) We will soften the claim that SERSEM 'detects genuine membership' and instead frame the result as: SERSEM achieves high AUC, but the current design cannot determine whether this reflects genuine memorization signals or dataset-origin artifacts. (3) We will note that a definitive test would require member and non成员文件","revision_made":"partial","referee_comment":"§2.2: The member set is drawn from The Stack Edu and the non-member set from The Heap. These are fundamentally different data sources with potentially different code distributions. The BoW curation step removes keyword-distinguishable files but does not address deeper distributional shifts. SERSEM targets features (comments, long identifiers, formatting anomalies, TODO markers) that may differ systematically between educational/curated code and general deduplicated code. A method detecting 'is this from The Stack Edu?' would generalize across StarCoder2 and Mellum and achieve high AUC without performing genuine membership inference. The held-out Mellum evaluation does not rule this out."},{"response":"The referee is correct. The 5× difference in evaluation set size between PAC and all other methods is a methodological inconsistency that should be surfaced in the results discussion, not buried in a footnote. We will make two changes: (1) We will add an explicit caveat in §4 (the results discussion) noting that PAC was evaluated on 1,000 samples due to runtime constraints, while all other methods used 5,000 samples, and that this makes PAC's AUC estimates not directly comparable. (2) We will add confidence intervals (bootstrap or DeLong) for all AUC estimates in Figure 1 and the results table, so readers can assess whether the differences between PAC (0.563–0.568) and the other baselines (0.591–0.613) are within noise. We note that even with wider confidence intervals, PAC's AUC remains near random guessing and the qualitative conclusion—that simple probability-based and perturbation-based baselines are insufficient—does not change. But the referee is right that readers should be able to make this assessment themselves.","revision_made":"yes","referee_comment":"Footnote 3: PAC was evaluated on only 1,000 samples while all other methods used 5,000. This 5× difference creates non-comparable AUC estimates with potentially higher variance for PAC. The paper presents all five methods side-by-side in Figure 1 and §4 without flagging this caveat in the discussion. A note in the results section (not just the footnote) and confidence intervals or standard errors would allow readers to assess whether the AUC differences between PAC (0.563–0.568) and the other baselines (0.591–0.613) are meaningful."}],"tokens_in":7610,"tokens_out":1056,"duration_ms":203402,"standing_objections":["The dataset-origin confound (Major Comment 1) cannot be fully resolved within this revision. A definitive test would require member and non-member files drawn from the same source distribution (e.g., both from The Stack Edu, with non-members being files confirmed excluded from training), or a control experiment where SERSEM is evaluated on Stack Edu vs. Heap classification without model involvement. These experiments require new dataset construction and re-evaluation that are beyond what we can complete in the revision window. We will honestly acknowledge this limitation in the revised manuscript rather than claim it is resolved."]},"desk_editor":{"model":"glm-5.2","letter":"The main thing to know: this is a 4-page competition report for a white-box MIA challenge on code LLMs. The infrastructure is the real contribution — held-out model (Mellum), held-out dataset, bag-of-words curation to remove keyword artifacts, uniform execution environment. The finding that structure-aware methods (SERSEM, AUC 0.77) substantially outperform probability-based baselines (AUC 0.56–0.61) on curated code data is genuinely new and useful for the community. The competition design is sound, and the authors are honest that even the best method is far from practically reliable. Credit where earned: the held-out evaluation on Mellum is the right call, and requiring runnable replication packages from participants is good practice. The self-citation to The Heap [11] is appropriate here — that dataset's construction is independently verifiable, and using it as the non-member source is a reasonable design choice given the alternatives. Now the soft spots. The stress-test concern about a cross-dataset confound is the real issue and I think it lands. Members come from The Stack Edu, non-members from The Heap. These are different source datasets with potentially different code distributions — comment density, identifier conventions, formatting. The BoW curation step only removes keyword-level artifacts; it does not address deeper distributional shifts. SERSEM specifically targets comments, long identifiers, string literals, and developer markers — exactly the features most likely to differ systematically between educational code and general deduplicated code. A method that detects 'is this from The Stack Edu?' rather than 'was this memorized?' would generalize across both StarCoder2 and Mellum and achieve high AUC without doing genuine membership inference. The held-out Mellum evaluation does not resolve this: both models were trained on The Stack v2 (which includes The Stack Edu), so the same source-dataset distinction applies. The reader's concerns about missing error bars and the PAC sample size mismatch (1,000 vs 5,000) are valid but minor in comparison. Two submissions is thin, but that is the nature of a first-edition competition. Who benefits: researchers working on MIA for code LLMs, benchmark designers thinking about contamination detection, and anyone building evaluation infrastructure for code models. The competition design deserves attention even if the SERSEM result needs scrutiny. This deserves a serious referee. The infrastructure contribution is solid and the negative result (likelihood-based methods fail on curated data) is important. But the reviewer should push hard on the cross-dataset confound — it is the load-bearing concern that determines whether SERSEM's performance means what the paper claims it means.","headline":"Competition report showing structure-aware MIA beats likelihood-based methods on code, but a cross-dataset confound undermines the central claim.","tokens_in":8840,"tokens_out":611,"would_cite":true,"duration_ms":82593,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Structure beats statistics for detecting LLM memorization","keywords":["membership inference","data contamination","code language models","software engineering evaluation","white-box attacks","SERSEM","The Heap","StarCoder2"],"falsifier":"If a subsequent study found that The Heap contains files that are near-duplicates of training data (e.g., through contamination pathways not covered by the locality-sensitive hashing deduplication), the non-member labels would be incorrect, and the AUC scores for all methods, including SERSEM, would be unreliable. Alternatively, if SERSEM's performance dropped to the baseline range (AUC ~0.56-0.61) when tested on additional held-out models beyond StarCoder2 and Mellum, the generalization claim would not hold.","tokens_in":7910,"feed_emoji":"🔍","tokens_out":1232,"duration_ms":219834,"temperature":0.7,"pith_summary":"This paper reports the results of a competition designed to test whether one can reliably detect if a specific code file was part of a large language model's training data, without having access to the training corpus itself. The competition frames this as a white-box membership inference task: participants are given open-weight code models (StarCoder2-3B and 7B) and curated sets of files known to be either in or out of the training data, and must build a method that distinguishes them. The central finding is that simple likelihood-based methods, which check whether a model finds certain tokens easy or hard to predict, perform barely above random chance (AUC 0.56-0.61) on carefully curated code data. In contrast, a method called SERSEM, which combines entropy-weighted token scoring with probing of the model's internal hidden-layer representations and focuses on human-authored signals like comments and identifiers, achieves substantially better performance (AUC 0.773 on StarCoder2-3B, 0.753 on a held-out model). The paper establishes that detecting code contamination requires structural and representational analysis rather than surface-level probability signals, and that this approach generalizes across different models and programming languages.","feed_headline":"Detecting LLM code memorization requires structure, not statistics","feed_subtitle":"A competition shows simple probability signals barely beat random chance, but probing internal model layers and human-authored code signals,","key_machinery":"SERSEM (Selective Entropy-Weighted Scoring for Membership Inference), a method that down-weights predictable syntactic boilerplate, emphasizes human-authored signals such as comments and developer markers, and combines weighted token-level scoring from output logits with probing of intermediate transformer activations. The competition evaluation framework itself is also a central object: curated member/non-member datasets built from The Stack Edu (seen) and The Heap (unseen, deduplicated against The Stack v2), with a bag-of-words classifier used to filter out files distinguishable by keyword distribution shifts, plus a held-out model and dataset for final evaluation.","core_discovery":"The core discovery is that membership inference on source code separates into two performance regimes: simple probability-based signals (loss, MinK%Prob, calibrated probabilities, perturbation-based calibration) cluster near random guessing with AUC scores between 0.56 and 0.61 on curated data, while SERSEM's structure-aware approach combining entropy-weighted token scoring with hidden-layer probing reaches AUC 0.773 on StarCoder2-3B and 0.753 on the held-out Mellum model. This gap demonstrates that memorization in code LLMs is more detectable through internal model representations and human-centric code artifacts (comments, long identifiers, string literals) than through aggregate token-lik","pith_inferences":["If the gap between likelihood-based and structure-aware methods holds across more models and training scales, it would suggest that memorization in code models is fundamentally a representational phenomenon that surface-level outputs only weakly reflect, which could redirect the entire membership inference research agenda toward internal probing.","The dependency on The Heap's deduplication quality means that if near-duplicate contamination pathways exist (e.g., files appearing in training data through sources not covered by the deduplication), all reported AUC scores could be systematically biased. A sensitivity analysis on label noise would clarify whether the SERSEM-baseline gap is robust to this risk.","SERSEM's reliance on human-centric signals (comments, identifiers, TODO markers) implies that contamination detection may be easier in code with high developer-authored content and harder in boilerplate-heavy or auto-generated code, which could bias contamination assessments toward certain project types."],"forward_implications":["If SERSEM's approach generalizes as reported, benchmark maintainers could run it on their evaluation sets against candidate models to estimate contamination risk without needing access to training corpora, which is especially relevant for closed or partially closed models.","The finding that hidden-layer representations carry stronger memorization signals than output probabilities suggests that future code LLMs designed to resist membership inference may need to address internal representation leakage, not just output-level calibration.","The precision-recall trade-off noted in the conclusion implies that practical contamination detection may be best used as a triage tool (identifying suspicious files for manual review) rather than a definitive classifier, since high-confidence detection requires accepting low recall.","The competition's dataset curation methodology (using a bag-of-words classifier to remove distribution-shift artifacts) could become a standard preprocessing step for future membership inference benchmarks across modalities beyond code."],"fun_headline_variants":["Structure-aware probing exposes LLM code memorization where probability fails","Code memorization detection fails with statistics but succeeds with structure","Detecting LLM code contamination requires structural probing, not statistics","Hidden-layer probing beats statistics for detecting LLM code memorization","Probability signals fail at LLM code memorization; structure succeeds"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The ground truth labels for non-membership depend entirely on The Heap being truly free of contamination from the training corpora of all target models. The Heap was deduplicated against The Stack v2 using locality-sensitive hashing, but if that deduplication missed near-duplicates or if indirect contamination pathways exist through other training data sources, the member/non-member labels would be noisy and all AUC scores would be affected.","fun_headline_variants_meta":{"raw":{"variants":["Structure-aware probing exposes LLM code memorization where probability fails","Code memorization detection fails with statistics but succeeds with structure","Detecting LLM code contamination requires structural probing, not statistics","Hidden-layer probing beats statistics for detecting LLM code memorization","Probability signals fail at LLM code memorization; structure succeeds","Internal model representations outperform statistics for code memorization detection"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":1724,"prompt_tokens":529,"completion_tokens":1195,"prompt_tokens_details":null},"tokens_in":529,"tokens_out":1195,"duration_ms":52622,"temperature":1.0,"reasoning_tokens":1153,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T09:29:33.820551+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a subsequent study found that The Heap contains files that are near-duplicates of training data (e.g., through contamination pathways not covered by the locality-sensitive hashing deduplication), the non-member labels would be incorrect, and the AUC scores for all methods, including SERSEM, would be unreliable. Alternatively, if SERSEM's performance dropped to the baseline range (AUC ~0.56-0.61) when tested on additional held-out models beyond StarCoder2 and Mellum, the generalization claim would not hold.","supporting_citations":[],"review_version":1}