{"id":"aac0cb8c-2ca3-488a-9038-f8a1f0debcb9","arxiv_id":"2412.16594","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AIGCodeSet adds 7,583 labeled Python code samples from CodeLlama, Codestral, and Gemini 1.5 Flash, and its benchmarks show a Bayesian classifier achieves the highest recall.","lead":"This paper introduces AIGCodeSet, a dataset of Python code where each sample is labeled human-written or AI-generated, built from three AI models and three coding tasks. It also reports baseline detection results, with a Bayesian classifier finding the most AI-generated code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 80/20 split in §4.1 is at snippet level, so the same CodeNet problem and sometimes the exact human source code appear in both train and test; the reported Bayes advantage may be an artifact of problem-level leakage rather than genuine detector skill.","rationale":"The paper makes a useful dataset contribution: AIGCodeSet is described in enough detail to be reusable, the data is shared, and the inclusion of three LLMs and three usage scenarios is a reasonable design choice. However, the abstract and conclusion rest on the experimental claim that the Bayes classifier outperforms other baselines. The reader identified the single random split as the main weakness. My stress-test finds a more specific and more damaging version of that concern: the split is at snippet level, not problem level, and the dataset construction creates explicit pairings between human source codes and AI-fixed codes. Because all snippets from a problem share problem-specific content, and because fix-scenario AI codes are derived from human codes that remain in the dataset, a snippet-level split can leak the very signal the detector is supposed to learn. This does not invalidate the dataset itself, but it means the reported ranking may not generalize to unseen problems. The reader's verdict of CONDITIONAL remains appropriate after this analysis, so I recommend no change to the verdict. The concrete test—problem-grouped cross-validation with source-code exclusion—would settle whether the Bayes advantage is real or a leakage artifact.","tokens_in":7913,"tokens_out":4273,"duration_ms":40160,"concrete_test":"Re-run the Table 3 comparison using leave-one-problem-out cross-validation (317 folds, grouping all human and AI snippets by CodeNet problem), and in the fix scenarios exclude from training the exact human source code that was used as the LLM prompt. If the Bayes classifier's F1/recall advantage over SVM with Ada embeddings shrinks or reverses, the snippet-level split was the cause. Report mean±std over folds, not a single random split.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states 'We randomly sample 80% of the dataset' with no grouping by the 317 CodeNet problems, so the split is at the level of individual code snippets. Because AIGCodeSet contains up to 24 snippets per problem (5 accepted, 5 runtime-error, and 5 wrong-answer human codes, plus up to 9 AI-generated codes), each problem appears in both training and test. Detectors using TF-IDF, Ada embeddings, or Bayes token features can memorize problem-specific identifiers and algorithmic structure, letting them appear to distinguish AI from human without learning a general stylistic signal. More directly, in the two 'fix' scenarios (§3.2), each AI-generated snippet is produced from a randomly selected human-written snippet that is itself in the dataset; if that source snippet is in the training split and its fixed counterpart is in the test split, a model can exploit near-duplicate content. Consequently, the F1/recall gap between the Bayes classifier (0.63/0.81) and SVM with Ada embeddings (0.50/0.37) in Table 3 may be inflated by split leakage rather than by detector quality. The central claim that 'a Bayesian classifier outperforms the other models' therefore depends on split independence, which the paper does not establish.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AIGCodeSet, a dataset of 7,583 Python code snippets: 4,755 human-written solutions from 317 CodeNet problems (five each for accepted, runtime-error, and wrong-answer statuses) and 2,828 AI-generated snippets produced by CodeLlama 34B, Codestral 22B, and Gemini 1.5 Flash under three prompting scenarios (generate from scratch, fix a runtime error, fix a wrong output). After filtering invalid outputs, the authors provide descriptive statistics and evaluate baseline detectors (RF/XGB/SVM with Ada embeddings or TF-IDF, and a Bayesian classifier from prior work). They report that the Bayesian classifier achieves the highest F1 (0.63) and recall (0.81) on a single 80/20 split, with recall varying by LLM and by usage scenario.","tokens_in":8126,"tokens_out":6654,"duration_ms":54615,"significance":"If the evaluation is made robust, AIGCodeSet is a useful public resource: it covers three LLMs that are less frequently studied in this setting, includes three code-generation usage scenarios, documents the prompts and filtering procedure, and makes the data available for reuse. The paper’s structural analyses, such as the comparison of code length, comments, and function definitions in Figure 1, are informative. However, the central empirical claim—that the Bayesian classifier outperforms the other baselines—is currently not established because the evaluation split is at snippet level and permits problem-level and source-target leakage; a corrected evaluation could change the ranking. The dataset contribution itself remains valuable, and the paper is appropriate in scope for the venue if the experimental evidence is strengthened.","major_comments":[{"comment":"The single 80/20 split in §4.1 is random at the individual-snippet level and does not group by CodeNet problem. Because AIGCodeSet contains up to 24 snippets per problem (five accepted, five runtime-error, and five wrong-answer human codes, plus up to nine AI-generated codes), each problem is represented in both training and test partitions. TF-IDF, Ada-embedding, and Bayesian token features can exploit problem-specific identifiers and algorithmic structure, which can inflate the reported F1/recall advantage of the Bayesian classifier (0.63/0.81) over SVM with Ada embeddings (0.50/0.37) in Table 3. Please re-run the experiments with a problem-grouped split (all snippets of a problem in one partition) or with repeated stratified cross-validation, and report variance or confidence intervals.","section":"§4.1, Table 3"},{"comment":"In the two “fix” scenarios, each AI-generated snippet is produced by prompting the model with a randomly selected human-written snippet that is itself in AIGCodeSet. If the source human snippet falls in the training split and the corresponding AI-fixed snippet falls in the test split, the test item is nearly a duplicate of training content after a light edit; a detector can then recognize the source code rather than learn a general AI style. The authors should quantify how often this exact source-target overlap occurs in the reported split and present results with such pairs removed, or force source and target into the same partition.","section":"§3.2, §4.1"},{"comment":"The filtering step in §3.3 removes only outputs that are not Python, empty, or meaningless; it does not verify that the generated code is correct or that the “fix” prompts actually fixed the runtime/wrong-answer errors. If a large share of the generated code is incorrect, the classifiers may be separating incorrect code from correct code rather than AI-generated from human-written code. Please report, at least on a sample, the correctness of the AI-generated snippets under each scenario, or add this as an explicit limitation before interpreting the detection results.","section":"§3.3"},{"comment":"All reported results come from a single random 80/20 split, with no standard deviations, confidence intervals, or significance tests over multiple splits. Given the small differences among several baselines, the ranking may be within random variation; the authors should report repeated-split or cross-validated results and a paired significance test (for example, McNemar’s test) for the headline comparison between the Bayesian classifier and the best embedding-based baseline.","section":"§4.1, Tables 3–5"}],"minor_comments":[{"comment":"The numbers “2.828” and “2,828” are used inconsistently; please use one thousands-separator convention throughout.","section":"Abstract, §3.3, §5"},{"comment":"The sentence “we asked the models to generate code from scratch by giving only the problem description as the correspond to the accepted codes” is ungrammatical and should be rewritten.","section":"§3.2"},{"comment":"The word “filei” in the first paragraph is a typo and should be “file.”","section":"§5"},{"comment":"The Codestral reference contains “accessed 14-November-20s24”; this should be corrected to a proper date such as “14 November 2024.”","section":"References"},{"comment":"The tables label their values as “percentage” but report proportions between 0 and 1 (for example, 0.74); please either convert to percentages or relabel the columns as proportions.","section":"Tables 4 and 5"},{"comment":"The phrase “Our observation are as follows” should be “Our observations are as follows.”","section":"§4.1.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the dataset is potentially reusable, but the central comparative claim depends on a split that may leak problem-level and source-target information. I recommend major revision rather than rejection because the issue can be addressed by regrouping the split, removing overlapping pairs, and reporting repeated-split or cross-validated results with significance testing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on AIGCodeSet. The dataset is the real contribution: 7.5K Python snippets from CodeNet with three LLMs (CodeLlama 34B, Codestral 22B, Gemini 1.5 Flash) and three usage scenarios including fixing runtime errors and wrong outputs. That combination is not in prior datasets I know, and they shared the data on HuggingFace. The construction is described in enough detail to reproduce, and the filtering is sensible.\n\nThe evaluation is weaker than the dataset. The 80/20 split in §4.1 is at snippet level, not grouped by the 317 CodeNet problems, so the same problem (and sometimes the exact human source code used to generate an AI-fixed snippet) appears in both train and test. That means the reported F1 and recall numbers measure within-problem memorization to a nontrivial degree, not detector skill on unseen problems. The stress-test concern about leakage is credible, though I'd phrase it as 'performance is likely inflated' rather than asserting the Bayes advantage is an artifact. The authors should rerun with problem-disjoint splits and report variance. They also don't report significance tests; the gap between Bayes F1 0.63 and SVM-Ada 0.50 could easily shrink on different splits.\n\nTwo more soft spots. The abstract says 'a Bayesian classifier outperforms the other models,' but on accuracy and precision it doesn't (SVM with Ada has higher accuracy and precision). What it has is high recall. That claim needs qualifying. And the AI-generated codes were only checked for syntactic validity, not whether they actually fix the error or produce correct output; that's fine for detection, but it should be stated clearly.\n\nNone of this kills the dataset. A reader who wants to train or evaluate AI-code detectors on new types of prompts, especially fix-style prompts, will find it useful. The paper deserves peer review, but it needs major revision: problem-level cross-validation, significance results, and a more careful headline claim.","headline":"A genuinely useful public dataset with a weak evaluation: the Bayesian-classifier headline is suspect because the train/test split leaks problem-level information.","tokens_in":8662,"tokens_out":4222,"would_cite":true,"duration_ms":38950,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces AIGCodeSet, a 7,583-sample Python dataset pairing human-written solutions with AI-generated code from three language models, and reports that a Bayesian classifier flags the most AI-written code among six baselines…","keywords":["AI-generated code detection","Python code dataset","CodeLlama","Codestral","Gemini 1.5 Flash","Bayesian classifier","benchmark","CodeNet"],"falsifier":"Re-run the same six detectors on AIGCodeSet under repeated random 80/20 splits or k-fold cross-validation and test whether the Bayesian classifier's F1 stays above the SVM-with-Ada-embeddings result; if the 0.13 F1 gap collapses or reverses across splits, the claimed outperformance is an artifact of the single split.","tokens_in":7707,"feed_emoji":"🤖","tokens_out":9074,"duration_ms":71132,"temperature":0.7,"pith_summary":"AIGCodeSet is an annotated dataset for the task of telling whether a piece of Python code was written by a person or generated by an AI model. It pairs 4,755 human-written solutions, drawn from 317 coding problems, with 2,828 outputs from three large language models used in three different ways: generating from a problem statement, fixing a runtime error, and correcting a wrong answer. The authors report baseline results from six detectors and find that a Bayesian classifier, a token-probability method, achieves the highest F1 (0.63) and recall (0.81). This makes the dataset a public benchmark for a problem with clear stakes in education, hiring, and code trust.","feed_headline":"AI-code detector with 81% recall leads new dataset benchmark","feed_subtitle":"AIGCodeSet pairs 7,583 human and AI Python snippets; token-frequency Bayes classifier beats embedding models on recall.","key_machinery":"The load-bearing mechanism is the controlled construction of AIGCodeSet itself: 317 problems, each with up to fifteen human submissions (five accepted, five runtime-error, five wrong-answer) and up to nine AI submissions (three models times three usage scenarios), so comparisons are matched by problem and scenario. On top of that, the experiments compare three feature views of code—Ada embeddings, TF-IDF vectors, and the token-probability Bayesian classifier—which together define the baseline landscape. The Bayesian classifier carries the main result: it detects 81% of AI-generated code, its lower precision showing that it trades false positives for sensitivity.","core_discovery":"On the paper's own terms, the discovery is that a deliberately varied, reproducible dataset for AI-generated-code detection is feasible, and that on that dataset a simple Bayesian classifier outperforms embedding- and TF-IDF-based models at catching AI-written code. The dataset is constructed so that human and AI code answer identical problem statements, and it deliberately includes harder cases: AI code produced by fixing human-written buggy code, which the experiments show resembles human style and drops every detector's recall. The authors also find that detection difficulty is not uniform: Gemini 1.5 Flash output is the hardest to detect, and code generated from scratch is much easier to detect than code produced in the two fixing scenarios.","pith_inferences":["Going beyond the paper, the reported ranking rests on a single random 80/20 split, so repeated splits or cross-validation could move the F1 scores; the 0.13 gap between the Bayesian classifier and the best SVM variant has no reported variance.","A natural testbed extension would be to regenerate the same 317 problems with newer or larger models and check whether the Bayesian baseline's recall degrades, revealing how quickly detectors need recalibration.","The fixing scenarios suggest real-world detection is closer to an authorship-attribution problem than to pure generation detection, so features such as edit distance to the original human code or structural diffs may outperform static token features."],"forward_implications":["AIGCodeSet gives researchers a public, problem-matched testbed for comparing AI-generated-code detectors on Python, with both accepted and buggy human submissions.","The Bayesian classifier's high recall suggests token-frequency patterns are a strong signal for catching AI output, though its precision (0.52) means it also flags many human-written solutions.","Reports of detector performance should separate results by generating model, because Gemini-generated code is noticeably harder to detect than CodeLlama or Codestral output.","Datasets that only include from-scratch generation will overstate how well detectors work in practice, since code produced by fixing existing human code is much harder to identify.","The released dataset and code allow the baselines to be reproduced and extended, making the reported F1 and recall numbers checkable by other groups."],"supporting_citations":[{"why":"Prior studies that used CodeNet, cited as the basis for choosing it as the source of human-written solutions.","marker":"Xu and Sheng, 2025, 2024"},{"why":"Defines the CodeLlama 34B model that generated one third of the AI code samples.","marker":"Roziere et al., 2023"},{"why":"Defines the Codestral 22B model that generated one third of the AI code samples.","marker":"team, 2024"},{"why":"Defines the Gemini 1.5 Flash model that generated one third of the AI code samples.","marker":"Team et al., 2024"},{"why":"Supplies the TF-IDF vector sizing and the Bayesian classifier implementation that the experiments reuse as their strongest baseline.","marker":"Oedingen et al., 2024"}],"fun_headline_variants":["New dataset for detecting AI-generated code: Bayes wins","AIGCodeSet: 7,583 Python codes to train AI-detection models","Bayesian classifier tops deep models in AI-code detection test","Hardest AI code to spot: when AI fixes human bugs","AI-code detection dataset: simple Bayes beats fancy models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the single random 80/20 split of AIGCodeSet represents the dataset well enough to rank the detectors; if that split is unrepresentative, the reported performance ordering could change.","fun_headline_variants_meta":{"raw":{"variants":["New dataset for detecting AI-generated code: Bayes wins","AIGCodeSet: 7,583 Python codes to train AI-detection models","Bayesian classifier tops deep models in AI-code detection test","Hardest AI code to spot: when AI fixes human bugs","AI-code detection dataset: simple Bayes beats fancy models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000576,"raw_usage":{"total_tokens":2641,"prompt_tokens":792,"completion_tokens":1849,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":408,"completion_tokens_details":{"reasoning_tokens":1763}},"tokens_in":408,"tokens_out":1849,"duration_ms":11602,"temperature":1.0,"reasoning_tokens":1763,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:25:52.529711+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same six detectors on AIGCodeSet under repeated random 80/20 splits or k-fold cross-validation and test whether the Bayesian classifier's F1 stays above the SVM-with-Ada-embeddings result; if the 0.13 F1 gap collapses or reverses across splits, the claimed outperformance is an artifact of the single split.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the TF-IDF vector sizing and the Bayesian classifier implementation that the experiments reuse as their strongest baseline."}],"review_version":1}