AIGCodeSet: A New Annotated Dataset for AI Generated Code Detection

Basak Demirok; Mucahid Kutlu

REVIEW 4 major objections 6 minor 2 cited by

AIGCodeSet: A New Annotated Dataset for AI Generated Code Detection

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces AIGCodeSet, a 7,583-sample Python dataset pairing human-written solutions with AI-generated code from three language models, and reports that a Bayesian classifier flags the most AI-written code among six baselines…

desk verdict A genuinely useful public dataset with a weak evaluation: the Bayesian-classifier headline is suspect because the train/test split leaks problem-level information. read the letter →

arxiv 2412.16594 v3 pith:XAYZRRXQ submitted 2024-12-21 cs.SE cs.AI

Basak Demirok , Mucahid Kutlu This is my paper · ORCID

classification cs.SEcs.AI

keywords AI-generatedcodedetectionPythondatasetLlamaCodestralGemini1.5FlashBayesianclassifierbenchmarkNet

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AIGCodeSet is an annotated dataset for the task of telling whether a piece of Python code was written by a person or generated by an AI model. It pairs 4,755 human-written solutions, drawn from 317 coding problems, with 2,828 outputs from three large language models used in three different ways: generating from a problem statement, fixing a runtime error, and correcting a wrong answer. The authors report baseline results from six detectors and find that a Bayesian classifier, a token-probability method, achieves the highest F1 (0.63) and recall (0.81). This makes the dataset a public benchmark for a problem with clear stakes in education, hiring, and code trust.

What carries the argument

The load-bearing mechanism is the controlled construction of AIGCodeSet itself: 317 problems, each with up to fifteen human submissions (five accepted, five runtime-error, five wrong-answer) and up to nine AI submissions (three models times three usage scenarios), so comparisons are matched by problem and scenario. On top of that, the experiments compare three feature views of code—Ada embeddings, TF-IDF vectors, and the token-probability Bayesian classifier—which together define the baseline landscape. The Bayesian classifier carries the main result: it detects 81% of AI-generated code, its lower precision showing that it trades false positives for sensitivity.

What would settle it

Re-run the same six detectors on AIGCodeSet under repeated random 80/20 splits or k-fold cross-validation and test whether the Bayesian classifier's F1 stays above the SVM-with-Ada-embeddings result; if the 0.13 F1 gap collapses or reverses across splits, the claimed outperformance is an artifact of the single split.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a deliberately varied, reproducible dataset for AI-generated-code detection is feasible, and that on that dataset a simple Bayesian classifier outperforms embedding- and TF-IDF-based models at catching AI-written code. The dataset is constructed so that human and AI code answer identical problem statements, and it deliberately includes harder cases: AI code produced by fixing human-written buggy code, which the experiments show resembles human style and drops every detector's recall. The authors also find that detection difficulty is not uniform: Gemini 1.5 Flash output is the hardest to detect, and code generated from scratch is much easier to detect than code produced in the two fixing scenarios.

Load-bearing premise

The load-bearing assumption is that the single random 80/20 split of AIGCodeSet represents the dataset well enough to rank the detectors; if that split is unrepresentative, the reported performance ordering could change.

Editorial extensions

If this is right

AIGCodeSet gives researchers a public, problem-matched testbed for comparing AI-generated-code detectors on Python, with both accepted and buggy human submissions.
The Bayesian classifier's high recall suggests token-frequency patterns are a strong signal for catching AI output, though its precision (0.52) means it also flags many human-written solutions.
Reports of detector performance should separate results by generating model, because Gemini-generated code is noticeably harder to detect than CodeLlama or Codestral output.
Datasets that only include from-scratch generation will overstate how well detectors work in practice, since code produced by fixing existing human code is much harder to identify.
The released dataset and code allow the baselines to be reproduced and extended, making the reported F1 and recall numbers checkable by other groups.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Going beyond the paper, the reported ranking rests on a single random 80/20 split, so repeated splits or cross-validation could move the F1 scores; the 0.13 gap between the Bayesian classifier and the best SVM variant has no reported variance.
A natural testbed extension would be to regenerate the same 317 problems with newer or larger models and check whether the Bayesian baseline's recall degrades, revealing how quickly detectors need recalibration.
The fixing scenarios suggest real-world detection is closer to an authorship-attribution problem than to pure generation detection, so features such as edit distance to the original human code or structural diffs may outperform static token features.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take on AIGCodeSet. The dataset is the real contribution: 7.5K Python snippets from CodeNet with three LLMs (CodeLlama 34B, Codestral 22B, Gemini 1.5 Flash) and three usage scenarios including fixing runtime errors and wrong outputs. That combination is not in prior datasets I know, and they shared the data on HuggingFace. The construction is described in enough detail to reproduce, and the filtering is sensible.

The evaluation is weaker than the dataset. The 80/20 split in §4.1 is at snippet level, not grouped by the 317 CodeNet problems, so the same problem (and sometimes the exact human source code used to generate an AI-fixed snippet) appears in both train and test. That means the reported F1 and recall numbers measure within-problem memorization to a nontrivial degree, not detector skill on unseen problems. The stress-test concern about leakage is credible, though I'd phrase it as 'performance is likely inflated' rather than asserting the Bayes advantage is an artifact. The authors should rerun with problem-disjoint splits and report variance. They also don't report significance tests; the gap between Bayes F1 0.63 and SVM-Ada 0.50 could easily shrink on different splits.

Two more soft spots. The abstract says 'a Bayesian classifier outperforms the other models,' but on accuracy and precision it doesn't (SVM with Ada has higher accuracy and precision). What it has is high recall. That claim needs qualifying. And the AI-generated codes were only checked for syntactic validity, not whether they actually fix the error or produce correct output; that's fine for detection, but it should be stated clearly.

None of this kills the dataset. A reader who wants to train or evaluate AI-code detectors on new types of prompts, especially fix-style prompts, will find it useful. The paper deserves peer review, but it needs major revision: problem-level cross-validation, significance results, and a more careful headline claim.

Referee Report

4 major / 6 minor

Summary. The paper introduces AIGCodeSet, a dataset of 7,583 Python code snippets: 4,755 human-written solutions from 317 CodeNet problems (five each for accepted, runtime-error, and wrong-answer statuses) and 2,828 AI-generated snippets produced by CodeLlama 34B, Codestral 22B, and Gemini 1.5 Flash under three prompting scenarios (generate from scratch, fix a runtime error, fix a wrong output). After filtering invalid outputs, the authors provide descriptive statistics and evaluate baseline detectors (RF/XGB/SVM with Ada embeddings or TF-IDF, and a Bayesian classifier from prior work). They report that the Bayesian classifier achieves the highest F1 (0.63) and recall (0.81) on a single 80/20 split, with recall varying by LLM and by usage scenario.

Significance. If the evaluation is made robust, AIGCodeSet is a useful public resource: it covers three LLMs that are less frequently studied in this setting, includes three code-generation usage scenarios, documents the prompts and filtering procedure, and makes the data available for reuse. The paper’s structural analyses, such as the comparison of code length, comments, and function definitions in Figure 1, are informative. However, the central empirical claim—that the Bayesian classifier outperforms the other baselines—is currently not established because the evaluation split is at snippet level and permits problem-level and source-target leakage; a corrected evaluation could change the ranking. The dataset contribution itself remains valuable, and the paper is appropriate in scope for the venue if the experimental evidence is strengthened.

major comments (4)

[§4.1, Table 3] The single 80/20 split in §4.1 is random at the individual-snippet level and does not group by CodeNet problem. Because AIGCodeSet contains up to 24 snippets per problem (five accepted, five runtime-error, and five wrong-answer human codes, plus up to nine AI-generated codes), each problem is represented in both training and test partitions. TF-IDF, Ada-embedding, and Bayesian token features can exploit problem-specific identifiers and algorithmic structure, which can inflate the reported F1/recall advantage of the Bayesian classifier (0.63/0.81) over SVM with Ada embeddings (0.50/0.37) in Table 3. Please re-run the experiments with a problem-grouped split (all snippets of a problem in one partition) or with repeated stratified cross-validation, and report variance or confidence intervals.
[§3.2, §4.1] In the two “fix” scenarios, each AI-generated snippet is produced by prompting the model with a randomly selected human-written snippet that is itself in AIGCodeSet. If the source human snippet falls in the training split and the corresponding AI-fixed snippet falls in the test split, the test item is nearly a duplicate of training content after a light edit; a detector can then recognize the source code rather than learn a general AI style. The authors should quantify how often this exact source-target overlap occurs in the reported split and present results with such pairs removed, or force source and target into the same partition.
[§3.3] The filtering step in §3.3 removes only outputs that are not Python, empty, or meaningless; it does not verify that the generated code is correct or that the “fix” prompts actually fixed the runtime/wrong-answer errors. If a large share of the generated code is incorrect, the classifiers may be separating incorrect code from correct code rather than AI-generated from human-written code. Please report, at least on a sample, the correctness of the AI-generated snippets under each scenario, or add this as an explicit limitation before interpreting the detection results.
[§4.1, Tables 3–5] All reported results come from a single random 80/20 split, with no standard deviations, confidence intervals, or significance tests over multiple splits. Given the small differences among several baselines, the ranking may be within random variation; the authors should report repeated-split or cross-validated results and a paired significance test (for example, McNemar’s test) for the headline comparison between the Bayesian classifier and the best embedding-based baseline.

minor comments (6)

[Abstract, §3.3, §5] The numbers “2.828” and “2,828” are used inconsistently; please use one thousands-separator convention throughout.
[§3.2] The sentence “we asked the models to generate code from scratch by giving only the problem description as the correspond to the accepted codes” is ungrammatical and should be rewritten.
[§5] The word “filei” in the first paragraph is a typo and should be “file.”
[References] The Codestral reference contains “accessed 14-November-20s24”; this should be corrected to a proper date such as “14 November 2024.”
[Tables 4 and 5] The tables label their values as “percentage” but report proportions between 0 and 1 (for example, 0.74); please either convert to percentages or relabel the columns as proportions.
[§4.1.1] The phrase “Our observation are as follows” should be “Our observations are as follows.”

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the dataset and baseline evaluations are self-contained, with no fitted input renamed as prediction.

full rationale

The paper's contribution is the construction of AIGCodeSet and the evaluation of baseline detectors on it. The Bayesian classifier is imported from prior work (Oedingen et al., 2024) and applied to the new dataset, not derived from the dataset or fitted to the target labels. No equation in the paper defines a predicted quantity in terms of an input quantity by construction, and no parameter is fitted to a subset of data and then reported as a prediction on a closely related quantity. The central claim that the Bayesian classifier outperforms other baselines is an empirical comparison obtained by running existing methods on a held-out test split. The potential problem-level leakage from the 80/20 random split is a legitimate experimental validity concern, but it is not a circularity: the reported scores could be inflated without the derivation chain collapsing into its inputs. There are no load-bearing self-citations: the cited prior implementation is external and its effectiveness is not asserted as a theorem that forces the paper's result. Therefore, no circular step is identified and the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the representativeness of the CodeNet-derived samples and the stability of a single data split; no free parameters are fitted to produce the reported scores.

assumptions (4)

domain assumption CodeNet Python Benchmark human submissions are representative of human-written Python code in general.
Section 3.1 uses only CodeNet competitive-programming solutions; these may differ from typical software engineering or student code.
domain assumption The three selected LLMs (CodeLlama 34B, Codestral 22B, Gemini 1.5 Flash) produce code representative of AI-generated code.
Section 3.2; generalizing beyond these models is an assumption.
domain assumption The Bayes classifier implementation from Oedingen et al. (2024) is correctly applied to this dataset.
Section 4.1 reuses their implementation without modification or verification.
domain assumption A single 80/20 random split provides reliable performance estimates.
Section 4.1 Experimental Setup; no repeated splits or confidence intervals are reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AIGCodeSet: A New Annotated Dataset for AI Generated Code Detection." pith.science (2026). https://pith.science/paper/XAYZRRXQ

@misc{pith2026241216594,
  author       = {Pith},
  title        = {Pith review of: AIGCodeSet: A New Annotated Dataset for AI Generated Code Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XAYZRRXQ}},
  note         = {Machine review of arXiv:2412.16594}
}

read the original abstract

While large language models provide significant convenience for software development, they can lead to ethical issues in job interviews and student assignments. Therefore, determining whether a piece of code is written by a human or generated by an artificial intelligence (AI) model is a critical issue. In this study, we present AIGCodeSet, which consists of 2.828 AI-generated and 4.755 human-written Python codes, created using CodeLlama 34B, Codestral 22B, and Gemini 1.5 Flash. In addition, we share the results of our experiments conducted with baseline detection methods. Our experiments show that a Bayesian classifier outperforms the other models.

Figures

Figures reproduced from arXiv: 2412.16594 by the authors.

**Figure 1.** Human written codes vs. Codes generated from scratch in AIGCodeSet. Human-written code is typically more extensive than code generated by LLMs. Among the LLMs, CodeLlama defines more functions than both humans and the other models. Codestral, on the other hand, uses blank lines and comments more frequently, aligning more closely with human-written code in this regard. 4 Experiments In this section, we explain our … view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Zero-Shot Detection of LLM-Generated Code via Approximated Task Conditioning
cs.CL 2025-06 conditional novelty 7.0 of 10

ATC detects AI-generated code by asking a language model to reconstruct the programming task, then scoring token entropy under that reconstructed task, outperforming prior zero-shot detectors on Python, C++, and Java ...
CodeMirage: A Multi-Lingual Benchmark for Detecting AI-Generated and Paraphrased Source Code from Production-Level LLMs
cs.SE 2025-05 conditional novelty 6.0 of 10

CodeMirage is a ten-language, ten-LLM benchmark with original and paraphrased AI code, and it shows current AI-generated-code detectors drop sharply under cross-model and low-false-alarm settings.

Reference graph

Works this paper leans on

21 extracted references · 11 canonical work pages · cited by 2 Pith papers

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Sri Haritha Ambati, Norah Ridley, Enrico Branca, and Natalia Stakhanova. 2024. Navigating (in) security of ai-generated code. In 2024 IEEE International Conference on Cyber Security and Resilience (CSR), pages 1--8. IEEE

work page 2024
[4]

Xuefei Deng and KD Joshi. 2024. Promoting ethical use of generative ai in education. ACM SIGMIS Database: the DATABASE for Advances in Information Systems, 55(3):6--11

work page 2024
[5]

Andrea Gurioli, Maurizio Gabbrielli, and Stefano Zacchiroli. 2024. Is this you, llm? recognizing ai-written programs with multilingual code stylometry. arXiv preprint arXiv:2412.14611

work page arXiv 2024
[6]

Oseremen Joy Idialu, Noble Saji Mathews, Rungroj Maipradit, Joanne M Atlee, and Mei Nagappan. 2024. Whodunit: Classifying code as human authored or gpt-4 generated-a case study on codechef problems. In Proceedings of the 21st International Conference on Mining Software Repositories, pages 394--406

work page 2024
[7]

Marc Oedingen, Raphael C Engelhardt, Robin Denz, Maximilian Hammer, and Wolfgang Konen. 2024. Chatgpt code detection: Techniques for uncovering the source of code. arXiv e-prints, pages arXiv--2405

work page 2024
[8]

Wei Hung Pan, Ming Jie Chok, Jonathan Leong Shan Wong, Yung Xin Shin, Yeong Shian Poon, Zhou Yang, Chun Yong Chong, David Lo, and Mei Kuan Lim. 2024. Assessing ai detectors in identifying ai-generated code: Implications for education. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training, ...

work page 2024

Show all 21 references

[9]

Hung Pham, Huyen Ha, Van Tong, Dung Hoang, Duc Tran, and Tuyen Ngoc Le. 2024. Magecode: Machine-generated code detection method using large language models. IEEE Access

2024
[10]

Md Fazle Rabbi, Arifa Islam Champa, Minhaz F Zibran, and Md Rakibul Islam. 2024. Ai writes, we analyze: The chatgpt python code saga. In Proceedings of the 21st International Conference on Mining Software Repositories, pages 177--181

2024
[11]

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

2023 arXiv
[12]

Hyunjae Suh, Mahan Tafreshipour, Jiawei Li, Adithya Bhattiprolu, and Iftekhar Ahmed. 2024. An empirical study on automatically detecting ai-generated source code: How far are we? arXiv e-prints, pages arXiv--2411

2024
[13]

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

2024 arXiv
[14]

Mistral AI team. 2024. Codestral: Hello, world! empowering developers and democratising coding with mistral ai. https://mistral.ai/news/codestral/. [Online; accessed 14-November-20s24]

2024
[15]

Jian Wang, Shangqing Liu, Xiaofei Xie, and Yi Li. 2023. Evaluating aigc detectors on code content. arXiv preprint arXiv:2304.05193

2023 arXiv
[16]

Jiexin Wang, Xitong Luo, Liuwen Cao, Hongkui He, Hailin Huang, Jiayuan Xie, Adam Jatowt, and Yi Cai. 2024. Is your ai-generated code really safe? evaluating large language models on secure code generation with codeseceval. arXiv preprint arXiv:2407.02395

2024 arXiv
[17]

Zhenyu Xu and Victor S Sheng. 2024. Detecting ai-generated code assignments using perplexity of large language models. In Proceedings of the aaai conference on artificial intelligence, volume 38, pages 23155--23162

2024
[18]

Zhenyu Xu and Victor S Sheng. 2025. Codevision: Detecting llm-generated code using 2d token probability maps and vision models. arXiv preprint arXiv:2501.03288

2025 arXiv
[19]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[20]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[21]

WZ5Zyz ; <8 F^zŋM<yI_d H`=

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

1999

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Sri Haritha Ambati, Norah Ridley, Enrico Branca, and Natalia Stakhanova. 2024. Navigating (in) security of ai-generated code. In 2024 IEEE International Conference on Cyber Security and Resilience (CSR), pages 1--8. IEEE

work page 2024

[4] [4]

Xuefei Deng and KD Joshi. 2024. Promoting ethical use of generative ai in education. ACM SIGMIS Database: the DATABASE for Advances in Information Systems, 55(3):6--11

work page 2024

[5] [5]

Andrea Gurioli, Maurizio Gabbrielli, and Stefano Zacchiroli. 2024. Is this you, llm? recognizing ai-written programs with multilingual code stylometry. arXiv preprint arXiv:2412.14611

work page arXiv 2024

[6] [6]

Oseremen Joy Idialu, Noble Saji Mathews, Rungroj Maipradit, Joanne M Atlee, and Mei Nagappan. 2024. Whodunit: Classifying code as human authored or gpt-4 generated-a case study on codechef problems. In Proceedings of the 21st International Conference on Mining Software Repositories, pages 394--406

work page 2024

[7] [7]

Marc Oedingen, Raphael C Engelhardt, Robin Denz, Maximilian Hammer, and Wolfgang Konen. 2024. Chatgpt code detection: Techniques for uncovering the source of code. arXiv e-prints, pages arXiv--2405

work page 2024

[8] [8]

Wei Hung Pan, Ming Jie Chok, Jonathan Leong Shan Wong, Yung Xin Shin, Yeong Shian Poon, Zhou Yang, Chun Yong Chong, David Lo, and Mei Kuan Lim. 2024. Assessing ai detectors in identifying ai-generated code: Implications for education. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training, ...

work page 2024