Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

CodeMirage: A Multi-Lingual Benchmark for Detecting AI-Generated and Paraphrased Source Code from Production-Level LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AI-code detectors become impractical when false alarms must be rare, a 10-language benchmark shows.

desk verdict Useful and comprehensive benchmark for AI-generated code detection, but the quality-control claims outrun the checks and the dataset is missing. read the letter →

arxiv 2506.11059 v1 pith:MNHGAWTE submitted 2025-05-27 cs.SE cs.CLcs.CYcs.LG

Hanxi Guo , Siyuan Cheng , Kaiyuan Zhang , Guangyu Shen , Xiangyu Zhang This is my paper
classification cs.SEcs.CLcs.CYcs.LG
keywords AI-generatedcodedetectionbenchmarkparaphrasinglargelanguagemodelssourcemultilingualadversarialevaluationplagiarism
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces CodeMirage, a benchmark for AI-generated source-code detection that pairs 10,000 human-written code files with roughly 200,000 AI-generated and AI-paraphrased counterparts from ten production-level LLMs across ten programming languages. The authors argue that existing benchmarks cover too few languages, use weaker open-source generators, and lack adversarial paraphrasing, so detector performance on them overstates real-world readiness. Using CodeMirage, they evaluate ten detectors from four methodological paradigms under four increasingly difficult configurations, reporting F1, TPR at 10% false positives, and TPR at 1% false positives. Their central empirical finding is that although detectors look strong under in-distribution F1, nearly all of them catch very few AI files when false alarms are held to 1%, which they read as a gap between laboratory results and practical deployment. A sympathetic reader would care because the benchmark is an attempt to make detector evaluation reflect how developers actually use modern LLMs, including copy-paste and paraphrase-evasion behavior.

What carries the argument

The benchmark pipeline is the machinery. Human files from a cleaned GitHub corpus are summarized by LLMs into structural descriptions that deliberately omit the original implementation; ten LLMs then generate code from those summaries; a rule-based inspector enforces line/character-length similarity and requires BLEU below 0.5 to prevent recitation; and a code-specific paraphrasing prompt applies six transformation types while preserving functionality. On this scaffold, CodeMirage defines four evaluation configurations—in-distribution, paraphrase, cross-model, and cross-model paraphrase—each progressively harder, and reports F1 plus TPR at 10% and 1% false-positive rates. The configuration ladder is what lets the paper separate detector strength from distribution overfitting.

What would settle it

Sample a few hundred CodeMirage AI-generated and paraphrased files, attempt to compile or run them in their target environments (with syntax checking for HTML/PHP/Ruby and unit execution where possible), and have experienced developers rate whether the code looks like production work; if a large share fails to compile or is rated implausible, the claim that the benchmark reflects real production-level AI code fails, and detector scores on it would not transfer.

Watch

Extended reading notes

Core claim

CodeMirage's core claim is that it supplies the most realistic benchmark yet for AI-written code detection, because it matches real-world conditions on three axes at once: ten widely used languages, outputs from ten current production LLMs (including reasoning models from six providers), and code that has been adversarially paraphrased by domain-specific transformation prompts. The paper reports nine findings from running ten detectors on this material. The most consequential is Finding 9: with the false-positive rate capped at 1%, all evaluated detectors show a dramatic drop in true-positive rate, generally below 0.3, meaning current detectors would miss most AI-generated code in settings where flagging human code is expensive. A supporting claim is that fine-tuning-based detectors, especially those whose backbones were pre-trained on large code corpora, lead the pack in overall F1, while zero-shot token-statistics detectors perform worst. The authors also report that paraphrasing hurts detectors unevenly, with reasoning models' outputs becoming notably harder to detect after paraphrase.

Load-bearing premise

The benchmark's claim to be a realistic testbed rests on a filter that only checks length similarity and token-level divergence, never whether the generated or paraphrased code compiles, runs, or is code a developer would actually accept.

Editorial extensions

If this is right

  • Detectors with strong F1 scores in same-model held-out tests should not be trusted for deployment; low-alarm TPR is the metric that separates laboratory results from practical use.
  • Cross-model shifts hurt most fine-tuned detectors more than paraphrasing does, so generalization to unseen generators is a central weakness.
  • Fine-tuning-based detectors with code-pretrained backbones such as CodeT5+ are the strongest category, and AST-based embeddings beat raw-code embeddings, pointing to syntax-aware representations as the productive direction.
  • Reasoning models' outputs behave differently from non-reasoning models after paraphrasing, so benchmarks that exclude reasoning models can mis-estimate robustness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not stated in the paper, but a direct extension would be human-preference or compile-check validation of a random sample of CodeMirage's generated files; if many fail to compile, detector rankings on this benchmark could still overstate real-world performance.
  • The paper's low-alarm finding suggests a practical deployment rule: detectors should be tuned with TPR@FPR=1% as the objective rather than F1, and applications with high false-alarm cost should assume near-zero utility from current detectors.
  • Because the benchmark samples whole files, not mixed human/AI snippets, the authors' own limitation note implies the findings may not transfer to AI-assisted editing where an LLM generates partial completions; a mixed-granularity variant is the obvious next test.
  • The paraphrase prompt's rule of keeping lengths similar and preserving functionality is itself a clue that harder evasion (semantic-preserving, style-matched paraphrases) may push all detectors lower; the benchmark's nine findings should be read as an upper bound on robustness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. CodeMirage proposes a large multilingual benchmark for detecting AI-generated and AI-paraphrased source code. It samples 1,000 human files per language across ten languages from CodeParrot, summarizes each file, prompts ten production-level LLMs to generate code from the summaries, and then paraphrases the generated code with a code-specific prompt that applies seven transformation types. The resulting roughly 210k samples are used to evaluate ten detectors from four paradigms under in-distribution, paraphrase, cross-model, and cross-model-paraphrase configurations, with F1 and TPR@FPR metrics. The paper reports nine findings, including that in-distribution performance exceeds out-of-distribution performance, fine-tuned detectors lead overall, common languages are slightly easier, reasoning models degrade more after paraphrasing, and all detectors are impractical at low false-positive rates.

Significance. If the construction pipeline and evaluation are sound, CodeMirage fills a concrete gap: existing benchmarks cover fewer languages, rely on older or smaller generators, and largely omit paraphrased-code adversarial testing. The manuscript is unusually detailed about its pipeline—summarization prompts, generation prompts, the rule-based inspector, paraphrasing instructions, model configurations, and per-model/per-language counts are all given in appendices—and the evaluation spans four detector paradigms under four configurations, which is a genuine strength. The nine findings are concrete and falsifiable, and the TPR@FPR analysis addresses an important practical question. However, the value of the benchmark as a community resource is currently limited by the absence of a released dataset and by the lack of validation that generated and paraphrased samples are realistic, valid code; the former prevents adoption and reproducibility, the latter directly affects the interpretation of every detector score and finding.

major comments (4)
  1. [Section 3.1 / Appendices C-D] The rule-based inspector verifies only line-count/character-length consistency and BLEU < 0.5; it never checks whether generated or paraphrased code parses, compiles, runs, or preserves semantics. Appendix C shows that the generation prompt requests only a snippet with given line and character counts, and Appendix D's paraphrase prompt asks to keep length similar but does not require syntactic validity. Because the paper's central claim is that CodeMirage is a rigorous and practical testbed (Abstract, Section 1), and all nine findings assume the AI and paraphrased samples are realistic production-like code, this omission is load-bearing. Figure 2's AST and data-flow match scores do not rule out malformed code: tree-sitter's error recovery and CodeBLEU's token and partial-parse matching can produce moderate-to-high scores on imperfectly parsed or semantically incoherent code. I request a per-language syntactic validation of all generated and paraphrased samples (e.g., parse with tree-sitter and report the error rate; compile for C, C++, C#, Go, Java where feasible), a report of rejection rates in the inspector and post-filtering, and a sensitivity analysis of the nine findings when invalid samples are excluded. Appendix I lists several limitations but does not acknowledge this missing validation, which is the most consequential one for the benchmark's realism claim.
  2. [Section 4 / Figures 3-5] The comparative findings (Finding 2: fine-tuned methods outperform; Finding 3: CodeT5+ slightly outperforms GPTSniffer; Finding 5: Embed-AST marginally outperforms Embed-Code; Finding 7: GPT and Llama score slightly higher) are supported only by mean F1 values with error bars equal to one standard deviation across languages or LLMs. No statistical significance tests are reported. Differences of a few hundredths of F1 may be within the observed variance, especially given that the standard-deviation bars in Figures 3-5 appear to overlap for several adjacent detectors. Please add paired tests across the ten languages (or ten LLMs), such as Wilcoxon signed-rank tests or paired bootstrap intervals, report effect sizes and p-values (with appropriate multiple-comparison correction), and report variance across detector training seeds. Without these, the ranking claims are not yet firmly established, and they are a central deliverable of the paper.
  3. [Section 1 / Appendices B-D] The manuscript repeatedly refers to "our CodeMirage dataset" (e.g., Appendices B-D) but provides no URL, repository, or data-availability statement. Since the contribution is explicitly a benchmark, the dataset and evaluation harness are the primary contribution; without release, the community cannot reproduce Table 3 or Figures 3-10, cannot use the benchmark as a testbed, and cannot verify the construction pipeline. Please add a data and code availability statement, include the generation/evaluation code if possible, and address the licensing and redistribution terms for the GitHub-derived human code, specifying which parts can be shared and under what conditions.
  4. [Section 4.1 / Finding 8] Finding 8 claims that reasoning models exhibit a larger performance drop after paraphrasing, with the explanation that they "better interpret paraphrased inputs and adjust outputs to match human-style reasoning, making any deviations more evident after paraphrasing." This explanation is difficult to follow: if reasoning models adjust outputs toward human style, one would expect detection to become harder, not easier, and the phrase "making any deviations more evident" appears to refer to the paraphrased output, not to the detector's behavior. The finding needs a clearer mechanism, or the paper should present supporting evidence (e.g., per-model TPR@FPR before and after paraphrasing, or a qualitative analysis of paraphrased outputs) rather than a post-hoc explanation. As written, the causal story is not convincing.
minor comments (6)
  1. [Section 3.1 / AI Code Summarization] It is not stated which of the ten LLMs performs the summarization step; Appendix B shows only a GPT-o3-mini example. Since the summaries are the sole input to the generation step, the paper should specify whether summaries come from a single model, from each generator model, or from a different configuration, as this may affect stylistic properties of the generated code.
  2. [Section 3.2 / Table 3] The 700/300 train/test split is defined for perfect 1,000-sample shards, but Table 3 shows several shards with 998 or 999 samples (e.g., DeepSeek-R1 C# and Go, Gemini-2.0-Pro C# and PHP). Please state how the split is adjusted for incomplete shards.
  3. [Section 4.2 / Finding 6] "Biscope" should be "BiScope", and the sentence "resulting in stronger representations for the former" is slightly redundant; a short revision would improve readability.
  4. [Figure 5 / Section 4.3] The x-axis labels in Figure 5 are truncated (e.g., "Gemini 2.0 Flash Thinking" and "Qwen 2.5 Coder 32B"); consider rotating the labels, abbreviating model names in the main figure, or adding a legend with full names.
  5. [Appendix G / Figure 6] The claim that "TPR@FPR=1% is generally lower than 0.3" should be supported by a numerical table of TPR@FPR=10% and TPR@FPR=1% values per detector and configuration; the small multi-panel bar chart is hard to read and does not give exact values.
  6. [Abstract / Section 1] The abstract and introduction describe the ten models as "production-level LLMs", and the text later notes that four of them are open-source. The term "production-level" appears to mean API-available rather than proprietary; a one-sentence clarification would avoid confusion.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; CodeMirage's detector evaluations are external to construction. One minor self-citation (BiScope) is acknowledged but is not load-bearing.

full rationale

CodeMirage is a benchmark construction, not a formal derivation chain. Human samples come from the external CodeParrot Github-Code-Clean dataset; AI and paraphrased samples are generated by ten production LLMs via API; detectors are pre-existing external methods evaluated on held-out splits. No equation in the paper defines a predicted quantity in terms of a fitted parameter or of the benchmark's own construction constraints: findings 1-9 are empirical comparisons across configurations, languages, LLMs, and metrics, and none reduces by construction to the inputs. The Section 3.1 rule-based inspector enforces line/character length and BLEU<0.5, and Figure 2 later reports similar lengths and low n-gram overlap; this is a descriptive consequence of the filtering choice rather than a circular prediction, though it is a limitation for realism (no compile/run check). The only author-overlap issue is BiScope [27] (same authors) being included as one of ten detectors; it is described in related work and evaluated alongside all others, no finding depends solely on BiScope, and no uniqueness theorem or ansatz is imported from it. Thus there is no load-bearing self-citation or circular reduction; score 2 reflects the minor, non-load-bearing self-citation only.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The benchmark's validity rests on data provenance and quality-filter assumptions; no numbers are fitted to detector outputs, but the BLEU threshold and sampling temperatures are hand-chosen design parameters that affect dataset composition and difficulty.

free parameters (2)
  • BLEU recitation threshold = 0.5
    Hand-chosen acceptance threshold: generated samples with BLEU >= 0.5 against the human original are discarded to avoid recitation. This directly shapes dataset composition and difficulty.
  • Sampling temperature per generator = 0.6 (Llama-3.3-70B), 0.7 (Qwen-2.5-Coder-32B), 1.0 (all others)
    Appendix A sets non-zero temperatures to mimic realistic usage; temperature controls output diversity and therefore detectability of generated code.
assumptions (3)
  • domain assumption CodeParrot GitHub-Code-Clean samples are genuinely human-authored because the dataset was collected in May 2022, before widespread LLM code deployment.
    Section 3.1 grounds all human labels on this provenance claim; no per-file verification is reported.
  • domain assumption The rule-based inspector, line/character-length match and BLEU below 0.5, is sufficient to ensure generated and paraphrased code is realistic and non-recitative.
    Section 3.1; no compilation check, functional test, or human quality review is performed.
  • domain assumption API calls with default or recommended temperatures reflect real production usage of these models.
    Appendix A; the benchmark's production-level realism claim depends on this sampling choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CodeMirage: A Multi-Lingual Benchmark for Detecting AI-Generated and Paraphrased Source Code from Production-Level LLMs." pith.science (2026). https://pith.science/paper/MNHGAWTE

@misc{pith2026250611059,
  author       = {Pith},
  title        = {Pith review of: CodeMirage: A Multi-Lingual Benchmark for Detecting AI-Generated and Paraphrased Source Code from Production-Level LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MNHGAWTE}},
  note         = {Machine review of arXiv:2506.11059}
}
read the original abstract

Large language models (LLMs) have become integral to modern software development, producing vast amounts of AI-generated source code. While these models boost programming productivity, their misuse introduces critical risks, including code plagiarism, license violations, and the propagation of insecure programs. As a result, robust detection of AI-generated code is essential. To support the development of such detectors, a comprehensive benchmark that reflects real-world conditions is crucial. However, existing benchmarks fall short -- most cover only a limited set of programming languages and rely on less capable generative models. In this paper, we present CodeMirage, a comprehensive benchmark that addresses these limitations through three major advancements: (1) it spans ten widely used programming languages, (2) includes both original and paraphrased code samples, and (3) incorporates outputs from ten state-of-the-art production-level LLMs, including both reasoning and non-reasoning models from six major providers. Using CodeMirage, we evaluate ten representative detectors across four methodological paradigms under four realistic evaluation configurations, reporting results using three complementary metrics. Our analysis reveals nine key findings that uncover the strengths and weaknesses of current detectors, and identify critical challenges for future work. We believe CodeMirage offers a rigorous and practical testbed to advance the development of robust and generalizable AI-generated code detectors.

Figures

Figures reproduced from arXiv: 2506.11059 by the authors.

Figure 1
Figure 1. Overview of the CodeMirage framework. We collect and preprocess human-written code from GitHub, then leverage 10 state-of-the-art LLMs to summarize, generate, and paraphrase code with quality inspection. Finally, CodeMirage evaluates 10 baseline AI-generated code detectors across four categories under four configurations, covering a wide range of real-world scenarios. checks during code generation. LLMGCode [87] exp… view at source ↗
Figure 2
Figure 2. Benchmark statistics of CodeMirage. and function implementations—we prompt the LLM to extract and summarize key elements such as the purpose, functionality, logic overview, and key features, along with the names of relevant libraries, functions, classes, structures, and variables. Optional contextual notes are also included to account for uncommon assumptions or dependencies in the source code. This summary serves a… view at source ↗
Figure 3
Figure 3. Comparison Between Evaluation Configurations and Detectors. The bar chart presents the average F1 scores of baseline detectors across all the programming languages and LLMs. the detector on human test samples paired with AI-generated samples from all other LLMs. The detector’s scores on these unseen-model test sets are averaged to yield the overall cross-model result. Cross-Model Paraphrase Configuration. This scena… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Comparison Between Different Programming Languages. The bar chart presents the average F1 scores of baseline detectors on different programming languages across LLMs. Performance varies across fine-tuning methods. For example, CodeT5+ slightly outperforms GPT￾Sniffer, …
Figure 5
Figure 5. Figure 5: Comparison Between Different LLMs. The bar chart shows the average F1 scores of baseline detectors on different LLMs across programming languages. Among all models, GPT-4o mini achieves the highest F1 scores, particularly under the In-Distribution and Paraphrase settin…
Figure 6
Figure 6. Figure 6: Comparison Between Different Evaluation Metrics. The bar charts illustrate the average F1 scores of baseline detectors on different LLMs across programming languages. classifier for detection. In CodeMirage, we employ the latest CodeXEmbed-2B [46] model as the embeddin…
Figure 7
Figure 7. Figure 7: Complete F1 scores of all baseline detectors across various LLMs and programming [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Complete F1 scores of all baseline detectors across various LLMs and programming [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Complete F1 scores of all baseline detectors across various LLMs and programming [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Complete F1 scores of all baseline detectors across various LLMs and programming [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A fine-tuned encoder-only CodeT5 model attributes LLM-generated C code to its source model with up to 97.56% binary and 95.40% five-class accuracy on a new 32,000-program benchmark.

Reference graph

Works this paper leans on

111 extracted references · 55 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    An empirical study of ai generated text detection tools.arXiv preprint arXiv:2310.01423, 2023

    Arslan Akram. An empirical study of ai generated text detection tools.arXiv preprint arXiv:2310.01423, 2023

    arXiv 2023

  3. [3]

    Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku, 2024

    Anthropic. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku, 2024

    2024

  4. [4]

    Claude 3.7 Sonnet and Claude Code, 2025

    Anthropic. Claude 3.7 Sonnet and Claude Code, 2025

    2025

  5. [5]

    Is github’s copilot as bad as humans at introducing vulnerabilities in code?Empirical Software Engineering, 28(6):129, 2023

    Owura Asare, Meiyappan Nagappan, and Nirmal Asokan. Is github’s copilot as bad as humans at introducing vulnerabilities in code?Empirical Software Engineering, 28(6):129, 2023

    2023

  6. [6]

    Random forests.Machine learning, 45:5–32, 2001

    Leo Breiman. Random forests.Machine learning, 45:5–32, 2001

    2001

  7. [7]

    Membership inference attacks from first principles

    Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In2022 IEEE symposium on security and privacy (SP), pages 1897–1914. IEEE, 2022

    1914

  8. [8]

    R package version 3.0.1.1

    Tianqi Chen, Tong He, Michael Benesty, Vadim Khotilovich, Yuan Tang, Hyunsu Cho, Kailong Chen, Rory Mitchell, Ignacio Cano, Tianyi Zhou, Mu Li, Junyuan Xie, Min Lin, Yifeng Geng, Yutian Li, Jiaming Yuan, and David Cortes.xgboost: Extreme Gradient Boosting, 2025. R package version 3.0.1.1

    2025

Show all 111 references
  1. [9]

    Github code clean dataset, 2022

    CodeParrot. Github code clean dataset, 2022

    2022

  2. [10]

    Github code dataset, 2022

    CodeParrot. Github code dataset, 2022

    2022

  3. [11]

    Vulnerabilities in ai code generators: Exploring targeted data poisoning attacks

    Domenico Cotroneo, Cristina Improta, Pietro Liguori, and Roberto Natella. Vulnerabilities in ai code generators: Exploring targeted data poisoning attacks. InIEEE/ACM International Conference on Program Comprehension (ICPC), pages 280–292, 2024

    2024

  4. [12]

    Cursor: The AI Code Editor, 2023

    Cursor. Cursor: The AI Code Editor, 2023

    2023

  5. [13]

    Plagiarism in the age of massive generative pre-trained transformers (gpt-3).Ethics in Science and Environmental Politics, 21:17–23, 2021

    Nassim Dehouche. Plagiarism in the age of massive generative pre-trained transformers (gpt-3).Ethics in Science and Environmental Politics, 21:17–23, 2021

    2021

  6. [14]

    Aigcodeset: A new annotated dataset for ai generated code detection

    Basak Demirok and Mucahid Kutlu. Aigcodeset: A new annotated dataset for ai generated code detection. arXiv preprint arXiv:2412.16594, 2024

    2024 arXiv

  7. [15]

    The deepfake detection challenge (dfdc) dataset.arXiv preprint arXiv:2006.07397, 2020

    Brian Dolhansky, Joanna Bitton, Ben Pflaum, Jikuo Lu, Russ Howes, Menglin Wang, and Cristian Canton Ferrer. The deepfake detection challenge (dfdc) dataset.arXiv preprint arXiv:2006.07397, 2020

    2006 arXiv

  8. [16]

    Codep: grammatical seq2seq model for general-purpose code generation

    Yihong Dong, Ge Li, and Zhi Jin. Codep: grammatical seq2seq model for general-purpose code generation. InACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), pages 188–198, 2023

    2023

  9. [17]

    Jacotext: a pretrained model for java code-text generation.arXiv preprint arXiv:2303.12869, 2023

    Jessica López Espejel, Mahaman Sanoussi Yahaya Alassan, Walid Dahhane, and El Hassane Ettifouri. Jacotext: a pretrained model for java code-text generation.arXiv preprint arXiv:2303.12869, 2023

    2023 arXiv

  10. [18]

    Out of the bleu: how should we assess quality of the code generation models?Journal of Systems and Software, 203:111741, 2023

    Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. Out of the bleu: how should we assess quality of the code generation models?Journal of Systems and Software, 203:111741, 2023

    2023

  11. [19]

    Codebert: A pre-trained model for programming and natural languages.https://arxiv.org/abs/2002.08155, 2020

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. Codebert: A pre-trained model for programming and natural languages.https://arxiv.org/abs/2002.08155, 2020

    2002 arXiv

  12. [20]

    Introducing GitHub Copilot: your AI pair programmer, 2022

    Friedman, Nat. Introducing GitHub Copilot: your AI pair programmer, 2022

    2022

  13. [21]

    What makes good in-context demonstrations for code intelligence tasks with llms? InIEEE/ACM International Conference on Automated Software Engineering (ASE), pages 761–773, 2023

    Shuzheng Gao, Xin-Cheng Wen, Cuiyun Gao, Wenxuan Wang, Hongyu Zhang, and Michael R Lyu. What makes good in-context demonstrations for code intelligence tasks with llms? InIEEE/ACM International Conference on Automated Software Engineering (ASE), pages 761–773, 2023

    2023

  14. [22]

    Gltr: Statistical detection and visualization of generated text

    Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. Gltr: Statistical detection and visualization of generated text. InAnnual Meeting of the Association for Computational Linguistics (ACL), 2019

    2019

  15. [23]

    A survey on the possibilities & impossibilities of ai-generated text detection.Transactions on Machine Learning Research (TMLR), 2023

    Soumya Suvra Ghosal, Souradip Chakraborty, Jonas Geiping, Furong Huang, Dinesh Manocha, and Amrit Bedi. A survey on the possibilities & impossibilities of ai-generated text detection.Transactions on Machine Learning Research (TMLR), 2023. 10

    2023

  16. [24]

    Deepfake video detection using recurrent neural networks

    David Güera and Edward J Delp. Deepfake video detection using recurrent neural networks. InIEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pages 1–6, 2018

    2018

  17. [25]

    Graphcodebert: Pre-training code representations with data flow

    Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, LIU Shujie, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. Graphcodebert: Pre-training code representations with data flow. In International Conference on Learning Representations (ICLR), 2021

    2021

  18. [26]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  19. [27]

    Biscope: Ai-generated text detection by checking memorization of preceding tokens

    Hanxi Guo, Siyuan Cheng, Xiaolong Jin, Zhuo Zhang, Kaiyuan Zhang, Guanhong Tao, Guangyu Shen, and Xiangyu Zhang. Biscope: Ai-generated text detection by checking memorization of preceding tokens. Advances in Neural Information Processing Systems (NeurIPS), 37:104065–104090, 2024

    2024

  20. [28]

    Spotting llms with binoculars: Zero-shot detection of machine-generated text

    Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Spotting llms with binoculars: Zero-shot detection of machine-generated text. InInternational Conference on Machine Learning (ICML), 2024

    2024

  21. [29]

    Denoising diffusion probabilistic models.Advances in Neural Information Processing Systems (NeurIPS), 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in Neural Information Processing Systems (NeurIPS), 33:6840–6851, 2020

    2020

  22. [30]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  23. [31]

    Rethinking plagiarism in the era of generative ai.Journal of Intelligent Communication, 3(2):20–31, 2024

    James Hutson. Rethinking plagiarism in the era of generative ai.Journal of Intelligent Communication, 3(2):20–31, 2024

    2024

  24. [32]

    Whodunit: Classifying code as human authored or gpt-4 generated-a case study on codechef problems

    Oseremen Joy Idialu, Noble Saji Mathews, Rungroj Maipradit, Joanne M Atlee, and Mei Nagappan. Whodunit: Classifying code as human authored or gpt-4 generated-a case study on codechef problems. In International Conference on Mining Software Repositories (MSR), pages 394–406, 2024

    2024

  25. [33]

    Automatic detection of generated text is easiest when humans are fooled

    Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of generated text is easiest when humans are fooled. InAnnual Meeting of the Association for Computational Linguistics (ACL), pages 1808–1822, 2020

    2020

  26. [34]

    Swe-bench: Can language models resolve real-world github issues? InInternational Conference on Learning Representations (ICLR), 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? InInternational Conference on Learning Representations (ICLR), 2024

    2024

  27. [35]

    Access the latest 2.0 experimental models in the gemini app., 2025

    Patrick Kane. Access the latest 2.0 experimental models in the gemini app., 2025

    2025

  28. [36]

    Vulnerability handling of ai- generated code-existing solutions and open challenges

    Sabrina Kaniewski, Dieter Holstein, Fabian Schmidt, and Tobias Heer. Vulnerability handling of ai- generated code-existing solutions and open challenges. InConference on AI, Science, Engineering, and Technology (AIxSET), pages 145–148, 2024

    2024

  29. [37]

    Gemini 2.0 is now available to everyone, 2025

    Koray Kavukcuoglu. Gemini 2.0 is now available to everyone, 2025

    2025

  30. [38]

    Does attitude towards plagiarism predict aigiarism using chatgpt?AI and Ethics, 5(1):677–688, 2025

    Mustafa Ali Khalaf. Does attitude towards plagiarism predict aigiarism using chatgpt?AI and Ethics, 5(1):677–688, 2025

    2025

  31. [39]

    Will chatgpt g et you caught? rethinking of plagiarism detection

    Mohammad Khalil and Erkan Er. Will chatgpt g et you caught? rethinking of plagiarism detection. In International Conference on Human-Computer Interaction, pages 475–487, 2023

    2023

  32. [40]

    How secure is code generated by chatgpt? InIEEE international conference on systems, man, and cybernetics (SMC), pages 2445–2451

    Raphaël Khoury, Anderson R Avila, Jacob Brunelle, and Baba Mamadou Camara. How secure is code generated by chatgpt? InIEEE international conference on systems, man, and cybernetics (SMC), pages 2445–2451. IEEE, 2023

    2023

  33. [41]

    Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense.Advances in Neural Information Processing Systems (NeurIPS), 2023

    Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense.Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  34. [42]

    Detecting fake content with relative entropy scoring

    Thomas Lavergne, Tanguy Urvoy, and François Yvon. Detecting fake content with relative entropy scoring. InProceedings of the International Conference on Uncovering Plagiarism, Authorship and Social Software Misuse (PAN), volume 377, pages 27–31, 2008

    2008

  35. [43]

    Protecting intellectual property of large language model-based code generation apis via watermarks

    Zongjie Li, Chaozheng Wang, Shuai Wang, and Cuiyun Gao. Protecting intellectual property of large language model-based code generation apis via watermarks. InACM SIGSAC Conference on Computer and Communications Security (CCS), pages 2336–2350, 2023. 11

    2023

  36. [44]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  37. [45]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in Neural Information Processing Systems (NeurIPS), 36:21558–21572, 2023

    2023

  38. [46]

    Codex- embed: A generalist embedding model family for multiligual and multi-task code retrieval.arXiv preprint arXiv:2411.12644, 2024

    Ye Liu, Rui Meng, Shafiq Joty, Silvio Savarese, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Codex- embed: A generalist embedding model family for multiligual and multi-task code retrieval.arXiv preprint arXiv:2411.12644, 2024

    2024 arXiv

  39. [47]

    Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692, 2019

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  40. [48]

    Raidar: generative ai detection via rewriting

    Chengzhi Mao, Carl V ondrick, Hao Wang, and Junfeng Yang. Raidar: generative ai detection via rewriting. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  41. [49]

    On the robustness of code generation techniques: An empirical study on github copilot

    Antonio Mastropaolo, Luca Pascarella, Emanuela Guglielmi, Matteo Ciniselli, Simone Scalabrino, Rocco Oliveto, and Gabriele Bavota. On the robustness of code generation techniques: An empirical study on github copilot. InInternational Conference on Software Engineering (ICSE), ...

    2023

  42. [50]

    Llama 3.3: Model cards & prompt formats, 2024

    Meta. Llama 3.3: Model cards & prompt formats, 2024

    2024

  43. [51]

    Detectgpt: Zero-shot machine-generated text detection using probability curvature

    Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curvature. InInternational Conference on Machine Learning (ICML), pages 24950–24962. PMLR, 2023

    2023

  44. [52]

    Is this snippet written by chatgpt? an empirical study with a codebert-based classifier.arXiv preprint arXiv:2307.09381, 2023

    Phuong T Nguyen, Juri Di Rocco, Claudio Di Sipio, Riccardo Rubei, Davide Di Ruscio, and Massimiliano Di Penta. Is this snippet written by chatgpt? an empirical study with a codebert-based classifier.arXiv preprint arXiv:2307.09381, 2023

    2023 arXiv

  45. [53]

    Gptsniffer: A codebert-based classifier to detect source code written by chatgpt.Journal of Systems and Software, 214:112059, 2024

    Phuong T Nguyen, Juri Di Rocco, Claudio Di Sipio, Riccardo Rubei, Davide Di Ruscio, and Massimiliano Di Penta. Gptsniffer: A codebert-based classifier to detect source code written by chatgpt.Journal of Systems and Software, 214:112059, 2024

    2024

  46. [54]

    Poisoned chatgpt finds work for idle hands: Exploring developers’ coding practices with insecure suggestions from poisoned ai models

    Sanghak Oh, Kiho Lee, Seonhye Park, Doowon Kim, and Hyoungshick Kim. Poisoned chatgpt finds work for idle hands: Exploring developers’ coding practices with insecure suggestions from poisoned ai models. InIEEE Symposium on Security and Privacy (S&P), pages 1141–1159, 2024

    2024

  47. [55]

    Introducing ChatGPT, 2022

    OpenAI. Introducing ChatGPT, 2022

    2022

  48. [56]

    Gpt-4o mini: advancing cost-efficient intelligence, 2024

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence, 2024

    2024

  49. [57]

    Openai o3-mini: Pushing the frontier of cost-effective reasoning, 2025

    OpenAI. Openai o3-mini: Pushing the frontier of cost-effective reasoning, 2025

    2025

  50. [58]

    Codet-m4: Detecting machine-generated code in multi-lingual, multi-generator and multi-domain settings.arXiv preprint arXiv:2503.13733, 2025

    Daniil Orel, Dilshod Azizov, and Preslav Nakov. Codet-m4: Detecting machine-generated code in multi-lingual, multi-generator and multi-domain settings.arXiv preprint arXiv:2503.13733, 2025

    2025 arXiv

  51. [59]

    Assessing ai detectors in identifying ai-generated code: Implications for education

    Wei Hung Pan, Ming Jie Chok, Jonathan Leong Shan Wong, Yung Xin Shin, Yeong Shian Poon, Zhou Yang, Chun Yong Chong, David Lo, and Mei Kuan Lim. Assessing ai detectors in identifying ai-generated code: Implications for education. InInternational Conference on Software Engineeri...

    2024

  52. [60]

    Bleu: A method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. InAnnual Meeting of the Association for Computational Linguistics (ACL), pages 311–318, 2002

    2002

  53. [61]

    Asleep at the keyboard? assessing the security of github copilot’s code contributions

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. Asleep at the keyboard? assessing the security of github copilot’s code contributions. InIEEE Symposium on Security and Privacy (S&P), pages 754–768, 2022

    2022

  54. [62]

    Magecode: Machine- generated code detection method using large language models.IEEE Access, 2024

    Hung Pham, Huyen Ha, Van Tong, Dung Hoang, Duc Tran, and Tuyen Ngoc Le. Magecode: Machine- generated code detection method using large language models.IEEE Access, 2024

    2024

  55. [63]

    Introducing gemini 2.0: our new ai model for the agentic era, 2024

    Sundar Pichai, Demis Hassabis, and Koray Kavukcuoglu. Introducing gemini 2.0: our new ai model for the agentic era, 2024. 12

    2024

  56. [64]

    Using tf-idf to determine word relevance in document queries

    Juan Ramos et al. Using tf-idf to determine word relevance in document queries. InProceedings of the first instructional conference on machine learning, volume 242, pages 29–48. Citeseer, 2003

    2003

  57. [65]

    Codebleu: a method for automatic evaluation of code synthesis.arXiv preprint arXiv:2009.10297, 2020

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis.arXiv preprint arXiv:2009.10297, 2020

    2009 arXiv

  58. [66]

    The perceptron: a probabilistic model for information storage and organization in the brain.Psychological review, 65(6):386, 1958

    Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain.Psychological review, 65(6):386, 1958

    1958

  59. [67]

    Faceforensics: A large-scale video dataset for forgery detection in human faces.arXiv preprint arXiv:1803.09179, 2018

    Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics: A large-scale video dataset for forgery detection in human faces.arXiv preprint arXiv:1803.09179, 2018

    2018 arXiv

  60. [68]

    Can ai-generated text be reliably detected?arXiv preprint arXiv:2303.11156, 2023

    Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected?arXiv preprint arXiv:2303.11156, 2023

    2023 arXiv

  61. [69]

    Automated detection of ai-obfuscated plagiarism in modeling assignments

    Timur Sa˘glam, Sebastian Hahner, Larissa Schmid, and Erik Burger. Automated detection of ai-obfuscated plagiarism in modeling assignments. InInternational Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET), pages 297–308, 2024

    2024

  62. [70]

    Between lines of code: Unraveling the distinct patterns of machine and human programmers

    Yuling Shi, Hongyu Zhang, Chengcheng Wan, and Xiaodong Gu. Between lines of code: Unraveling the distinct patterns of machine and human programmers. InInternational Conference on Software Engineering (ICSE), pages 51–62, 2025

    2025

  63. [71]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InInternational Conference on Machine Learning (ICML), pages 2256–2265. pmlr, 2015

    2015

  64. [72]

    2024 Stack Overflow Developer Survey, 2024

    Stack Overflow. 2024 Stack Overflow Developer Survey, 2024

    2024

  65. [73]

    Developer perspectives on licensing and copyright issues arising from generative ai for coding.arXiv preprint arXiv:2411.10877, 2024

    Trevor Stalnaker, Nathan Wintersgill, Oscar Chaparro, Laura A Heymann, Massimiliano Di Penta, Daniel M German, and Denys Poshyvanyk. Developer perspectives on licensing and copyright issues arising from generative ai for coding.arXiv preprint arXiv:2411.10877, 2024

    2024 arXiv

  66. [74]

    Plagiarism in ai empowered world

    Aiste Steponenaite and Basel Barakat. Plagiarism in ai empowered world. InInternational Conference on Human-Computer Interaction, pages 434–442, 2023

    2023

  67. [75]

    An empirical study on automatically detecting ai-generated source code: How far are we? InInternational Conference on Software Engineering (ICSE), 2025

    Hyunjae Suh, Mahan Tafreshipour, Jiawei Li, Adithya Bhattiprolu, and Iftekhar Ahmed. An empirical study on automatically detecting ai-generated source code: How far are we? InInternational Conference on Software Engineering (ICSE), 2025

    2025

  68. [76]

    Bugs in large language models generated code: An empirical study.Empirical Software Engineering, 30(3):1–48, 2025

    Florian Tambon, Arghavan Moradi-Dakhel, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Giuliano Antoniol. Bugs in large language models generated code: An empirical study.Empirical Software Engineering, 30(3):1–48, 2025

    2025

  69. [77]

    How secure is ai-generated code: a large-scale comparison of large language models.Empirical Software Engineering, 30(2):1–42, 2025

    Norbert Tihanyi, Tamas Bisztray, Mohamed Amine Ferrag, Ridhi Jain, and Lucas C Cordeiro. How secure is ai-generated code: a large-scale comparison of large language models.Empirical Software Engineering, 30(2):1–42, 2025

    2025

  70. [78]

    Llms in web development: Evaluating llm-generated php code unveiling vulnerabilities and limitations

    Rebeka Tóth, Tamas Bisztray, and László Erd˝odi. Llms in web development: Evaluating llm-generated php code unveiling vulnerabilities and limitations. InInternational Conference on Computer Safety, Reliability, and Security, pages 425–437, 2024

    2024

  71. [79]

    Turingbench: A benchmark environ- ment for turing test in the age of neural text generation

    Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang, and Dongwon Lee. Turingbench: A benchmark environ- ment for turing test in the age of neural text generation. InFindings of the Association for Computational Linguistics: EMNLP 2021, pages 2001–2016, 2021

    2021

  72. [80]

    A critical look at ai-generate software: Coding with the new ai tools is both irresistible and dangerous.IEEE Spectrum, 60(7):34–39, 2023

    Jaideep Vaidya and Hafiz Asif. A critical look at ai-generate software: Coding with the new ai tools is both irresistible and dangerous.IEEE Spectrum, 60(7):34–39, 2023

    2023

  73. [81]

    Attention is all you need.Advances in Neural Information Processing Systems (NeurIPS), 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in Neural Information Processing Systems (NeurIPS), 30, 2017

    2017

  74. [82]

    Is your ai-generated code really safe? evaluating large language models on secure code generation with codeseceval.arXiv preprint arXiv:2407.02395, 2024

    Jiexin Wang, Xitong Luo, Liuwen Cao, Hongkui He, Hailin Huang, Jiayuan Xie, Adam Jatowt, and Yi Cai. Is your ai-generated code really safe? evaluating large language models on secure code generation with codeseceval.arXiv preprint arXiv:2407.02395, 2024. 13

    2024 arXiv

  75. [83]

    Openhands: An open platform for ai software developers as generalist agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  76. [84]

    Codet5+: Open code large language models for code understanding and generation

    Yue Wang, Hung Le, Akhilesh Gotmare, Nghi Bui, Junnan Li, and Steven Hoi. Codet5+: Open code large language models for code understanding and generation. InConference on Empirical Methods in Natural Language Processing (EMNLP), pages 1069–1088, 2023

    2023

  77. [85]

    A new era of plagiarism the danger of cheating using ai

    Yunkai Xiao, Soumyadeep Chatterjee, and Edward Gehringer. A new era of plagiarism the danger of cheating using ai. InInternational Conference on Information Technology Based Higher Education and Training (ITHET), pages 1–6, 2022

    2022

  78. [86]

    Do llms know to respect copyright notice? InConference on Empirical Methods in Natural Language Processing (EMNLP), pages 20604–20619, 2024

    Jialiang Xu, Shenglan Li, Zhaozhuo Xu, and Denghui Zhang. Do llms know to respect copyright notice? InConference on Empirical Methods in Natural Language Processing (EMNLP), pages 20604–20619, 2024

    2024

  79. [87]

    Investigating efficacy of perplexity in detecting llm-generated code.arXiv preprint arXiv:2412.16525, 2024

    Jinwei Xu, He Zhang, Yanjin Yang, Zeru Cheng, Jun Lyu, Bohan Liu, Xin Zhou, Lanxin Yang, Alberto Bacchelli, Yin Kia Chiam, et al. Investigating efficacy of perplexity in detecting llm-generated code.arXiv preprint arXiv:2412.16525, 2024

    2024 arXiv

  80. [88]

    Licoeval: Evaluating llms on license compliance in code generation.arXiv preprint arXiv:2408.02487, 2024

    Weiwei Xu, Kai Gao, Hao He, and Minghui Zhou. Licoeval: Evaluating llms on license compliance in code generation.arXiv preprint arXiv:2408.02487, 2024

    2024 arXiv

  81. [89]

    Distin- guishing llm-generated from human-written code by contrastive learning.ACM Transactions on Software Engineering and Methodology, 34(4):1–31, 2025

    Xiaodan Xu, Chao Ni, Xinrong Guo, Shaoxuan Liu, Xiaoya Wang, Kui Liu, and Xiaohu Yang. Distin- guishing llm-generated from human-written code by contrastive learning.ACM Transactions on Software Engineering and Methodology, 34(4):1–31, 2025

    2025

  82. [90]

    Detecting ai-generated code assignments using perplexity of large language models

    Zhenyu Xu and Victor S Sheng. Detecting ai-generated code assignments using perplexity of large language models. InAAAI Conference on Artificial Intelligence (AAAI), volume 38, pages 23155–23162, 2024

    2024

  83. [91]

    An {LLM-Assisted}{Easy-to-Trigger} backdoor attack on code completion models: Injecting disguised vulnerabilities against strong detection

    Shenao Yan, Shen Wang, Yue Duan, Hanbin Hong, Kiho Lee, Doowon Kim, and Yuan Hong. An {LLM-Assisted}{Easy-to-Trigger} backdoor attack on code completion models: Injecting disguised vulnerabilities against strong detection. InUSENIX Security Symposium (USENIX Security), pages 1...

    2024

  84. [92]

    Zero-shot detection of machine-generated codes.arXiv preprint arXiv:2310.05103, 2023

    Xianjun Yang, Kexun Zhang, Haifeng Chen, Linda Petzold, William Yang Wang, and Wei Cheng. Zero-shot detection of machine-generated codes.arXiv preprint arXiv:2310.05103, 2023

    2023 arXiv

  85. [93]

    Uncovering llm-generated code: A zero-shot synthetic code detector via code rewriting

    Tong Ye, Yangkai Du, Tengfei Ma, Lingfei Wu, Xuhong Zhang, Shouling Ji, and Wenhai Wang. Uncovering llm-generated code: A zero-shot synthetic code detector via code rewriting. InAAAI Conference on Artificial Intelligence (AAAI), volume 39, pages 968–976, 2025

    2025

  86. [94]

    Codeipprompt: intellectual property infringement assessment of code language models

    Zhiyuan Yu, Yuhao Wu, Ning Zhang, Chenguang Wang, Yevgeniy V orobeychik, and Chaowei Xiao. Codeipprompt: intellectual property infringement assessment of code language models. InInternational Conference on Machine Learning (ICML), pages 40373–40389, 2023

    2023

  87. [95]

    Inducing vulnerable code generation in llm coding assistants.arXiv preprint arXiv:2504.15867, 2025

    Binqi Zeng, Quan Zhang, Chijin Zhou, Gwihwan Go, Yu Jiang, and Heyuan Shi. Inducing vulnerable code generation in llm coding assistants.arXiv preprint arXiv:2504.15867, 2025

    2025 arXiv

  88. [96]

    How well does llm generate security tests?arXiv preprint arXiv:2310.00710, 2023

    Ying Zhang, Wenjia Song, Zhengjie Ji, Na Meng, et al. How well does llm generate security tests?arXiv preprint arXiv:2310.00710, 2023

    2023

  89. [97]

    Genimage: A million-scale benchmark for detecting ai-generated image.Advances in Neural Information Processing Systems (NeurIPS), 36:77771–77782, 2023

    Mingjian Zhu, Hanting Chen, Qiangyu Yan, Xudong Huang, Guanyu Lin, Wei Li, Zhijun Tu, Hailin Hu, Jie Hu, and Yunhe Wang. Genimage: A million-scale benchmark for detecting ai-generated image.Advances in Neural Information Processing Systems (NeurIPS), 36:77771–77782, 2023

    2023

  90. [98]

    Wilddeepfake: A challenging real-world dataset for deepfake detection

    Bojia Zi, Minghao Chang, Jingjing Chen, Xingjun Ma, and Yu-Gang Jiang. Wilddeepfake: A challenging real-world dataset for deepfake detection. InProceedings of the 28th ACM international conference on multimedia, pages 2382–2390, 2020. 14 To further support and validate ourCode...

    2020

  91. [99]

    Functionality:Describe the main tasks performed by the code, including inputs, outputs, and their roles, without referencing exact code or variable names

  92. [100]

    Logic Overview:Explain the key logic, algorithms, or patterns conceptually, avoiding specific code structures or syntax

  93. [101]

    5.Key Libraries:Include the key libraries used in the code snippet

    Key Features:Highlight unique approaches or techniques without mentioning explicit implementa- tion details. 5.Key Libraries:Include the key libraries used in the code snippet. 6.Key Functions:Include the key function names used in the code snippet

  94. [102]

    Key Class, Structure, Variable:Include the key class, structure, and variable names used in the code snippet

  95. [103]

    Contextual Notes (optional):Any assumptions, dependencies, or inferred background information. Below, we present a concrete Python code example with the summary generated by GPT-o3-mini [57], illustrating the consistency and descriptive quality of the summarization process inC...

  96. [104]

    e n o c e a n _ b i n a r y _ s e n s o r

    Purpose:The code integrates binary sensors that communicate using an EnOcean protocol into a larger home automation framework. It enables sensor events to be received, processed, and dispatched within the automation system. 2.Functionality:The implementation defines a platform...

  97. [105]

    Rename variables, functions, and classes with meaningful, human-like names

  98. [106]

    Adjust formatting (indentation, spacing, line breaks) and reorganize imports logically

  99. [107]

    Rewrite logic with equivalent constructs (e.g., refactor loops, conditionals)

  100. [108]

    Replace common standard library functions with alternatives or custom helpers

  101. [109]

    Vary expressions (e.g., ‘a + b’→‘sum([a, b])’) and simplify inline logic into reusable functions

  102. [110]

    double quotes, ‘10’ → ‘0xA’) and reorder independent code blocks

    Transform literals (e.g., single vs. double quotes, ‘10’ → ‘0xA’) and reorder independent code blocks

  103. [111]

    Keep the line of code and character length similar to the original code and only output the transformed code

    Add subtle redundancies (e.g., intermediate variables) to resemble human-written code. Keep the line of code and character length similar to the original code and only output the transformed code. No any other additional text and information. We then apply the paraphrasing pro...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.