Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Detecting LLM-generated Code with Subtle Modification by Adversarial Training

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that small human edits such as renaming a variable or converting a loop can hide LLM-generated code from detectors, and that adversarial fine-tuning with matched examples restores detection without hurting accuracy on…

desk verdict A solid engineering paper with a circularity problem: the headline robustness gain is measured on test samples generated by the same MIST module used for training, so the real-world claim is not yet supported. read the letter →

arxiv 2507.13123 v1 pith:TRILML67 submitted 2025-07-17 cs.SE

Xin Yin , Xinrui Li , Chao Ni , Xiaodan Xu , Xiaohu Yang This is my paper
classification cs.SE
keywords LLM-generatedcodedetectionadversarialtrainingmodificationrobustnessidentifierrenamingstructuretransformationmulti-objectiveoptimizationGPTSensor
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to show that the brittleness of LLM-generated-code detectors under small human edits is fixable. It claims that a detector fine-tuned on adversarial samples crafted to resemble such edits, renamed identifiers and loop or branch conversions, keeps its accuracy on unmodified code while nearly recovering its accuracy on edited code. The authors report that accuracy on the modified Python test set rises from 0.260 to 0.969 and on Java from 0.325 to 0.944, with original-set accuracy unchanged at 0.992 and 0.970. If the claim holds, a detector like CodeGPTSensor+ could be deployed where edited code is the norm, such as student submissions or refactored production code.

What carries the argument

The load-bearing object is MIST, a genetic-algorithm-style module that generates the adversarial training samples. Mutation either renames an identifier, selected by importance score and replaced by a CodeT5 context-aware prediction, or applies an equivalent code-structure transformation such as converting a for loop to a while loop, chosen probabilistically to match the coding style of human-written reference samples. Crossover combines identifier-renaming maps of two parents, and NSGA-II non-dominated sorting keeps the N samples that best balance three objectives: adversarial loss (confidence in the true class), semantic distance (cosine similarity of identifiers), and edit distance (Levenshtein distance). This module is what lets the model learn the edit patterns instead of memorizing raw samples.

What would settle it

Collect a separate test set of genuinely human-modified LLM-generated functions, for example student solutions or developer commits that edited GPT-written code, draw no samples from MIST, and measure CodeGPTSensor+ accuracy there; a return to near-chance accuracy would show the robustness is specific to MIST-style edits rather than to human-style modification generically.

Watch

Extended reading notes

Core claim

CodeGPTSensor+ is CodeGPTSensor adversarially fine-tuned on a 70/30 mix of original training samples and MIST-generated adversarial samples, with 10% of the training set as the source of attacks. The central discovery is that a detection model trained this way resists minor modifications, including variable renaming, loop-to-while conversions, increment and decrement rewrites, and compound assignment expansion, while leaving its behavior on unmodified code essentially intact. The paper also claims that MIST's samples are better attack material than prior methods, achieving higher attack success with fewer identifier changes and fewer model queries, and that fine-tuning on them defends against samples produced by other attack methods as well. The authors present this as a practical route to robustness against the human rewrites that currently defeat state-of-the-art detectors.

Load-bearing premise

The evaluation assumes that the HMCorp-adv test edits, produced by the same MIST module used to create training samples, represent the small changes humans actually make to LLM-generated code; if real edits come from a different distribution, such as function splitting or third-party libraries, the reported gains may not transfer.

Editorial extensions

If this is right

  • Detection accuracy on subtly modified LLM-generated code jumps from 0.260 to 0.969 on Python and from 0.325 to 0.944 on Java, which the paper measures as relative improvements of 272.8% and 190.8%.
  • Accuracy on unmodified code stays essentially unchanged at 0.992 on Python and 0.970 on Java, so the robustness gain does not come at the cost of original detection performance.
  • Models fine-tuned on MIST-generated samples also defend against adversarial samples produced by ALERT, CODA, and MOAA, indicating that the training samples transfer across attack methods.
  • MIST itself achieves higher attack success rates and lower identifier change rates and query counts than the baselines in most settings, which the paper links to its joint use of structure transformation and multi-objective optimization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An independent test set of real human-modified LLM code, collected outside the MIST edit family, would be the decisive check on whether the reported gains reflect human-style modification generally or only MIST-style edits.
  • The same adversarial training recipe could plausibly harden other code classifiers, such as authorship attribution or vulnerability detectors, against the rewriting tricks that currently evade them.
  • The style-aware probabilistic transformation mechanism suggests a broader design principle: adversarially trained detectors should be tuned against the edit distribution of the population they will actually face, not just against maximal perturbations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes CodeGPTSensor+, an adversarially fine-tuned version of the existing CodeGPTSensor detector for LLM-generated code. The enhancement is driven by MIST, a black-box adversarial sample generation module that combines identifier replacement with syntax-preserving structural transformations and uses a multi-objective optimization framework to trade off attack success, semantic distance, and edit distance. The authors use MIST to generate adversarial examples from 10% of the HMCorp training set, mix them with original samples, fine-tune CodeGPTSensor for one epoch, and evaluate on both the original HMCorp test set and a MIST-generated adversarial test set (HMCorp-adv). They report that CodeGPTSensor+ preserves near-original accuracy on unmodified code while raising adversarial-set accuracy from 0.260 to 0.969 on Python and from 0.325 to 0.944 on Java. They also compare MIST against ALERT, CODA, and MOAA on attack metrics, and evaluate cross-attack robustness in RQ3 by adversarial fine-tuning with one attack family and testing on another.

Significance. If the claimed robustness transfers to real human modifications, the paper makes a useful contribution: it shows that a simple adversarial fine-tuning recipe over a multi-objective mixture of identifier and structural perturbations can substantially harden a code-origin classifier, and it ships an open-source implementation and dataset. RQ3 is a genuine strength because it tests MIST-trained models on samples generated by other attack algorithms, which partially breaks the train/test circularity of RQ1. However, the central real-world claim is only as strong as the representativeness of MIST as a model of minor human edits, and the evaluation would need to be supplemented with human-modified code or held-out perturbation families before the robustness claim can be accepted at face value.

major comments (4)
  1. [4.1, 3.2, Table 2] The headline robustness result is measured on HMCorp-adv, which is produced by the same MIST module used to create the adversarial training data in Section 3.2. The model therefore trains and tests on the same perturbation family, so the large gains in Table 2 may largely reflect learning MIST-specific artifacts (CodeT5 top-k identifier substitutions and the five structural rules in Table 1) rather than robustness to the broader class of real minor edits. Section 6 itself concedes that real modifications such as function splitting or third-party library introduction are outside MIST's perturbation strategies. RQ3 provides useful cross-attack evidence, but ALERT, CODA, and MOAA also operate in the same discrete code-perturbation space and share context-aware identifier substitution, so they do not establish transfer to human edits. The paper should either evaluate on human-modified code (e.g., a curated set of renamed/refactored functions) or on held-out perturbation types, or explicitly restrict the conclusion to the tested perturbation space.
  2. [5.1, 5.3] All experiments report point estimates with no error bars, no multiple seeds, and no significance tests. Several components are stochastic (MIST population initialization, random 10% sampling of the training set, random mixing of adversarial and original samples), so the reported differences, especially the smaller ones in RQ3 (e.g., 0.968 vs 0.970) and the TOPSIS composite scores, need variance estimates. Reporting means and standard deviations over at least five independent fine-tuning runs, together with a paired test where appropriate, would materially strengthen the comparison.
  3. [5.2, Table 3] The claim of MIST's overall superiority is only partially supported by the per-metric results. MIST's ASR on HMCorp-Java against GPTSniffer (48.41%) is lower than MOAA's (59.94%), and on HMCorp-Python MIST's AMQ (61.95) is substantially higher than MOAA's (14.53). The conclusion therefore rests heavily on the TOPSIS aggregation with equal weights; under the ASR-prioritized weighting (ASR weight 0.6) Figure 3 should be inspected for whether MIST remains dominant in all four settings. The paper should present the per-metric tradeoffs and justify the chosen weight configurations rather than stating unqualified superiority.
  4. [4.4, 3.2] Key hyperparameters that likely control the balance between robustness and original accuracy are fixed without any sensitivity analysis: identifier replacement probability r=0.5, candidate count k=40, 10% training-set sampling, 70/30 mix ratio, and one fine-tuning epoch. Since a central claim is that CodeGPTSensor+ 'maintains high accuracy on the original test set,' an ablation varying the mix ratio and r would show whether the reported retention is robust or a lucky point in parameter space.
minor comments (4)
  1. [4.4] The model name is inconsistent: Section 4.4 refers to 'GPTCodeSensor' and 'GPTCodeSensor+' while the rest of the paper uses 'CodeGPTSensor' and 'CodeGPTSensor+'.
  2. [References] Reference [6] contains a typo: 'Mulit-Objedtive' should be 'Multi-Objective'.
  3. [Figure 3] The TOPSIS composite-score figure is difficult to read in the provided version; larger labels and explicit score values would help the reader verify the rankings claimed in Section 5.2.
  4. [Table 2] The 'Improvement' row expresses changes such as recall 0.065 to 0.974 as a 1396.8% relative increase; such percentages are inflated by the small denominator and should be accompanied by absolute differences.

Circularity Check

1 steps flagged · score 5.0 of 10

The headline adversarial-test improvement is measured on HMCorp-adv, generated by the same MIST module whose outputs form the adversarial training augmentation, so Table 2 largely reflects in-distribution fitting to MIST perturbations rather than demonstrated robustness to real human edits; RQ3 provides partial but incomplete mitigation.

  1. fitted input called prediction [Section 3.2 (Adversarial Training Strategy), Section 4.1 (Studied Dataset), and Section 5.1 Table 2]
    "During the adversarial fine-tuning phase... For each target sample in the set, MIST is used to perform an adversarial attack to generate an adversarial sample. ... the samples from the original training set and the generated adversarial samples are mixed at a ratio of 70% original samples and 30% adversarial samples to construct the augmented training set. ... we generate an adversarial test set (HMCorp-adv) by applying MIST to modify samples in the original HMCorp test set. ..."

    The test quantity behind the paper's central robustness claim (accuracy on HMCorp-adv) is generated by the same MIST module that generated the adversarial samples mixed into the augmented training set. Both the training augmentation and the test set use the same MIST perturbation process: identifier replacements selected by importance score and CodeT5 top-k prediction, structure transformations sampled from a style reference table built from the HMCorp training set, and NSGA-II selection under the objectives of Eq. (6).

full rationale

The most important circularity is the train/test alignment: MIST is used both to construct the adversarial training set (Section 3.2) and to construct the adversarial test set HMCorp-adv (Section 4.1). The headline robustness improvement is therefore partially a measure of how well the model fits the exact perturbation distribution it was trained on. This is not a fully external robustness evaluation against human-modified code, despite the paper's repeated framing that HMCorp-adv 'mimic[s] minor code modifications encountered in real-world scenarios.' The paper's own Section 6 acknowledges that real modifications can include function splitting or third-party libraries, which MIST does not model. However, the paper is not wholly circular: original-test-set accuracy remains essentially unchanged, and RQ3 provides cross-attack evidence that MIST-trained models generalize to ALERT, CODA, and MOAA-generated adversarial samples, supporting a weaker claim of robustness across several automated code-perturbation generators in the same family. No load-bearing self-citation chain was found; the reliance on CodeGPTSensor [57] is as base model and baseline, not as the justification of the robustness contribution. Overall, the central real-world robustness claim is partly reduced by construction to fitting MIST's own perturbation distribution, but RQ3 keeps the circularity from being total.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The core method rests on a set of hand-set hyperparameters (r, k, sampling ratio, mix ratio, epochs) and on domain assumptions that the transformation rules preserve semantics and that the adversarial label equals the original provenance label. No engineered constants are fit to optimize the reported accuracy, but the hyperparameters are not sensitivity-analyzed, which leaves room for implicit tuning.

free parameters (5)
  • identifier_replacement_probability_r = 0.5
    Hand-set mutation probability for identifier vs structure transformation; no sensitivity analysis reported (Section 4.4).
  • candidate_identifier_count_k = 40
    Number of CodeT5 candidate identifiers generated per iteration; borrowed from MOAA, not tuned here (Section 4.4).
  • adversarial_sampling_ratio = 10%
    Fraction of the original training set used as targets for adversarial sample generation; chosen to balance computation and accuracy with no reported search (Section 3.2).
  • augmented_train_mix_ratio = 70/30
    Ratio of original to adversarial samples in the augmented training set; no sensitivity analysis reported (Section 3.2).
  • fine_tuning_epochs = 1
    Adversarial fine-tuning is limited to one epoch; no ablation varying epochs is provided (Section 4.4).
assumptions (4)
  • domain assumption The code structure transformation rules in Table 1 preserve syntax and functional equivalence for all target programs.
    Section 3.1.2 and Section 3.1.3 rely on this to claim adversarial samples are syntactically correct and functionality-invariant. Edge cases in Java and Python (for to while with continue, compound assignment on typed or subscripted targets) can violate the equivalence.
  • domain assumption Adversarial samples inherit the provenance label of their original sample: a modified LLM-generated snippet is still LLM-generated, and a modified human-written snippet is still human-written.
    Equation 7 and the surrounding text assign the original true label to each adversarial sample during fine-tuning. This is plausible for provenance but is assumed without empirical check.
  • domain assumption The style reference table, computed from structure distribution frequencies in HMCorp, captures human versus LLM code style, and aligning a sample's structure distribution with the reference improves attack naturalness.
    Section 3.1.2 builds a style reference table and probabilistically transforms structures toward the target source's style. This assumes style statistics on HMCorp generalize and that alignment makes adversarial samples more human-like.
  • domain assumption MIST-generated perturbations are representative of the subtle human modifications that occur in real-world usage of LLM-generated code.
    This underpins the central evaluation in Section 4.1 and Section 5.1. The authors acknowledge in Section 6 that actual modifications can include function splitting or third-party libraries, which MIST does not model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Detecting LLM-generated Code with Subtle Modification by Adversarial Training." pith.science (2026). https://pith.science/paper/TRILML67

@misc{pith2026250713123,
  author       = {Pith},
  title        = {Pith review of: Detecting LLM-generated Code with Subtle Modification by Adversarial Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TRILML67}},
  note         = {Machine review of arXiv:2507.13123}
}
read the original abstract

With the rapid development of Large Language Models (LLMs), their powerful code-generation capabilities have been widely applied in tasks like code completion and automated development, demonstrating the value of improving coding efficiency. However, the extensive use of LLM-generated code also raises several new challenges. On the one hand, issues such as the regulation of code provenance, copyright disputes, and code quality have become increasingly concerning. How to effectively detect LLM-generated code and ensure its compliant and responsible use has become a critical and urgent issue. On the other hand, in practical applications, LLM-generated code is often subject to manual modifications, such as variable renaming or structural adjustments. Although some recent studies have proposed training-based and zero-shot methods for detecting LLM-generated code, these approaches show insufficient robustness when facing modified LLM-generated code, and there is a lack of an effective solution. To address the real-world scenario where LLM-generated code may undergo minor modifications, we propose CodeGPTSensor+, an enhanced version of CodeGPTSensor, which employs adversarial training to improve robustness against input perturbations. CodeGPTSensor+ integrates an adversarial sample generation module, Multi-objective Identifier and Structure Transformation (MIST), which systematically generates both high-quality and representative adversarial samples. This module effectively enhances the model's resistance against diverse adversarial attacks. Experimental results on the HMCorp dataset demonstrate that CodeGPTSensor+ significantly improves detection accuracy on the adversarial test set while maintaining high accuracy on the original test set, showcasing superior robustness compared to CodeGPTSensor.

Figures

Figures reproduced from arXiv: 2507.13123 by the authors.

Figure 1
Figure 1. Detection results of CodeGPTSensor on original and modified test samples 2.1.2 Limited effectiveness of adversarial example generation methods. In recent years, adversarial training has gained widespread attention to improve model robustness [7, 50, 61, 75]. By incor￾porating adversarial examples during the training process, adversarial training enables models to learn the characteristics of these examples, thereby … view at source ↗
Figure 2
Figure 2. The framework of CodeGPTSensor+ 3.1 Adversarial Sample Generation Module This section draws on the ideas of genetic algorithms to design an adversarial sample generation module called MIST, where the Mutation and Crossover operations are key components. The mutation operation introduces small modifications to the original code, such as identifier replace￾ment or structural adjustments, increasing the diversity of th… view at source ↗
Figure 3
Figure 3. shows the composite scores of each adversarial attack method calculated using TOP￾SIS [43] under two weight configurations. The results indicate that when equal weights are assigned to all five metrics (left figure), MIST achieves the highest composite score, significantly outper￾forming other methods. When higher weight is assigned to ASR (right figure), MIST achieves the highest composite score in most cases, surp… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The effect of adversarial samples generated by different adversarial attack algorithms on improving the robustness of the target model In summary, the experimental results validate the superiority of MIST, a multi-objective optimization￾based approach, in generating hi…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Detecting AI Coding Agents in Open Source: A Validated Multi-Method Census of 180 Million Repositories

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    A validated multi-method census of 180M repositories shows AI coding agents generate over 320k commits per month, with bot-account detection recovering only 3.3% of Claude Code activity and commit/PR channels capturin...

  2. OOM-RL: Out-of-Money Reinforcement Learning Market-Driven Alignment for LLM-Based Multi-Agent Systems

    cs.AI 2026-04 unverdicted novelty 5.0 of 10

    OOM-RL aligns multi-agent LLM systems for software engineering by using real financial market losses as an un-hackable negative gradient, resulting in a mature-phase annualized Sharpe ratio of 2.06 via a strict test-d...

Reference graph

Works this paper leans on

79 extracted references · 50 canonical work pages · cited by 2 Pith papers

  1. [1]

    Adam Bannister. 2021. DevSecAI: GitHub Copilot prone to writing security flaws. https://portswigger.net/daily- swig/devsecai-github-copilot-prone-to-writing-security-flaws

    work page 2021

  2. [2]

    Hayden Cheers, Yuqing Lin, and Shamus P Smith. 2019. Spplagiarise: A tool for generating simulated semantics- preserving plagiarism of java source code. In 2019 IEEE 10th International conference on software engineering and service science (ICSESS). IEEE, 617–622

    work page 2019

  3. [3]

    Yutian Chen, Hao Kang, Vivian Zhai, Liangze Li, Rita Singh, and Bhiksha Ramakrishnan. 2023. Gpt-sentinel: Distin- guishing human and chatgpt generated content. arXiv preprint arXiv:2305.07969 (2023)

    arXiv 2023

  4. [4]

    Jonathan H Choi, Kristin E Hickman, Amy Monahan, and Daniel Schwarcz. 2023. Chatgpt goes to law school.A vailable at SSRN (2023). https://doi.org/10.2139/ssrn.4335905

    doi:10.2139/ssrn.4335905 2023

  5. [5]

    Copyleaks. 2023. Copyleaks: AI Content Detector. https://copyleaks.com/ai-content-detector

    work page 2023

  6. [6]

    Kalyanmoy Deb. 2000. A Fast Elitist Non-Dominated Sorting Genetic Algorithm for Mulit-Objedtive Optimization: NSGA-II. KanGAL report 200001 (2000)

    work page 2000

  7. [7]

    Xiaohu Du, Ming Wen, Zichao Wei, Shangwen Wang, and Hai Jin. 2023. An extensive study on adversarial attack against pre-trained models of code. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 489–501

    work page 2023

  8. [8]

    Yunhe Feng, Sreecharan Vanam, Manasa Cherukupally, Weijian Zheng, Meikang Qiu, and Haihua Chen. 2023. In- vestigating Code Generation Performance of Chat-GPT with Crowdsourcing Social Data. In 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC) . IEEE, 876–885

    work page 2023

Show all 79 references
  1. [9]

    Michael Fu, Chakkrit Tantithamthavorn, Trung Le, Van Nguyen, and Dinh Phung. 2022. VulRepair: a T5-based automated software vulnerability repair. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineeri...

    2022

  2. [10]

    Siddhant Garg and Goutham Ramakrishnan. 2020. Bae: Bert-based adversarial examples for text classification. arXiv preprint arXiv:2004.01970 (2020)

    2020 arXiv

  3. [11]

    Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. 2019. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv:1906.04043 (2019)

    2019 arXiv

  4. [12]

    GitHub. 2024. ALERT. https://github.com/soarsmu/attack-pretrain-models-of-code

    2024

  5. [13]

    GitHub. 2024. CODA. https://github.com/tianzhaotju/CODA

    2024

  6. [14]

    GitHub. 2024. MOAA. https://github.com/COLA-Laboratory/MOAA

    2024

  7. [15]

    GitHub. 2024. Tree-sitter. https://tree-sitter.github.io/tree-sitter

    2024

  8. [16]

    GPTZero. 2023. GPTZero. https://gptzero.me/

    2023

  9. [17]

    Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection. arXiv preprint arxiv:2301.07597 (2023)

    2023 arXiv

  10. [18]

    Hello-SimpleAI. 2023. RoBERTa-QA. https://huggingface.co/Hello-SimpleAI/chatgpt-qa-detector-roberta

    2023

  11. [19]

    Jordan Henkel, Goutham Ramakrishnan, Zi Wang, Aws Albarghouthi, Somesh Jha, and Thomas Reps. 2022. Semantic robustness of models of source code. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 526–537

    2022

  12. [20]

    Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho. 2024. Radar: Robust ai-text detection via adversarial learning. Advances in Neural Information Processing Systems 36 (2024), 15077–15095

    2024

  13. [21]

    Raphaël Khoury, Anderson R Avila, Jacob Brunelle, and Baba Mamadou Camara. 2023. How Secure is Code Generated by ChatGPT? arXiv preprint arXiv:2304.09655 (2023)

    2023 arXiv

  14. [22]

    John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models. In International Conference on Machine Learning . PMLR, 17061–17084

    2023

  15. [23]

    Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. 2024. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. Advances in Neural Information Processing Systems 36 (2024), 27469–27500

    2024

  16. [24]

    Taehyun Lee, Seokhee Hong, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, and Gunhee Kim. 2023. Who wrote this code? watermarking for code generation. arXiv preprint arXiv:2305.15060 (2023)

    2023 arXiv

  17. [25]

    Zhen Li, Guenevere Chen, Chen Chen, Yayi Zou, and Shouhuai Xu. 2022. Ropgen: Towards robust code authorship attribution via automatic coding style transformation. In Proceedings of the 44th International Conference on Software Engineering. 1906–1918

    2022

  18. [26]

    Zhen Li, Xiang Huang, Yangrui Li, and Guenevere Chen. 2023. A comparative study of adversarial training methods for neural models of source code. Future Generation Computer Systems 142 (2023), 165–181

    2023

  19. [27]

    Xiaoming Liu, Zhaohan Zhang, Yichen Wang, Hang Pu, Yu Lan, and Chao Shen. 2022. Coco: Coherence-enhanced machine-generated text detection under data limitation with contrastive learning. arXiv preprint arXiv:2212.10341 (2022). ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, A...

    2022 arXiv

  20. [28]

    Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D Le, and David Lo. 2023. Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues.arXiv preprint arXiv:2307.12596 (2023)

    2023 arXiv

  21. [29]

    Zhijie Liu, Yutian Tang, Xiapu Luo, Yuming Zhou, and Liang Feng Zhang. 2023. No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT. arXiv preprint arXiv:2308.04838 (2023)

    2023 arXiv

  22. [30]

    Fatemehsadat Mireshghallah, Justus Mattern, Sicun Gao, Reza Shokri, and Taylor Berg-Kirkpatrick. 2023. Smaller language models are better black-box machine-generated text detectors. arXiv preprint arXiv:2305.09859 (2023)

    2023 arXiv

  23. [31]

    Manning, and Chelsea Finn

    Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature. , 24950–24962 pages. https://arxiv.org/ abs/2301.11305

    2023 arXiv

  24. [32]

    Madhav Nair, Rajat Sadhukhan, and Debdeep Mukhopadhyay. 2023. Generating secure hardware using chatgpt resistant to cwes. Cryptology ePrint Archive (2023). https://eprint.iacr.org/2023/212.pdf

    2023

  25. [33]

    Nguyen, Juri Di Rocco, Claudio Di Sipio, Riccardo Rubei, Davide Di Ruscio, and Massimiliano Di Penta

    Phuong T. Nguyen, Juri Di Rocco, Claudio Di Sipio, Riccardo Rubei, Davide Di Ruscio, and Massimiliano Di Penta

  26. [34]

    Chao Ni, Wei Wang, Kaiwen Yang, Xin Xia, Kui Liu, and David Lo. 2022. The Best of Both Worlds: Integrating Semantic Features with Expert Features for Defect Prediction and Localization. In Proceedings of the 2022 30th ACM Joint Meeting on European Software Engineering Conferen...

    2022

  27. [35]

    Chao Ni, Xin Xia, David Lo, Xiaohu Yang, and Ahmed E. Hassan. 2022. Just-In-Time Defect Prediction on JavaScript Projects: A Replication Study. ACM Transactions on Software Engineering and Methodology (2022)

    2022

  28. [36]

    MT Nietzel. 2023. More than half of college students believe using chatgpt to complete assignments is cheat- ing. https://www.forbes.com/sites/michaeltnietzel/2023/03/20/more-than-half-of-college-students-believe-using- chatgpt-to-complete-assignments-is-cheating/?sh=5d4d763c18f9

    2023

  29. [37]

    OpenAI. 2019. GPT-2 Output Detector. https://github.com/openai/gpt-2-output-dataset/tree/master/detector

    2019

  30. [38]

    OpenAI. 2022. Chatgpt: Optimizing language models for dialogue. https://chat.openai.com

    2022

  31. [39]

    OpenAI. 2023. AI Text Classifier. https://platform.openai.com/ai-textclassifier

    2023

  32. [40]

    Stack Overflow. 2023. Why posting GPT and ChatGPT generated answers is not currently acceptable. https: //stackoverflow.com/help/gpt-policy

    2023

  33. [41]

    Wei Hung Pan, Ming Jie Chok, Jonathan Leong Shan Wong, Yung Xin Shin, Yeong Shian Poon, Zhou Yang, Chun Yong Chong, David Lo, and Mei Kuan Lim. 2024. Assessing AI Detectors in Identifying AI-Generated Code: Implications for Education. arXiv preprint arXiv:2401.03676 (2024)

    2024 arXiv

  34. [42]

    Maryam Vahdat Pour, Zhuo Li, Lei Ma, and Hadi Hemmati. 2021. A search-based testing framework for deep neural networks of source code embedding. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 36–46

    2021

  35. [43]

    PyPI. 2024. TopsisPy. https://pypi.org/project/topsispy/

    2024

  36. [44]

    Md Rafiqul Islam Rabin, Nghi DQ Bui, Ke Wang, Yijun Yu, Lingxiao Jiang, and Mohammad Amin Alipour. 2021. On the generalizability of neural program models with respect to semantic-preserving program transformations. Information and Software Technology 135 (2021), 106552

    2021

  37. [45]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  38. [46]

    Sapling. 2023. Sapling. https://sapling.ai/ai-content-detector

    2023

  39. [47]

    Shashank Srikant, Sijia Liu, Tamara Mitrovska, Shiyu Chang, Quanfu Fan, Gaoyuan Zhang, and Una-May O’Reilly

  40. [48]

    Teo Susnjak. 2022. ChatGPT: The End of Online Exam Integrity? arXiv preprint arXiv:2212.09292 (2022)

    2022 arXiv

  41. [49]

    Yuchuan Tian, Hanting Chen, Xutao Wang, Zheyuan Bai, Qinghua Zhang, Ruifeng Li, Chao Xu, and Yunhe Wang

  42. [50]

    Zhao Tian, Junjie Chen, and Zhi Jin. 2023. Code difference guided adversarial example generation for deep code models. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 850–862

    2023

  43. [51]

    Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. 2023. Ghostbuster: Detecting text ghostwritten by large language models. arXiv preprint arXiv:2305.15047 (2023)

    2023 arXiv

  44. [52]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder- decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 (2021)

    2021 arXiv

  45. [53]

    Writer. 2023. Writer: AI Content Detector. https://writer.com/ai-content-detector/

    2023

  46. [54]

    Kangxi Wu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. 2023. Llmdet: A large language models detection tool. arXiv preprint arXiv:2305.15004 (2023). ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: September 2025. 24 Xin Yin, Xinrui Li,...

    2023 arXiv

  47. [55]

    Chunqiu Steven Xia and Lingming Zhang. 2023. Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. arXiv preprint arXiv:2304.00385 (2023)

    2023 arXiv

  48. [56]

    Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming . 1–10

    2022

  49. [57]

    Xiaodan Xu, Chao Ni, Xinrong Guo, Shaoxuan Liu, Xiaoya Wang, Kui Liu, and Xiaohu Yang. 2024. Distinguishing LLM-generated from Human-written Code by Contrastive Learning. ACM Transactions on Software Engineering and Methodology (2024)

    2024

  50. [58]

    Xianjun Yang, Wei Cheng, Linda Petzold, William Yang Wang, and Haifeng Chen. 2023. Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text. arXiv preprint arXiv:2305.17359 (2023)

    2023 arXiv

  51. [59]

    Xianjun Yang, Liangming Pan, Xuandong Zhao, Haifeng Chen, Linda Petzold, William Yang Wang, and Wei Cheng

  52. [60]

    Xianjun Yang, Kexun Zhang, Haifeng Chen, Linda Petzold, William Yang Wang, and Wei Cheng. 2023. Zero-shot detection of machine-generated codes. arXiv preprint arXiv:2310.05103 (2023)

    2023 arXiv

  53. [61]

    Zhou Yang, Jieke Shi, Junda He, and David Lo. 2022. Natural attack for pre-trained models of code. In Proceedings of the 44th International Conference on Software Engineering . 1482–1493

    2022

  54. [62]

    Noam Yefet, Uri Alon, and Eran Yahav. 2020. Adversarial examples for models of code. Proceedings of the ACM on Programming Languages 4, OOPSLA (2020), 1–30

    2020

  55. [63]

    arXiv preprint arXiv:2310.15654 (2023)

    A survey on detection of llms-generated content. arXiv preprint arXiv:2310.15654 (2023)

    2023 arXiv

  56. [64]

    Xin Yin, Chao Ni, and Shaohua Wang. 2024. Multitask-based evaluation of open-source llm on software vulnerability. IEEE Transactions on Software Engineering (2024)

    2024

  57. [65]

    Xin Yin, Chao Ni, Shaohua Wang, Zhenhao Li, Limin Zeng, and Xiaohu Yang. 2024. Thinkrepair: Self-directed automated program repair. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1274–1286

    2024

  58. [66]

    Xin Yin, Chao Ni, Xiaodan Xu, and Xiaohu Yang. 2025. What You See is What You Get: Attention-Based Self-Guided Automatic Unit Test Generation. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) . IEEE, 1039–1051

    2025

  59. [67]

    Xin Yin. 2025. Replication of this paper. https://github.com/vinci-grape/CodeGPTSensor-plus

    2025

  60. [68]

    Zhengran Zeng, Hanzhuo Tan, Haotian Zhang, Jing Li, Yuqun Zhang, and Lingming Zhang. 2022. An extensive study on pre-trained models for program understanding and generation. InProceedings of the 31st ACM SIGSOFT international symposium on software testing and analysis . 39–51

    2022

  61. [69]

    ZeroGPT. 2023. AI Text Detector. https://www.zerogpt.com

    2023

  62. [70]

    Haolan Zhan, Xuanli He, Qiongkai Xu, Yuxiang Wu, and Pontus Stenetorp. 2023. G3detector: General gpt-generated text detector. arXiv preprint arXiv:2305.12680 (2023)

    2023 arXiv

  63. [71]

    Xiao Yu, Yuang Qi, Kejiang Chen, Guoqiang Chen, Xi Yang, Pengyuan Zhu, Weiming Zhang, and Nenghai Yu. 2023. Gpt paternity test: Gpt generated text detection with gpt genetic inheritance. arXiv preprint arXiv:2305.12519 (2023)

    2023 arXiv

  64. [72]

    Huangzhao Zhang, Zhuo Li, Ge Li, Lei Ma, Yang Liu, and Zhi Jin. 2020. Generating adversarial examples for holding robustness of source code processing models. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 34. 1169–1176

    2020

  65. [73]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022)

    2022 arXiv

  66. [74]

    Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. 2020. Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology (TIST) 11, 3 (2020), 1–41

    2020

  67. [75]

    Huangzhao Zhang, Zhiyi Fu, Ge Li, Lei Ma, Zhehao Zhao, Hua’an Yang, Yizhe Sun, Yang Liu, and Zhi Jin. 2022. Towards robustness of deep program processing models—detection, estimation, and enhancement. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 3 (2022), 1–40

    2022

  68. [79]

    Shasha Zhou, Mingyu Huang, Yanan Sun, and Ke Li. 2024. Evolutionary multi-objective optimization for contextual adversarial example generation. Proceedings of the ACM on Software Engineering 1, FSE (2024), 2285–2308. ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . P...

    2024

  69. [2021]

    arXiv preprint arXiv:2103.11882 (2021)

    Generating adversarial computer programs using optimized obfuscations. arXiv preprint arXiv:2103.11882 (2021)

    2021 arXiv

  70. [2023]

    arXiv preprint arXiv:2305.18149 (2023)

    Multiscale positive-unlabeled detection of ai-generated texts. arXiv preprint arXiv:2305.18149 (2023)

    2023 arXiv

  71. [2024]

    Journal of Systems and Software 214 (2024), 112059

    GPTSniffer: A CodeBERT-based classifier to detect source code written by ChatGPT. Journal of Systems and Software 214 (2024), 112059. https://doi.org/10.1016/j.jss.2024.112059

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.