{"id":"49a50f22-4297-486f-aed6-e9077cd647b7","arxiv_id":"2507.13123","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CodeGPTSensor+, trained with adversarial samples that combine identifier renaming and structure transformation, is substantially more robust to subtle modifications of LLM-generated code than the original CodeGPTSensor.","lead":"The paper adds an adversarial training module called MIST to CodeGPTSensor, a detector of LLM-generated code, so the detector still works after small edits like variable renames or loop rewrites. On the HMCorp benchmark, the upgraded detector raises accuracy on modified code from about 0.26 to 0.97 in Python and from 0.33 to 0.94 in Java while keeping original-test accuracy nearly unchanged.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central real-world robustness claim rests on HMCorp-adv being representative, but HMCorp-adv is generated by the same MIST module used for adversarial training; the observed gains may reflect learning MIST-specific artifacts rather than robustness to human edits.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: HMCorp-adv is generated by the same MIST module whose outputs are used for adversarial training, so the strong Table 2 results may reflect distribution overlap rather than robustness to real human modifications. This is the single most important threat to the paper's central claim, because the entire motivation of the work is the real-world scenario of manual code editing. The paper's internal-validity evidence (RQ3 cross-attack results, detailed methodology, open-source code) is genuine and supports the claim that adversarial training on MIST improves robustness against MIST-family perturbations. That evidence does not, however, establish that MIST's perturbation distribution matches human editing behavior. The Section 6 limitation statement explicitly acknowledges the gap. A conditional verdict is appropriate: the method appears sound within its benchmark, but the real-world robustness claim requires evaluation on independently human-modified code or a held-out perturbation family. I therefore recommend no change to the reader's conditional verdict.","tokens_in":22909,"tokens_out":2623,"duration_ms":35806,"concrete_test":"Evaluate CodeGPTSensor+ and CodeGPTSensor on a held-out set of independently human-modified LLM-generated code, for example pairs of ChatGPT-generated functions and the same functions after actual edits by human developers collected from student submissions, GitHub pull requests, or Stack Overflow edits. If CodeGPTSensor+ retains high accuracy on this human-modified set (comparable to its HMCorp-adv performance), the representativeness concern is largely resolved. If accuracy drops toward the original CodeGPTSensor level, Table 2's gains are at least partly artifacts of training and testing on the same MIST perturbation family. To be decisive, use a few hundred samples per language with edits that go beyond MIST's rule set, including function splitting and third-party library additions mentioned in Section 6.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline improvement (Python adversarial accuracy 0.260 to 0.969, Java 0.325 to 0.944 in Table 2) is measured on HMCorp-adv, a test set constructed by applying MIST to original HMCorp test samples (Section 4.1). The same MIST module generates the adversarial samples used in training: Section 3.2 says adversarial fine-tuning uses MIST to attack 10% of the original training set and mixes those samples into the augmented training set. Thus the training and test distributions are generated by the same perturbation family, against the same target model (the original CodeGPTSensor), and are optimized with the same objectives (Equation 6). The model may therefore learn to recognize MIST-typical identifier substitutions from CodeT5's top-k candidates and MIST's style-aware structure transformations, rather than to detect the broader class of real human minor modifications. The authors' own Section 6 (Threats to External Validity) concedes that MIST does not model function splitting or third-party library introduction, which occur in practice. RQ3 provides some cross-attack evidence: models trained on MIST samples generalize to ALERT, CODA, and MOAA-generated test samples. However, those baselines also operate in the same discrete code-perturbation space and partly share context-aware identifier substitution techniques, so they do not establish representativeness with respect to human edits. The central claim that CodeGPTSensor+ is robust to 'minor modifications in real-world scenarios' therefore rests on the untested assumption that MIST's perturbation distribution is representative of human modification patterns.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CodeGPTSensor+, an adversarially fine-tuned version of the existing CodeGPTSensor detector for LLM-generated code. The enhancement is driven by MIST, a black-box adversarial sample generation module that combines identifier replacement with syntax-preserving structural transformations and uses a multi-objective optimization framework to trade off attack success, semantic distance, and edit distance. The authors use MIST to generate adversarial examples from 10% of the HMCorp training set, mix them with original samples, fine-tune CodeGPTSensor for one epoch, and evaluate on both the original HMCorp test set and a MIST-generated adversarial test set (HMCorp-adv). They report that CodeGPTSensor+ preserves near-original accuracy on unmodified code while raising adversarial-set accuracy from 0.260 to 0.969 on Python and from 0.325 to 0.944 on Java. They also compare MIST against ALERT, CODA, and MOAA on attack metrics, and evaluate cross-attack robustness in RQ3 by adversarial fine-tuning with one attack family and testing on another.","tokens_in":1466,"tokens_out":1678,"duration_ms":61902,"significance":"If the claimed robustness transfers to real human modifications, the paper makes a useful contribution: it shows that a simple adversarial fine-tuning recipe over a multi-objective mixture of identifier and structural perturbations can substantially harden a code-origin classifier, and it ships an open-source implementation and dataset. RQ3 is a genuine strength because it tests MIST-trained models on samples generated by other attack algorithms, which partially breaks the train/test circularity of RQ1. However, the central real-world claim is only as strong as the representativeness of MIST as a model of minor human edits, and the evaluation would need to be supplemented with human-modified code or held-out perturbation families before the robustness claim can be accepted at face value.","major_comments":[{"comment":"The headline robustness result is measured on HMCorp-adv, which is produced by the same MIST module used to create the adversarial training data in Section 3.2. The model therefore trains and tests on the same perturbation family, so the large gains in Table 2 may largely reflect learning MIST-specific artifacts (CodeT5 top-k identifier substitutions and the five structural rules in Table 1) rather than robustness to the broader class of real minor edits. Section 6 itself concedes that real modifications such as function splitting or third-party library introduction are outside MIST's perturbation strategies. RQ3 provides useful cross-attack evidence, but ALERT, CODA, and MOAA also operate in the same discrete code-perturbation space and share context-aware identifier substitution, so they do not establish transfer to human edits. The paper should either evaluate on human-modified code (e.g., a curated set of renamed/refactored functions) or on held-out perturbation types, or explicitly restrict the conclusion to the tested perturbation space.","section":"4.1, 3.2, Table 2"},{"comment":"All experiments report point estimates with no error bars, no multiple seeds, and no significance tests. Several components are stochastic (MIST population initialization, random 10% sampling of the training set, random mixing of adversarial and original samples), so the reported differences, especially the smaller ones in RQ3 (e.g., 0.968 vs 0.970) and the TOPSIS composite scores, need variance estimates. Reporting means and standard deviations over at least five independent fine-tuning runs, together with a paired test where appropriate, would materially strengthen the comparison.","section":"5.1, 5.3"},{"comment":"The claim of MIST's overall superiority is only partially supported by the per-metric results. MIST's ASR on HMCorp-Java against GPTSniffer (48.41%) is lower than MOAA's (59.94%), and on HMCorp-Python MIST's AMQ (61.95) is substantially higher than MOAA's (14.53). The conclusion therefore rests heavily on the TOPSIS aggregation with equal weights; under the ASR-prioritized weighting (ASR weight 0.6) Figure 3 should be inspected for whether MIST remains dominant in all four settings. The paper should present the per-metric tradeoffs and justify the chosen weight configurations rather than stating unqualified superiority.","section":"5.2, Table 3"},{"comment":"Key hyperparameters that likely control the balance between robustness and original accuracy are fixed without any sensitivity analysis: identifier replacement probability r=0.5, candidate count k=40, 10% training-set sampling, 70/30 mix ratio, and one fine-tuning epoch. Since a central claim is that CodeGPTSensor+ 'maintains high accuracy on the original test set,' an ablation varying the mix ratio and r would show whether the reported retention is robust or a lucky point in parameter space.","section":"4.4, 3.2"}],"minor_comments":[{"comment":"The model name is inconsistent: Section 4.4 refers to 'GPTCodeSensor' and 'GPTCodeSensor+' while the rest of the paper uses 'CodeGPTSensor' and 'CodeGPTSensor+'.","section":"4.4"},{"comment":"Reference [6] contains a typo: 'Mulit-Objedtive' should be 'Multi-Objective'.","section":"References"},{"comment":"The TOPSIS composite-score figure is difficult to read in the provided version; larger labels and explicit score values would help the reader verify the rankings claimed in Section 5.2.","section":"Figure 3"},{"comment":"The 'Improvement' row expresses changes such as recall 0.065 to 0.974 as a 1396.8% relative increase; such percentages are inflated by the small denominator and should be accompanied by absolute differences.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The main issue is external validity rather than internal soundness: the RQ1 evaluation is confounded by using the same perturbation generator for training and testing, and the paper would be substantially strengthened by a human-modified benchmark or held-out modification types. The authors should also be asked to provide code/repository verification, since the availability statement is central to the paper's reproducibility contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a competent engineering contribution, but its central robustness claim is weaker than advertised. The big accuracy jump on HMCorp-adv (0.260->0.969 Python, 0.325->0.944 Java) is measured on test samples produced by the same MIST module whose outputs are used in adversarial training. That makes Table 2 partly a test of how well the model fits MIST-specific artifacts, not how well it handles the broader class of human minor edits. The authors' own threats-to-validity section admits MIST does not model function splitting or third-party library introduction, so the real-world generalization claim is an assumption, not a demonstrated result.\n\nWhat is actually new: MIST integrates identifier replacement (importance-score selection plus CodeT5 context-aware prediction) with structure transformations in one multi-objective NSGA-II search. The pieces come from ALERT, MOAA, and CODA, but the combination and the style-aware probabilistic transformation are not in the earlier papers. The paper does the work: clear algorithms, open-sourced code and data, a large benchmark, and RQ3's cross-attack experiment, which shows MIST-trained models do generalize to ALERT/CODA/MOAA adversarial test sets. That is real evidence and should not be dismissed.\n\nSoft spots beyond the circularity: no error bars, no multiple seeds, no significance tests, and hyperparameter sensitivity is not examined (r, k, mix ratio). The cross-attack evidence in RQ3 partially addresses the circularity, but those baselines operate in the same discrete perturbation space and share context-aware identifier substitution, so they don't establish representativeness with respect to actual human edits. The paper would be much stronger with an evaluation on independently human-modified code or a held-out perturbation family.\n\nBottom line: this is a reasonable paper for the LLM-generated code detection community, not a breakthrough. With a careful revision that adds variance reporting and an out-of-distribution evaluation, it could be a solid venue contribution. I'd send it to peer review rather than desk reject, but I would not take the real-world robustness claim at face value.","headline":"A solid engineering paper with a circularity problem: the headline robustness gain is measured on test samples generated by the same MIST module used for training, so the real-world claim is not yet supported.","tokens_in":23789,"tokens_out":2224,"would_cite":true,"duration_ms":25724,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that small human edits such as renaming a variable or converting a loop can hide LLM-generated code from detectors, and that adversarial fine-tuning with matched examples restores detection without hurting accuracy on…","keywords":["LLM-generated code detection","adversarial training","code modification robustness","identifier renaming","structure transformation","multi-objective optimization","CodeGPTSensor"],"falsifier":"Collect a separate test set of genuinely human-modified LLM-generated functions, for example student solutions or developer commits that edited GPT-written code, draw no samples from MIST, and measure CodeGPTSensor+ accuracy there; a return to near-chance accuracy would show the robustness is specific to MIST-style edits rather than to human-style modification generically.","tokens_in":22657,"feed_emoji":"🕵️","tokens_out":4373,"duration_ms":47965,"temperature":0.7,"pith_summary":"The paper is trying to show that the brittleness of LLM-generated-code detectors under small human edits is fixable. It claims that a detector fine-tuned on adversarial samples crafted to resemble such edits, renamed identifiers and loop or branch conversions, keeps its accuracy on unmodified code while nearly recovering its accuracy on edited code. The authors report that accuracy on the modified Python test set rises from 0.260 to 0.969 and on Java from 0.325 to 0.944, with original-set accuracy unchanged at 0.992 and 0.970. If the claim holds, a detector like CodeGPTSensor+ could be deployed where edited code is the norm, such as student submissions or refactored production code.","feed_headline":"Adversarial training revives LLM-code detection after small edits","feed_subtitle":"Renaming variables or converting loops drops accuracy to 0.26; a fine-tuned model recovers to 0.97 on Python and 0.94 on Java.","key_machinery":"The load-bearing object is MIST, a genetic-algorithm-style module that generates the adversarial training samples. Mutation either renames an identifier, selected by importance score and replaced by a CodeT5 context-aware prediction, or applies an equivalent code-structure transformation such as converting a for loop to a while loop, chosen probabilistically to match the coding style of human-written reference samples. Crossover combines identifier-renaming maps of two parents, and NSGA-II non-dominated sorting keeps the N samples that best balance three objectives: adversarial loss (confidence in the true class), semantic distance (cosine similarity of identifiers), and edit distance (Levenshtein distance). This module is what lets the model learn the edit patterns instead of memorizing raw samples.","core_discovery":"CodeGPTSensor+ is CodeGPTSensor adversarially fine-tuned on a 70/30 mix of original training samples and MIST-generated adversarial samples, with 10% of the training set as the source of attacks. The central discovery is that a detection model trained this way resists minor modifications, including variable renaming, loop-to-while conversions, increment and decrement rewrites, and compound assignment expansion, while leaving its behavior on unmodified code essentially intact. The paper also claims that MIST's samples are better attack material than prior methods, achieving higher attack success with fewer identifier changes and fewer model queries, and that fine-tuning on them defends against samples produced by other attack methods as well. The authors present this as a practical route to robustness against the human rewrites that currently defeat state-of-the-art detectors.","pith_inferences":["An independent test set of real human-modified LLM code, collected outside the MIST edit family, would be the decisive check on whether the reported gains reflect human-style modification generally or only MIST-style edits.","The same adversarial training recipe could plausibly harden other code classifiers, such as authorship attribution or vulnerability detectors, against the rewriting tricks that currently evade them.","The style-aware probabilistic transformation mechanism suggests a broader design principle: adversarially trained detectors should be tuned against the edit distribution of the population they will actually face, not just against maximal perturbations."],"forward_implications":["Detection accuracy on subtly modified LLM-generated code jumps from 0.260 to 0.969 on Python and from 0.325 to 0.944 on Java, which the paper measures as relative improvements of 272.8% and 190.8%.","Accuracy on unmodified code stays essentially unchanged at 0.992 on Python and 0.970 on Java, so the robustness gain does not come at the cost of original detection performance.","Models fine-tuned on MIST-generated samples also defend against adversarial samples produced by ALERT, CODA, and MOAA, indicating that the training samples transfer across attack methods.","MIST itself achieves higher attack success rates and lower identifier change rates and query counts than the baselines in most settings, which the paper links to its joint use of structure transformation and multi-objective optimization."],"supporting_citations":[{"why":"Supplies the CodeGPTSensor baseline, the HMCorp dataset, and the original detection task that this paper extends with adversarial training.","marker":"[57]"},{"why":"The ALERT identifier-replacement attack serves as both a baseline for comparison and the inspiration for MIST's context-aware identifier prediction.","marker":"[61]"},{"why":"The MOAA multi-objective adversarial example method is a baseline, and its population size and maximum iteration settings are reused for MIST.","marker":"[75]"},{"why":"The CODA structure-transformation attack is a baseline, and its train/test split methodology is used to evaluate how adversarial training samples improve robustness.","marker":"[50]"},{"why":"NSGA-II provides the non-dominated sorting algorithm that MIST uses to select adversarial samples across its multiple objectives.","marker":"[6]"},{"why":"CodeT5 is the model MIST uses to predict context-aware replacement identifiers for the generated adversarial samples.","marker":"[52]"},{"why":"GPTSniffer is the CodeBERT-based target model used alongside CodeGPTSensor in the adversarial attack evaluation.","marker":"[33]"}],"fun_headline_variants":["Adversarial training shields LLM-code detection from edits","CodeGPTSensor+ beats altered AI code through adversarial training","Renamed, tweaked code? New detector still spots AI origin","Fine-tuning on adversarial samples revives AI-code detection","Robust detector survives manual tweaks to LLM-generated code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the HMCorp-adv test edits, produced by the same MIST module used to create training samples, represent the small changes humans actually make to LLM-generated code; if real edits come from a different distribution, such as function splitting or third-party libraries, the reported gains may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Adversarial training shields LLM-code detection from edits","CodeGPTSensor+ beats altered AI code through adversarial training","Renamed, tweaked code? New detector still spots AI origin","Fine-tuning on adversarial samples revives AI-code detection","Robust detector survives manual tweaks to LLM-generated code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00016,"raw_usage":{"total_tokens":1260,"prompt_tokens":1004,"completion_tokens":256,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":172}},"tokens_in":620,"tokens_out":256,"duration_ms":3012,"temperature":1.0,"reasoning_tokens":172,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:30:04.721933+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a separate test set of genuinely human-modified LLM-generated functions, for example student solutions or developer commits that edited GPT-written code, draw no samples from MIST, and measure CodeGPTSensor+ accuracy there; a return to near-chance accuracy would show the robustness is specific to MIST-style edits rather than to human-style modification generically.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CodeGPTSensor baseline, the HMCorp dataset, and the original detection task that this paper extends with adversarial training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The ALERT identifier-replacement attack serves as both a baseline for comparison and the inspiration for MIST's context-aware identifier prediction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The MOAA multi-objective adversarial example method is a baseline, and its population size and maximum iteration settings are reused for MIST."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The CODA structure-transformation attack is a baseline, and its train/test split methodology is used to evaluate how adversarial training samples improve robustness."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NSGA-II provides the non-dominated sorting algorithm that MIST uses to select adversarial samples across its multiple objectives."},{"cited_title":"Nguyen, Juri Di Rocco, Claudio Di Sipio, Riccardo Rubei, Davide Di Ruscio, and Massimiliano Di Penta","cited_arxiv_id":null,"evidence_quote":"GPTSniffer is the CodeBERT-based target model used alongside CodeGPTSensor in the adversarial attack evaluation."}],"review_version":1}