{"id":"83e9dfbf-7447-4c0d-8b6d-75ba4bf54651","arxiv_id":"2505.20158","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Combining Token Sequence Normalization and Subsequence Match Merging substantially improves JPlag's resilience to automated obfuscation, including AI-based rewriting, without raising false-positive rates.","lead":"This paper tests two add-on defenses for the plagiarism detector JPlag against automated code obfuscation, including GPT-4-based rewriting, using more than four million program comparisons. It finds that combining both defenses sharply improves detection of obfuscated plagiarism and slightly improves separation of AI-generated from human code, with little effect on false positives.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Load-bearing concern is external validity: Section 4.4's synthetic obfuscations are the only plagiarism instances, yet Section 5's gains vary from 99.65 pp (insertion) to 19 pp (AI-based). If real student obfuscation resembles the AI-based end, the practical benefit is much smaller.","rationale":"I reviewed the strongest claim and the evaluation. The most load-bearing condition is that synthetic obfuscations represent real students; this is acknowledged in Section 6 but not tested. The internal statistics are mostly careful: no arbitrary thresholds, effect sizes reported, six datasets, and over four million pair comparisons. Minor internal inconsistencies exist (Table 2 counts do not reconcile with the text's 787/851 figures, and the Q5 label repeats in Section 5.6), but they do not threaten the central direction. The external-validity concern is the one that would change practical conclusions, especially because the AI-based attack results are the weakest part of the evaluation. The proposed test uses the authors' own filtered labeled data, so it is cheap and decisive. Since the reader already conditioned on exactly this assumption, the verdict remains unchanged.","tokens_in":34561,"tokens_out":8694,"duration_ms":99233,"concrete_test":"Re-run the evaluation on the labeled human plagiarism pairs in the Ljubovic-Pajic Homework datasets (which the authors removed during preprocessing), computing the same Δmedian and Cliff's delta for the JPlag baseline vs TSN+SMM against the original-pair distribution. If the real labeled pairs show separation gains comparable to the synthetic insertion/refactoring results, the representativeness objection is answered; if gains resemble the weak AI-based results (or vanish), the claim must be narrowed to specific attack types.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 5 that TSN+SMM 'highly effectively' separates obfuscated plagiarism from unrelated pairs is measured entirely on plagiarism instances synthesized in Section 4.4. No evaluation uses known human plagiarism instances: the authors explicitly filter labeled human plagiarism out of the Ljubovic-Pajic datasets and remove verbatim copies elsewhere (Section 4.3). The assumption that MOSSad/PlagGen insertions, Spoon refactorings, and 15 GPT-4 prompts span realistic student behavior is therefore load-bearing. It is not merely philosophical: measured gains differ by an order of magnitude across attack types (Table 4: Δmedian up to 99.65 pp for insertion; Table 6: 22 pp for refactoring; Table 8: 19 pp for AI), and for AI-based obfuscation on the C++ Homework datasets the combined defense shows negligible or non-significant effects (Table 9: Homework-1 δ=0.024, Homework-5 p=0.11). If students today obfuscate more like GPT-4 than like dead-code insertion, the practical resilience gain is much smaller than the headline suggests. Section 6 acknowledges generalizability limitations but does not test them.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper evaluates two defense mechanisms for token-based plagiarism detectors, Token Sequence Normalization (TSN) and Subsequence Match Merging (SMM), against automated obfuscation attacks. The evaluation uses JPlag as the baseline detector and six real-world student datasets in Java and C++, covering dead-code insertion (via PlagGen and MOSSad), Spoon-based semantic-preserving refactoring, GPT-4-based obfuscation with 15 prompts, and GPT-4-based whole-program generation. The primary outcome metric is the similarity separation between plagiarism-to-source pairs and unrelated original pairs, reported via median/mean differences, Wilcoxon signed-rank tests, and Cliff's delta with confidence intervals across more than 4.1 million pairwise comparisons. The central claim is that the combined TSN+SMM defense substantially increases this separation for insertion-based attacks (up to 99.65 percentage points), moderately for refactoring attacks (up to 22 points), and to a smaller but still significant degree for GPT-4-based obfuscation (up to 19 points) and AI-generated programs (up to 8.92 points), while leaving unrelated-pair similarity practically unchanged. The paper also discusses implications for academic integrity and provides a replication package.","tokens_in":34734,"tokens_out":9399,"duration_ms":112204,"significance":"The evaluation is carefully designed: it avoids arbitrary similarity thresholds, uses multiple real-world datasets, reports both statistical and practical significance with confidence intervals, and packages the artifacts for replication. The separation-metric framing is a methodological improvement over threshold-based evaluations that are sensitive to chosen cutoffs. If the results hold, the defenses offer a low-cost upgrade for JPlag and, plausibly, other token-based detectors, with the strongest benefit against insertion-based obfuscation and a measurable but weaker benefit against AI-based obfuscation and generation. The main limitations are that only JPlag is used as the detector, all plagiarism instances are synthetically generated rather than taken from real student plagiarism, and the AI-based attack results for the C++ Homework datasets are weak or non-significant. These limitations constrain the strength of the paper's broader practical conclusions but do not invalidate the core empirical comparison.","major_comments":[{"comment":"The external validity of the obfuscation results is a load-bearing concern. As stated in Section 6, 'all plagiarism instances are generated through controlled automated obfuscation', and Section 4.3 explains that labeled human plagiarism was deliberately removed from the Ljubovic-Pajic datasets. The measured gains vary by roughly an order of magnitude across attack types, from 99.65 percentage points for insertion-based attacks (Table 4) to about 19 points for GPT-4 obfuscation (Table 8). For the C++ Homework datasets under GPT-4 obfuscation, the combined defense shows negligible or non-significant effects (Table 9, Homework-1: delta=0.024; Homework-5: p=0.11). Since the paper's Section 5 conclusion that the defenses are 'highly effective across a range of automated obfuscation attacks' supports practical academic-integrity recommendations, the lack of any evaluation on real student obfuscations leaves the practical magnitude of the benefit uncertain. A concrete test using the labeled human plagiarism instances that were filtered during preprocessing, or a sensitivity analysis with alternative obfuscation distributions, would directly address this threat to validity.","section":"Sections 4.3, 6; Tables 8-9"},{"comment":"The paper claims in Section 3 that TSN and SMM are 'agnostic to the underlying detection system' and suitable for integration into any state-of-the-art token-based detector such as MOSS, JPlag, or Dolos, but all empirical results in Section 5 are obtained with JPlag only. Section 4.2 justifies excluding MOSS and Dolos for practical reasons, and Figure 4 demonstrates that all three tools are vulnerable to insertion-based obfuscation at baseline, yet no experiment reports the defense mechanisms applied to a second detector. If the contribution is intended to be detector-agnostic, the generalizability claim is under-supported. At minimum, the evaluation should include a second open-source token-based detector, for example Dolos on the single-file datasets, or the claims should be narrowed to JPlag.","section":"Sections 3, 4.2; Section 5"}],"minor_comments":[{"comment":"The row labeled 'Alteration-based Obf.' is not described in Section 4.4 and its counts duplicate the 'Insertion-based Obf.' row. The caption states 851 programs in total, which matches the sum of the other four rows, not five rows; the row should be removed or explained, and all related totals should be reconciled.","section":"Table 2"},{"comment":"The text says the four techniques create 787 plagiarized programs, while Table 2's caption says 851 in total; the discrepancy should be resolved by clarifying which instances were excluded, for example due to invalid GPT-4 outputs.","section":"Section 4.4"},{"comment":"The answer block at the end of Section 5.6 is labeled 'Answer to Q5' but should be 'Answer to Q6'.","section":"Section 5.6"},{"comment":"In the paragraph on Token Sequence Normalization, the phrase 'has little to no effect effect' contains a duplicated word and should be corrected.","section":"Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"The empirical core is solid, and the replication package is a genuine strength. The main risk is that the title and Discussion generalize beyond the evaluation's attack and detector scope: all obfuscations are synthetic, and only JPlag is tested. These concerns are fixable, either by adding evidence on a second detector and on real/human obfuscations or by substantially narrowing the claims. I do not see grounds for rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the quick take on arXiv:2505.20158. The paper evaluates two defense mechanisms (Token Sequence Normalization and Subsequence Match Merging) against automated obfuscation attacks, including GPT-4 rewriting, across six real-world datasets and over four million pairwise runs. The defenses themselves come from the authors' earlier papers; what's new here is the systematic, multi-attack evaluation of the combination, plus the AI-generated code angle.\n\nThe methodology is genuinely good. The authors avoid fixed thresholds, report one-sided Wilcoxon tests with Cliff's delta and confidence intervals, use six datasets of different sizes/languages, and ship a replication package. For insertion-based and refactoring-based obfuscation, the combined defense produces large, consistent improvements in separation between plagiarism pairs and unrelated pairs. That part of the story holds up well, and it's useful evidence for educators who rely on JPlag.\n\nNow the soft spots, in proportion. The stress-test concern is on target: every plagiarism instance is synthetic, generated by MOSSad/PlagGen, Spoon refactorings, or GPT-4 prompts. The authors deliberately filter out any human-labeled plagiarism. So the measured resilience is specific to these attack generators, not necessarily to what students actually do. The magnitude of the gains varies wildly by attack type: up to 99.65 percentage points for insertion, 22 for refactoring, but only 19 for AI-based obfuscation. And on the C++ homework datasets, the AI-based results for the combined defense are effectively negligible or non-significant (Homework-1 δ=0.024; Homework-5 p=0.11). If real student behavior today is closer to the AI-rewriting end, the practical pay-off is much smaller than the headline numbers suggest. The paper acknowledges this generalizability limitation in Section 6 but doesn't test it, so it remains a genuine open question. Also, only JPlag is evaluated; the claims that MOSS and Dolos would behave similarly are plausible but unverified.\n\nThat said, these are standard external-validity caveats for this kind of benchmark, and the paper is upfront about most of them. The central claim about effectiveness against insertion and refactoring attacks is well supported. The AI-based results are honestly reported as more modest.\n\nVerdict: worth a serious referee. I'd send it out, but the reviewer should push for a clear statement about the scope of the conclusions and, if possible, some sanity check on real plagiarism instances. It's not a desk reject, and it's not a strong accept as-is either.","headline":"Solid broad evaluation of two existing defenses, but the external validity caveat about synthetic obfuscations is real and matters most for the AI-based results.","tokens_in":35325,"tokens_out":3163,"would_cite":true,"duration_ms":29984,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Layered defenses beat AI code obfuscation in plagiarism checks","keywords":["source code plagiarism","plagiarism detection","obfuscation attacks","token sequence normalization","subsequence match merging","JPlag","GPT-4 obfuscation","AI-generated code"],"falsifier":"One concrete test: collect confirmed, naturally obfuscated student plagiarism pairs from academic-integrity case files and measure the median similarity separation under JPlag with TSN+SMM; if the separation is close to the 10-20 point range typical of unrelated pairs, the defense gains do not transfer to real student behavior. Another test: run the same pipeline against obfuscation from a newer LLM that rewrites code idiomatically rather than inserting dead statements; if few fragmented matches remain, SMM will have little to repair and the measured resilience will drop.","tokens_in":34295,"feed_emoji":"🤖","tokens_out":6848,"duration_ms":66713,"temperature":0.7,"pith_summary":"The paper sets out to show that two existing defenses against code-plagiarism obfuscation, Token Sequence Normalization (TSN) and Subsequence Match Merging (SMM), remain effective when combined and when tested against a broader threat model than before, including LLM-based attacks. Across more than four million pairwise comparisons on six real course datasets, the authors find that the combined defenses widen the similarity gap between plagiarized pairs and unrelated pairs for insertion-based, refactoring-based, and GPT-4-based obfuscation. The gains matter because automated obfuscation has made token-based detectors such as JPlag much easier to evade; if the results hold in practice, educators can substantially harden existing tools with two relatively cheap, language-independent post-processing steps. A secondary result is that the same defenses improve the separation between AI-generated and human-written programs, even though they were not designed for that purpose.","feed_headline":"Layered defenses beat AI code obfuscation in plagiarism checks","feed_subtitle":"Token normalization plus match merging lifts median separation by up to 99.65 points, with no false-positive cost.","key_machinery":"The load-bearing machinery is a two-stage pipeline wrapped around a token-based detector such as JPlag. TSN first builds a Token Normalization Graph, a graph-based abstraction of token interdependencies, to identify and remove dead-code nodes and to topologically sort reordered tokens, effectively de-obfuscating the token sequence before comparison. SMM then operates on the matching results, heuristically merging neighboring matched subsequences and iteratively filling the gaps that obfuscation has introduced, until no more neighboring matches can be merged. TSN is attack-specific, targeting structural insertions and reordering, while SMM is attack-independent, and their combination is what produces the measured separation.","core_discovery":"On the paper's own terms, the central discovery is that the two defenses are complementary rather than redundant: TSN virtually eliminates insertion-based obfuscation (median similarity separation up to 99.65 percentage points on Java datasets), SMM carries most of the gains against refactoring-based attacks (up to 22 percentage points), and the combined pipeline still yields substantial improvements against GPT-4-based obfuscation (up to 19 percentage points) where either mechanism alone struggles. The same combined pipeline improves the median similarity separation between AI-generated and human programs by up to 8.92 percentage points, and it does so while leaving the similarity of unrelated, independently written programs practically unchanged, with measured effect sizes on unrelated pairs ranging from negligible to small. The authors therefore claim that token-based plagiarism detectors can be made substantially more resistant to automated and AI-assisted plagiarism without a meaningful false-positive cost, and that threshold-based obfuscation, which keeps inserting dead code until a similarity target is reached, becomes too slow and too conspicuous to be practical.","pith_inferences":["If real student obfuscation resembles the insertion and refactoring patterns tested here, then adopting the TSN+SMM pipeline in existing JPlag-like systems is a low-risk hardening step; but the paper's own threat-model discussion implies that a shift toward full reimplementation or idiomatic semantic rewrites would bypass both defenses.","The improved separation of AI-generated programs suggests a practical side channel for detecting LLM-assisted submissions: when many students prompt the same model for the same assignment, the resulting programs are more similar to each other than human work, and SMM amplifies that signal.","A natural next test is to re-run the same pipeline against newer LLMs; as model outputs become more varied and less insertion-laden, SMM's gap-merging heuristic may find fewer fragmented matches to repair, so the measured resilience should be expected to decay.","The layering logic generalizes beyond plagiarism detection: attack-specific countermeasures should be paired with attack-independent heuristics so that unknown or emerging transformations still leave some detectable trace."],"forward_implications":["Insertion-based obfuscation, currently one of the cheapest ways to defeat JPlag, is rendered near-ineffective: the combined defenses achieve up to 99.65 percentage points of median similarity separation and eliminate measurable overlap between plagiarized and unrelated pairs.","Refactoring-based obfuscation is substantially mitigated through SMM, with median separation gains up to 22 percentage points, even though TSN alone adds almost nothing for this attack.","LLM-based obfuscation remains the hardest regime: gains reach only about 19 percentage points and vary by dataset and prompt, so AI-rewritten code still requires more human review than algorithmically obfuscated code.","The defenses also separate AI-generated from human submissions by up to 8.92 percentage points of median similarity, meaning the same tooling can flag suspicious clusters of LLM-written programs even without dedicated AI detectors.","Threshold-based obfuscation becomes slow and visible: MOSSad required up to roughly six hours per small C++ program and inflated code by up to about 1300 percent, making the obfuscated output conspicuous by size alone."],"supporting_citations":[{"why":"Introduces Token Sequence Normalization and the Token Normalization Graph, the attack-specific defense under evaluation.","marker":"[60]"},{"why":"Introduces Subsequence Match Merging, the attack-independent defense under evaluation.","marker":"[63]"},{"why":"Provides MOSSad, the threshold-based insertion obfuscation tool used for C++ attacks and for the threshold-based runtime analysis.","marker":"[18]"},{"why":"Provides PlagGen, the deterministic insertion obfuscation tool used for Java datasets.","marker":"[8]"},{"why":"Provides the Spoon-based refactoring attack used to generate refactoring-based obfuscation.","marker":"[40]"},{"why":"Supplies GPT-4, the model used for both AI-based obfuscation and AI-based generation attacks.","marker":"[1]"},{"why":"Supplies the PROGpedia datasets, two Java-based real course assignment collections used in the evaluation.","marker":"[48]"},{"why":"Supplies the Ljubovic and Pajic homework datasets, two C++-based real course assignment collections used in the evaluation.","marker":"[37]"},{"why":"Defines JPlag, the token-based baseline detector whose similarity scores all defense variants are compared against.","marker":"[54]"}],"fun_headline_variants":["Token normalization plus match merging lifts detection against AI obfuscation","Combined defenses beat GPT-4 obfuscation, no false positives","Layered defenses lift plagiarism detection, no false-positive spike","Merging and normalization defuse AI code obfuscation in plagiarism"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation's external validity rests on the assumption that the synthetic plagiarism instances generated by MOSSad, PlagGen, Spoon-based refactorings, and GPT-4 prompts are representative of the ways real students actually obfuscate copied assignments.","fun_headline_variants_meta":{"raw":{"variants":["Token normalization plus match merging lifts detection against AI obfuscation","Combined defenses beat GPT-4 obfuscation, no false positives","Layered defenses lift plagiarism detection, no false-positive spike","Merging and normalization defuse AI code obfuscation in plagiarism"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001085,"raw_usage":{"total_tokens":4552,"prompt_tokens":978,"completion_tokens":3574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":3497}},"tokens_in":594,"tokens_out":3574,"duration_ms":28420,"temperature":1.0,"reasoning_tokens":3497,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:58:24.046028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete test: collect confirmed, naturally obfuscated student plagiarism pairs from academic-integrity case files and measure the median similarity separation under JPlag with TSN+SMM; if the separation is close to the 10-20 point range typical of unrelated pairs, the defense gains do not transfer to real student behavior. Another test: run the same pipeline against obfuscation from a newer LLM that rewrites code idiomatically rather than inserting dead statements; if few fragmented matches remain, SMM will have little to repair and the measured resilience will drop.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides MOSSad, the threshold-based insertion obfuscation tool used for C++ attacks and for the threshold-based runtime analysis."},{"cited_title":"Preventing Automatic Code Plagiarism Generation Through Token String Normalization","cited_arxiv_id":null,"evidence_quote":"Provides PlagGen, the deterministic insertion obfuscation tool used for Java datasets."},{"cited_title":"Preventing Refactoring Attacks on Software Plagiarism Detec- tion through Graph-Based Structural Normalization","cited_arxiv_id":null,"evidence_quote":"Provides the Spoon-based refactoring attack used to generate refactoring-based obfuscation."},{"cited_title":"Gpt-4 technical report","cited_arxiv_id":null,"evidence_quote":"Supplies GPT-4, the model used for both AI-based obfuscation and AI-based generation attacks."},{"cited_title":"Plagiarism detection in computer programming using feature extraction from ultra-fine-grained repositories","cited_arxiv_id":null,"evidence_quote":"Supplies the Ljubovic and Pajic homework datasets, two C++-based real course assignment collections used in the evaluation."},{"cited_title":"Finding plagiarisms among a set of programs with jplag","cited_arxiv_id":null,"evidence_quote":"Defines JPlag, the token-based baseline detector whose similarity scores all defense variants are compared against."}],"review_version":1}