Pith. sign in

REVIEW 3 major objections 5 minor 19 references

k-gram software birthmarks still catch LLM-paraphrased Java code, especially with short k and set-based similarity.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 05:26 UTC pith:F4VD3EWQ

load-bearing objection Solid empirical stress-test of classic opcode k-grams against three modern LLMs; the absolute “remain effective” numbers rest on best-ε Hmean with no holdout, but the relative trends look real and useful. the 3 major comments →

arxiv 2607.03014 v1 pith:F4VD3EWQ submitted 2026-07-03 cs.SE

Detection of LLM-assisted Code Plagiarism Using k-gram Software Birthmarks

Nikolay Fedorov , Akito Monden , Hiroki Inayoshi , Haruaki Tamada , Masateru Tsunoda This is my paper
classification cs.SE
keywords Javacode plagiarismcode clonecode paraphrasingplagiarism detectionsoftware birthmarkLLM-assisted plagiarismk-gram
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models can rewrite existing programs so they look different while keeping the same behavior, which threatens open-source licensing and ordinary plagiarism checks that rely on source text. This paper asks whether a classical executable-side fingerprint—the k-gram software birthmark built from unique short sequences of Java opcodes—can still recognize such paraphrased clones. Using three current models to rewrite compilable Java files drawn from actively maintained BSD-2-Clause projects, the authors extract birthmarks for k from 1 to 6 and compare them with several similarity measures. They find that the birthmarks remain effective: short sequences (especially 2-grams) paired with set-based measures such as the Dice index produce high harmonic means of resilience and credibility, while sequence-edit distance fails. Coding-oriented models produce harder clones, but none of the three models fully erase the birthmark signal.

Core claim

k-gram software birthmarks extracted from Java opcodes remain effective against LLM-assisted code paraphrasing. Across three contemporary models, short k values—especially 2-grams—combined with set-based similarity (Dice best) yield strong detection performance measured by the harmonic mean of resilience and credibility, whereas edit-distance similarity collapses and larger k gradually loses power. ChatGPT-5.1-Codex-Mini produces the most difficult-to-detect clones among the models tested.

What carries the argument

The k-gram software birthmark: the set of unique contiguous length-k sequences of Java bytecode opcodes extracted from a compiled class. Similarity between two such sets is scored by cosine, Dice, Jaccard, Simpson, or Levenshtein-derived measures; a threshold then decides plagiarism versus independence.

Load-bearing premise

Performance is reported as the best harmonic mean obtainable by choosing the decision threshold on the entire similarity set with no held-out split, so the numbers may overstate what a fixed or trained threshold would achieve in deployment.

What would settle it

Re-run the same LLM paraphrasing and birthmark pipeline with a fixed or cross-validated decision threshold on a held-out portion of the files (or a larger multi-license corpus) and check whether 2-gram Dice Hmean remains high enough to separate paraphrased pairs from unrelated pairs.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates whether k-gram software birthmarks (unique contiguous sequences of Java bytecode opcodes, k = 1..6) remain effective against LLM-assisted code plagiarism realized by code paraphrasing. Using three contemporary LLMs (ChatGPT-5.1-Codex-Mini, DeepSeek-V4-Flash, Claude-Haiku-4.5) and a filtered set of individually compilable Java source files drawn from actively maintained BSD-2-Clause GitHub projects, the authors extract birthmarks via javap/ASM, compare them with six similarity functions (cosine-count, cosine-TF-IDF, Dice, Jaccard, Simpson, Levenshtein-based), and report harmonic means of resilience and credibility rates. They conclude that short k (especially 2-gram) combined with set-based measures (Dice best) yields high Hmean, that ChatGPT produces the hardest-to-detect clones, and that coding-oriented models are stronger at the plagiarism task.

Significance. If the absolute effectiveness numbers hold under a realistic decision protocol, the work supplies timely empirical evidence that a simple, executable-level birthmark can still flag LLM-paraphrased Java modules. The multi-LLM, multi-k, multi-similarity design and the explicit resilience/credibility framing are useful contributions for the software-protection and open-source-compliance communities. The relative ranking of k and of set-based versus sequence-based similarity is already informative even if absolute Hmeans are optimistic.

major comments (3)
  1. §4.4 (and the Hmean definition that follows): the central claim that birthmarks “remain effective” rests on reporting the single best Hmean obtained by sweeping the decision threshold ε over the entire balanced similarity set for every (k, similarity, LLM) configuration, with no train/test split or fixed-threshold protocol. With only 352 LLC-filtered files (further reduced by the compile failures in Table 2), this post-hoc selection can substantially inflate absolute performance relative to any deployment setting in which ε must be chosen without seeing the test pairs. Relative rankings of k and of Dice vs. edit-distance may survive, but the numerical support for “effective” (e.g., Dice avg Hmean ≈ 0.946) is not secured by the reported protocol. A held-out or cross-validated threshold procedure, or at least a sensitivity plot of Hmean versus ε, is required before the absolute claim can b
  2. Abstract vs. §2.2 / Figure 2: the abstract states that five similarity measures are compared, yet the body evaluates six (two cosine variants plus Dice, Jaccard, Simpson, edit-distance). The discrepancy should be resolved and the abstract made consistent with the actual experimental design.
  3. §4.3–4.5 and Table 1–2: after LLC and compilability filtering the effective sample is modest (352 originals, fewer after LLM compile failures). Credibility comparisons use “an equal number of unrelated files,” but the paper never states how those negatives are sampled (same project? different projects? size-matched?). Without that detail it is impossible to judge whether the high credibility rates are partly an artifact of easy negatives. Clarification and, if necessary, a more stringent negative-sampling regime are needed.
minor comments (5)
  1. Figure 1 and Figure 2 lack error bars, confidence intervals, or any indication of variance across files/LLMs; adding them would strengthen the visual claims.
  2. §4.3: repository collection date is given as “June 14th, 2026” while the arXiv stamp is 3 Jul 2026; a brief note on the exact crawl window would avoid confusion.
  3. Appendix A is announced but the actual prompt text is missing from the supplied manuscript; it should be included for reproducibility.
  4. Several minor wording issues: “semantically identical program” (abstract) should be plural; “out previous work” (§4.3) → “our”; “paraphrased code that were” (§6) → “that was”.
  5. The claim in the abstract that findings “confirm the higher performance of coding-oriented models for plagiarism task” is only weakly supported by the three-model comparison; a more cautious phrasing would be preferable.

Circularity Check

1 steps flagged

Mild protocol optimism from best-ε Hmean on the full set; no derivation or self-definitional circularity in the empirical claims.

specific steps
  1. fitted input called prediction [§4.4 Evaluation Procedure (threshold rule and Hmean reporting)]
    "Because the available dataset is relatively small, we do not divide the data into separate training and testing subsets. Instead, for each experimental configuration, we report the best achievable Hmean obtained from the calculated similarity values. This approach allows us to compare the relative effectiveness of different k-gram configurations, similarity functions, and LLMs under identical conditions."

    ε is chosen post-hoc to maximize Hmean on the identical set of pairwise similarities that define the TP/TN/FP/FN counts. The reported effectiveness figure is therefore the maximum attainable under that selection rather than an out-of-sample or fixed-threshold measurement; absolute Hmean numbers are partly a function of the evaluation observations themselves.

full rationale

This is an empirical comparison of k-gram birthmarks (k=1..6) and five similarity measures against LLM-paraphrased Java files, not a first-principles derivation. Resilience/credibility rates and Hmean are computed from observed birthmark similarities; the central claim that short-k set-based measures remain effective is supported by those measurements, not forced by definition. Self-citations to the authors’ prior work supply the similarity functions and LLC filter but are not load-bearing for the LLM-detection numbers. The sole mild issue is the evaluation protocol that selects, for every configuration, the ε maximizing Hmean on the same similarity values used to score performance (no hold-out). That makes absolute Hmean values optimistic upper bounds by construction, yet relative rankings of k and of set vs. edit-distance remain informative. No self-definitional loop, uniqueness theorem, ansatz smuggling, or renaming of a known result is present. Score 2 reflects only this limited fitted-threshold practice.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests on standard birthmark theory, a specific evaluation protocol with hand-chosen filters and best-case thresholds, and the assumption that LLM paraphrases of standalone compilable Java files represent the plagiarism threat of interest. No new physical entities; free parameters are experimental knobs (k, ε, LLC cutoffs) that directly shape reported Hmean.

free parameters (4)
  • similarity threshold ε
    Decision rule Sim(X,Y) with 0.5≤ε<1; paper reports best achievable Hmean over similarity values rather than a pre-registered or cross-validated ε, so reported performance depends on this choice.
  • k-gram length k
    Scanned from 1 to 6; conclusions privilege k=2/3 based on observed Hmean on this dataset.
  • logical line count (LLC) bounds
    Files kept only if LLC >30 and ≤1000; these cutoffs define the evaluation population and are taken from prior author work plus context-window practicality.
  • repository popularity/maintenance filters
    >100 GitHub stars and push since 2025-06-14; hand-chosen inclusion criteria that determine which 352 files enter the study.
axioms (4)
  • domain assumption A resilient, credible software birthmark can be defined from unique contiguous opcode k-grams of compiled Java bytecode (Myles & Collberg lineage).
    §2–2.1 treats this representation as the detector under test; effectiveness against LLM paraphrasing is the empirical question, not re-proved.
  • domain assumption Module-wise, individually compilable source files are an appropriate proxy for real-world executable plagiarism scenarios.
    §4.3 justifies focusing on standalone compilable files because malicious actors may not publish source; whole-project or dependency-rich plagiarism is out of scope by construction.
  • ad hoc to paper Best Hmean over thresholds on the full balanced TP/TN set fairly ranks configurations when the dataset is small.
    §4.4 explicitly adopts this protocol instead of train/test split; it is a methodological choice that loads the effectiveness claim.
  • domain assumption LLM outputs that fail to compile can be discarded without invalidating conclusions about detectability of successful paraphrases.
    §4.5 excludes non-compilable outputs because birthmarks require bytecode; residual selection bias is acknowledged only lightly.

pith-pipeline@v1.1.0-grok45 · 14360 in / 3292 out tokens · 28743 ms · 2026-07-12T05:26:29.054681+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) have significantly lowered the technical barrier to software plagiarism. By transforming existing source code while preserving its functionality, modern LLMs can generate semantically identical program that may evade traditional plagiarism detection techniques. Among such attacks, code paraphrasing modifies the syntax and structure of a program while preserving its behavior. This paper investigates whether software birthmarks can detect such LLM-assisted plagiarism. As a starting point, we employ k-gram software birthmarks based on unique k-grams of Java opcodes, with k ranging from 1 to 6. We employ three contemporary LLMs: ChatGPT-5.1-Codex-Mini, DeepSeek-V4-Flash, and Claude-Haiku-4.5. The dataset consists of individually compilable source files extracted from actively maintained BSD-2-Clause licensed Java projects. We further compare five similarity measures for birthmark matching: cosine similarity, Dice index, Jaccard coefficient, Simpson index, and edit-distance-based similarity. The results demonstrate that k-gram software birthmarks remain effective for detecting LLM-assisted plagiarism. Among the evaluated models, ChatGPT-5.1-Codex-Mini generated the most difficult-to-detect clones. Furthermore, the findings confirm the higher performance of coding-oriented models for plagiarism task.

Figures

Figures reproduced from arXiv: 2607.03014 by Akito Monden, Haruaki Tamada, Hiroki Inayoshi, Masateru Tsunoda, Nikolay Fedorov.

Figure 1
Figure 1. Figure 1: Detection performance of k-gram birthmarks against LLM-generated code clones in terms [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of similarity functions for plagiarism detection in terms of Hmean (excluding [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Impact of LLM selection on plagiarism detection performance measured using Hmean [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 4 canonical work pages

  1. [1]

    Dynamic K-Gram Based Software Birthmark

    Yameng Bai et al. “Dynamic K-Gram Based Software Birthmark”. In:19th Australian Con- ference on Software Engineering (aswec 2008). 2008, pp. 644–649.doi:10.1109/ASWEC.2008. 4483257

    work page doi:10.1109/aswec.2008 2008

  2. [2]

    Can Large Language Model Detect Pla- giarism in Source Code?

    William Brach, Kristi´ an Koˇ st’´ al, and Michal Ries. “Can Large Language Model Detect Pla- giarism in Source Code?” In:2024 2nd International Conference on Foundation and Large Language Models (FLLM). 2024, pp. 370–377.doi:10.1109/FLLM63129.2024.10852497

    doi:10.1109/fllm63129.2024.10852497 2024

  3. [3]

    Yihong Dong et al.A Survey on Code Generation with LLM-based Agents. 2025. arXiv:2508. 00083 [cs.SE].url:https://arxiv.org/abs/2508.00083. 9

    Pith/arXiv arXiv 2025

  4. [4]

    Comparison of Similarity Functions for n-gram Software Birthmarks

    Nikolay Fedorov et al. “Comparison of Similarity Functions for n-gram Software Birthmarks”. In:Proceedings of the 2024 The 6th World Symposium on Software Engineering (WSSE). WSSE ’24. Association for Computing Machinery, 2024, pp. 169–176.isbn: 9798400717086. doi:10.1145/3698062.3698087.url:https://doi.org/10.1145/3698062.3698087

    doi:10.1145/3698062.3698087.url:https://doi.org/10.1145/3698062.3698087 2024

  5. [5]

    Nikolay Fedorov et al.Project-wise Comparison of Software Birthmarks Using Weighted Partial Similarity. 2026. arXiv:2606.25418 [cs.SE].url:https://arxiv.org/abs/2606.25418

    Pith/arXiv arXiv 2026

  6. [6]

    A Study of Potential Code Borrowing and License Violations in Java Projects on GitHub

    Yaroslav Golubev et al. “A Study of Potential Code Borrowing and License Violations in Java Projects on GitHub”. In:Proceedings of the 17th International Conference on Mining Software Repositories. MSR ’20. Seoul, Republic of Korea: Association for Computing Machinery, 2020, pp. 54–64.isbn: 9781450375177.doi:10.1145/3379597.3387455.url:https://doi.org/ 10...

    doi:10.1145/3379597.3387455.url:https://doi.org/ 2020

  7. [7]

    Daya Guo et al.DeepSeek-Coder: When the Large Language Model Meets Programming – The Rise of Code Intelligence. 2024. arXiv:2401.14196 [cs.SE].url:https://arxiv.org/abs/ 2401.14196

    Pith/arXiv arXiv 2024

  8. [8]

    Minghua He et al.ExeCoder: Empowering Large Language Models with Executability Repre- sentation for Code Translation. 2025. arXiv:2501.18460 [cs.SE].url:https://arxiv.org/ abs/2501.18460

    arXiv 2025

  9. [9]

    Educating Computer Programming Students about Plagiarism through Use of a Code Similarity Detection Tool

    Tri Le et al. “Educating Computer Programming Students about Plagiarism through Use of a Code Similarity Detection Tool”. In:2013 Learning and Teaching in Computing and Engineering. 2013, pp. 98–105.doi:10.1109/LaTiCE.2013.37

    doi:10.1109/latice.2013.37 2013

  10. [10]

    API-Based Software Birthmarking Method Using Fuzzy Hashing

    Donghoon Lee et al. “API-Based Software Birthmarking Method Using Fuzzy Hashing”. In: IEICE Trans. Inf. Syst.99-D (2016), pp. 1836–1851.url:https://api.semanticscholar. org/CorpusID:35679862

    2016

  11. [11]

    A Software Birthmark Based on Dynamic Opcode n-gram

    Bin Lu et al. “A Software Birthmark Based on Dynamic Opcode n-gram”. In:International Conference on Semantic Computing (ICSC 2007). 2007, pp. 37–44.doi:10.1109/ICSC.2007. 15

    work page doi:10.1109/icsc.2007 2007

  12. [12]

    K-gram based software birthmarks

    Ginger Myles and Christian Collberg. “K-gram based software birthmarks”. In:Proceedings of the 2005 ACM Symposium on Applied Computing. SAC ’05. Santa Fe, New Mexico: Association for Computing Machinery, 2005, pp. 314–318.isbn: 1581139640.doi:10 . 1145 / 1066677 . 1066753.url:https://doi.org/10.1145/1066677.1066753

    work page doi:10.1145/1066677.1066753 2005

  13. [13]

    Software Birthmark Design and Estimation: A Systematic Literature Review

    Shah Nazir, Sara Shahzad, and Neelam Mukhtar. “Software Birthmark Design and Estimation: A Systematic Literature Review”. In:Arabian Journal for Science and Engineering44 (Jan. 2019), pp. 3905–3927.doi:10.1007/s13369-019-03718-9

    doi:10.1007/s13369-019-03718-9 2019

  14. [14]

    Detecting code paraphrased by large language models using coding style features

    Shinwoo Park et al. “Detecting code paraphrased by large language models using coding style features”. In:Engineering Applications of Artificial Intelligence162 (2025), p. 112454.issn: 0952-1976.doi:https : / / doi . org / 10 . 1016 / j . engappai . 2025 . 112454.url:https : //www.sciencedirect.com/science/article/pii/S0952197625024856

    2025

  15. [15]

    Stephen Romansky et al.Sourcerer’s Apprentice and the study of code snippet migration. 2018. arXiv:1808.00106 [cs.SE].url:https://arxiv.org/abs/1808.00106

    Pith/arXiv arXiv 2018

  16. [16]

    Design and evaluation of birth- marks for detecting theft of java programs

    Haruaki Tamada, Masahide Nakamura, and Akito Monden. “Design and evaluation of birth- marks for detecting theft of java programs”. In:IASTED Conf. on Software Engineering. 2004. url:https://api.semanticscholar.org/CorpusID:2113555

    2004

  17. [17]

    Design and Evaluation of Dynamic Software Birthmarks Based on API Calls

    Haruaki Tamada et al. “Design and Evaluation of Dynamic Software Birthmarks Based on API Calls”. In:Information Science Technical Report, NAIST-IS-TR2007011, ISSN 0919- 9527, Graduate School of Information Science, Nara Institute of Science and Technology(May 2007)

    2007

  18. [18]

    Dynamic software birthmarks to detect the theft of windows applica- tions

    Haruaki Tamada et al. “Dynamic software birthmarks to detect the theft of windows applica- tions”. In:Proceedings of the 2004 International Symposium on Future Software Technology. Vol. 20. 22. Oct. 2004

    2004

  19. [19]

    Java Birthmarks - Detecting the Software Theft -

    Haruaki Tamada et al. “Java Birthmarks - Detecting the Software Theft - .” In:IEICE Trans- actions on Information and Systems88-D (Sept. 2005), pp. 2148–2158.doi:10.1093/ietisy/ e88-d.9.2148. 10 A Used Prompt The prompt used for generating paraphrased code. 11

    work page doi:10.1093/ietisy/ 2005