{"id":"62766e66-28f6-4952-bdba-04b2d375f1f5","arxiv_id":"2607.03014","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Opcode k-gram software birthmarks, especially 2-grams with Dice similarity, detect LLM code paraphrasing with high Hmean, though ChatGPT-style models are hardest to catch.","lead":"k-gram birthmarks from Java bytecode still catch most LLM-paraphrased code clones, especially with short k and set-based similarity. The result matters for open-source license enforcement when only binaries, not source, are available.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"Optimistic best-ε Hmean without holdout is the load-bearing soft spot for the 'remain effective' claim.","rationale":"The reader's weakest_assumption correctly isolates the evaluation design in §4.4 as the softest load-bearing point under the strongest claim. The paper itself acknowledges the small dataset and the choice to report best Hmean rather than a split protocol; that choice makes the headline numbers (0.946 Dice, 0.830 for 2-gram, 0.8603 for ChatGPT) optimistic upper bounds rather than unbiased estimates. Relative trends (short k better, set-based >> edit-distance, ChatGPT hardest) are still informative and the methods are standard, so the contribution remains publishable with clearer caveats and preferably a holdout or fixed-threshold re-analysis—hence CONDITIONAL is unchanged. No stronger internal inconsistency (e.g., in birthmark definition or compile filtering) displaces this concern. Artifacts and larger N would further harden the claim but are secondary to the threshold-selection issue for the absolute effectiveness language.","tokens_in":10266,"tokens_out":554,"duration_ms":4802,"concrete_test":"For the best reported configuration (2-gram + Dice), fix ε once on a random 50% of the original/paraphrase and original/unrelated pairs, then evaluate Hmean only on the held-out 50%; repeat over 10 stratified splits. If mean held-out Hmean falls below ~0.80 (or drops >0.10 from the paper's best-ε figure), the absolute 'remain effective' claim is overstated for deployment.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that k-gram birthmarks 'remain effective' (short k, set-based sim, Dice ~0.946 avg Hmean) rests on reporting the best achievable Hmean by selecting the decision threshold ε on the full similarity set for every (k, sim, LLM) configuration (§4.4: 'we report the best achievable Hmean obtained from the calculated similarity values'; no train/test split). Resilience/credibility rates and thus Hmean are therefore maximized post hoc rather than measured under a fixed or held-out threshold. With only 352 LLC-filtered files (and further attrition from compile failures, Table 2), this optimistic selection can substantially inflate absolute performance relative to any realistic deployment protocol that must choose ε without seeing the test pairs. The relative ranking of k and of set vs. edit-distance may still hold, but the absolute claim of effectiveness (and the numerical Hmeans used to support it) is not secured by the reported protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper investigates whether k-gram software birthmarks (unique contiguous sequences of Java bytecode opcodes, k = 1..6) remain effective against LLM-assisted code plagiarism realized by code paraphrasing. Using three contemporary LLMs (ChatGPT-5.1-Codex-Mini, DeepSeek-V4-Flash, Claude-Haiku-4.5) and a filtered set of individually compilable Java source files drawn from actively maintained BSD-2-Clause GitHub projects, the authors extract birthmarks via javap/ASM, compare them with six similarity functions (cosine-count, cosine-TF-IDF, Dice, Jaccard, Simpson, Levenshtein-based), and report harmonic means of resilience and credibility rates. They conclude that short k (especially 2-gram) combined with set-based measures (Dice best) yields high Hmean, that ChatGPT produces the hardest-to-detect clones, and that coding-oriented models are stronger at the plagiarism task.","tokens_in":10506,"tokens_out":991,"duration_ms":13563,"significance":"If the absolute effectiveness numbers hold under a realistic decision protocol, the work supplies timely empirical evidence that a simple, executable-level birthmark can still flag LLM-paraphrased Java modules. The multi-LLM, multi-k, multi-similarity design and the explicit resilience/credibility framing are useful contributions for the software-protection and open-source-compliance communities. The relative ranking of k and of set-based versus sequence-based similarity is already informative even if absolute Hmeans are optimistic.","major_comments":[{"comment":"§4.4 (and the Hmean definition that follows): the central claim that birthmarks “remain effective” rests on reporting the single best Hmean obtained by sweeping the decision threshold ε over the entire balanced similarity set for every (k, similarity, LLM) configuration, with no train/test split or fixed-threshold protocol. With only 352 LLC-filtered files (further reduced by the compile failures in Table 2), this post-hoc selection can substantially inflate absolute performance relative to any deployment setting in which ε must be chosen without seeing the test pairs. Relative rankings of k and of Dice vs. edit-distance may survive, but the numerical support for “effective” (e.g., Dice avg Hmean ≈ 0.946) is not secured by the reported protocol. A held-out or cross-validated threshold procedure, or at least a sensitivity plot of Hmean versus ε, is required before the absolute claim can b","section":null},{"comment":"Abstract vs. §2.2 / Figure 2: the abstract states that five similarity measures are compared, yet the body evaluates six (two cosine variants plus Dice, Jaccard, Simpson, edit-distance). The discrepancy should be resolved and the abstract made consistent with the actual experimental design.","section":null},{"comment":"§4.3–4.5 and Table 1–2: after LLC and compilability filtering the effective sample is modest (352 originals, fewer after LLM compile failures). Credibility comparisons use “an equal number of unrelated files,” but the paper never states how those negatives are sampled (same project? different projects? size-matched?). Without that detail it is impossible to judge whether the high credibility rates are partly an artifact of easy negatives. Clarification and, if necessary, a more stringent negative-sampling regime are needed.","section":null}],"minor_comments":[{"comment":"Figure 1 and Figure 2 lack error bars, confidence intervals, or any indication of variance across files/LLMs; adding them would strengthen the visual claims.","section":null},{"comment":"§4.3: repository collection date is given as “June 14th, 2026” while the arXiv stamp is 3 Jul 2026; a brief note on the exact crawl window would avoid confusion.","section":null},{"comment":"Appendix A is announced but the actual prompt text is missing from the supplied manuscript; it should be included for reproducibility.","section":null},{"comment":"Several minor wording issues: “semantically identical program” (abstract) should be plural; “out previous work” (§4.3) → “our”; “paraphrased code that were” (§6) → “that was”.","section":null},{"comment":"The claim in the abstract that findings “confirm the higher performance of coding-oriented models for plagiarism task” is only weakly supported by the three-model comparison; a more cautious phrasing would be preferable.","section":null}],"recommendation":"major_revision","confidential_remarks":"The evaluation-protocol issue (best-ε on the full set) is the single load-bearing weakness; once addressed the paper is a solid, timely empirical contribution. Model names and dates are consistent with a mid-2026 submission, so no authenticity concern. Fit for a software-engineering / software-protection venue is good."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing: this is a clean, limited empirical check of whether Myles–Collberg-style unique opcode k-grams still catch LLM code paraphrasing on compilable Java modules. They do, especially at k=2–3 with set-based similarity (Dice best). ChatGPT-5.1-Codex-Mini is the hardest of the three models they tried. That is the actual new result—prior work had birthmarks or LLM paraphrasing separately; this combines them on a filtered BSD-2-Clause corpus.\n\nWhat they do well is straightforward. Pipeline is coherent: compile, javap, ASM opcodes, unique k-grams, five similarity families, resilience/credibility harmonic mean. They filter for individually compilable files and LLC >30, report compile-failure counts by model, and the RQ trends are internally consistent (1-gram weak, edit distance collapses, short k and set overlap win). Citations to Myles/Collberg, Tamada, Park et al., and their own prior similarity work are appropriate; no invented entities. For people who care about binary-level license enforcement or birthmark design, the ranking of k and of Dice/Simpson vs. Levenshtein is useful guidance.\n\nSoft spots are real but proportionate. The load-bearing one is evaluation design: they explicitly report the best achievable Hmean by choosing ε on the full similarity set for every (k, sim, LLM) cell, with no train/test split and N=352 after filtering (further attrition from non-compilable paraphrases). That optimistically inflates the absolute “remain effective” claim (Dice ~0.946, ChatGPT 0.86). Relative orderings of k and of set vs. sequence measures are still credible; the absolute numbers are not deployment-ready. Dataset is small and license-narrow, no public artifacts, no strong non-birthmark baselines. None of that makes the experiment incoherent—just modest.\n\nThis is for the software-plagiarism / birthmark / open-source compliance crowd, not a general SE audience. It deserves a serious referee who will push for held-out or fixed-threshold numbers and artifact release. I would cite the relative findings if I were working on executable plagiarism detectors; I would not treat the headline Hmeans as settled. Send it to review.","headline":"Solid empirical stress-test of classic opcode k-grams against three modern LLMs; the absolute “remain effective” numbers rest on best-ε Hmean with no holdout, but the relative trends look real and useful.","tokens_in":11127,"tokens_out":570,"would_cite":true,"duration_ms":5062,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"k-gram software birthmarks still catch LLM-paraphrased Java code, especially with short k and set-based similarity.","keywords":["Java","code plagiarism","code clone","code paraphrasing","plagiarism detection","software birthmark","LLM-assisted plagiarism","k-gram"],"falsifier":"Re-run the same LLM paraphrasing and birthmark pipeline with a fixed or cross-validated decision threshold on a held-out portion of the files (or a larger multi-license corpus) and check whether 2-gram Dice Hmean remains high enough to separate paraphrased pairs from unrelated pairs.","tokens_in":11160,"feed_emoji":"🔍","tokens_out":642,"duration_ms":5059,"temperature":0.7,"pith_summary":"Large language models can rewrite existing programs so they look different while keeping the same behavior, which threatens open-source licensing and ordinary plagiarism checks that rely on source text. This paper asks whether a classical executable-side fingerprint—the k-gram software birthmark built from unique short sequences of Java opcodes—can still recognize such paraphrased clones. Using three current models to rewrite compilable Java files drawn from actively maintained BSD-2-Clause projects, the authors extract birthmarks for k from 1 to 6 and compare them with several similarity measures. They find that the birthmarks remain effective: short sequences (especially 2-grams) paired with set-based measures such as the Dice index produce high harmonic means of resilience and credibility, while sequence-edit distance fails. Coding-oriented models produce harder clones, but none of the three models fully erase the birthmark signal.","feed_headline":"Opcode k-grams still catch LLM-rewritten Java code","feed_subtitle":"2-grams plus Dice similarity keep high detection scores even after model paraphrasing","key_machinery":"The k-gram software birthmark: the set of unique contiguous length-k sequences of Java bytecode opcodes extracted from a compiled class. Similarity between two such sets is scored by cosine, Dice, Jaccard, Simpson, or Levenshtein-derived measures; a threshold then decides plagiarism versus independence.","core_discovery":"k-gram software birthmarks extracted from Java opcodes remain effective against LLM-assisted code paraphrasing. Across three contemporary models, short k values—especially 2-grams—combined with set-based similarity (Dice best) yield strong detection performance measured by the harmonic mean of resilience and credibility, whereas edit-distance similarity collapses and larger k gradually loses power. ChatGPT-5.1-Codex-Mini produces the most difficult-to-detect clones among the models tested.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["k-gram opcode birthmarks catch LLM-paraphrased Java","2-gram Dice still flags LLM-rewritten Java clones","Opcode birthmarks detect ChatGPT code paraphrasing","Short k-grams plus Dice beat LLM Java plagiarism","Java opcode k-grams remain effective vs LLM clones"],"cache_read_input_tokens":128,"weakest_assumption_plain":"Performance is reported as the best harmonic mean obtainable by choosing the decision threshold on the entire similarity set with no held-out split, so the numbers may overstate what a fixed or trained threshold would achieve in deployment.","fun_headline_variants_meta":{"raw":{"variants":["k-gram opcode birthmarks catch LLM-paraphrased Java","2-gram Dice still flags LLM-rewritten Java clones","Opcode birthmarks detect ChatGPT code paraphrasing","Short k-grams plus Dice beat LLM Java plagiarism","Java opcode k-grams remain effective vs LLM clones"]},"model":"grok-4.5","effort":"low","cost_usd":0.004324,"raw_usage":{"total_tokens":1324,"prompt_tokens":809,"num_sources_used":0,"completion_tokens":63,"cost_in_usd_ticks":43240000,"prompt_tokens_details":{"text_tokens":809,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":452,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":809,"tokens_out":63,"duration_ms":4381,"temperature":1.0,"reasoning_tokens":452,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T05:26:29.054681+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the same LLM paraphrasing and birthmark pipeline with a fixed or cross-validated decision threshold on a held-out portion of the files (or a larger multi-license corpus) and check whether 2-gram Dice Hmean remains high enough to separate paraphrased pairs from unrelated pairs.","supporting_citations":[],"review_version":1}