{"id":"295b25c0-18d0-4080-a40b-26dd8dfb7c5a","arxiv_id":"2507.05512","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Obfuscation reduces detection of N-gram watermarked code to random guessing, and the authors prove this is unavoidable under a distribution consistency assumption.","lead":"This paper shows that standard N-gram-based watermarks on AI-generated code can be defeated by off-the-shelf code obfuscators that rename variables and restructure code while preserving behavior. It proves a conditional impossibility result and shows that detection accuracy collapses to coin-flipping across three state-of-the-art watermarking schemes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption 5.1 as formalized does not entail the stationary-distribution step used in Theorem 5.1, and its direct validation covers only the ideal watermark.","rationale":"The reader's weakest assumption identifies the incomplete validation of distribution consistency for real schemes, which is a real and important gap. My stress-test sharpens this into a formal correctness issue: the proof of Theorem 5.1 uses a stronger property than Definition 5.1 states, because the stationary distribution π_j over an equivalence class is not covered by the quantified 'for all Q' implication. The paper's informal prose suggests the intended stronger property, but the formal definition and the proof are mismatched. This is an internal consistency concern, not merely a disagreement with the community consensus. The empirical attack remains convincing: three real schemes show AUROC near 0.5 after obfuscation across models, languages, and obfuscators, and the ideal-watermark experiment provides strong supporting evidence that even a very powerful N-gram watermark collapses. However, the theoretical impossibility claim as written is not fully established unless the assumption is restated and validated on the real schemes. The reader's CONDITIONAL verdict already captures this: the paper should be accepted with the condition that the assumption be reformulated precisely and its direct validation extended to the schemes the theorem targets. My concern does not change that verdict, so I recommend UNCHANGED. I mark agreement as partial because the reader focused on the validation gap while the formal definition/proof mismatch is an additional, more fundamental issue.","tokens_in":47763,"tokens_out":17224,"duration_ms":228187,"concrete_test":"Re-derive Theorem 5.1 from Definition 5.1 without adding the stronger assertion E[Detect | c ~ π_j] = ε_pos for the random-walk stationary distribution; if the derivation fails, restate the assumption explicitly. Separately, run the Appendix F distribution-consistency test on SWEET, WLLM, and SynthID by constructing approximate equivalent spaces around real watermarked outputs (using a normalizer/de-normalizer pair for each language) and comparing detector-score distributions inside those spaces to the global non-watermarked distribution; if either step fails, the theoretical claim needs restatement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 5.1's proof requires that for each rule-based equivalence class Φ_j, the random-walk stationary distribution π_j satisfies E[Detect | c ~ π_j] = ε_pos. Definition 5.1, however, only states that for any distribution Q over the code space whose marginal Detect distribution is D, conditioning Q on any Φ_j leaves the Detect distribution unchanged. For Q = π_j, which is supported entirely on a single Φ_j, the marginal and the conditional are the same distribution, so the definition imposes no constraint on π_j. Thus the formal assumption does not logically imply the equality used in the proof; the paper's informal discussion states a stronger 'within each equivalent space, detection is distributed as over the whole code space' property, but that stronger property is not what Definition 5.1 formalizes. Moreover, the direct experimental validation of distribution consistency (Appendix F, Algorithm 4) is performed only on the synthetic ideal watermarking scheme, not on SWEET, WLLM, or SynthID. The observed AUROC collapse on the real schemes is consistent with the assumption but does not directly verify it, so the theoretical impossibility result for real schemes rests on an under-specified premise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that N-gram-based LLM code watermarks are fundamentally fragile under code obfuscation. It formalizes obfuscation as a random walk over a graph of semantics-preserving transformations, proves under a 'distribution consistency' assumption that after the walk mixes the detector's false negative rate rises to 1 minus its false positive rate (Theorem 5.1), and reports extensive experiments on SWEET, WLLM, and SynthID across two LLMs, two languages, four benchmarks, and four obfuscators, showing that post-obfuscation AUROC is tightly clustered around 0.5. The paper also constructs an 'ideal' N-gram watermarking scheme and shows that even that scheme collapses after obfuscation, and it reports a direct statistical test of its core assumption on that ideal scheme only.","tokens_in":48034,"tokens_out":4373,"duration_ms":53898,"significance":"If the formal gap in Theorem 5.1 is repaired, this is a significant negative result for a widely used family of code watermarks: it indicates that off-the-shelf, semantics-preserving obfuscators can defeat detection by deployed N-gram-based schemes. The empirical sweep is unusually broad for this area, covering multiple watermarking schemes, models, languages, and obfuscators, and the AUROC collapse to roughly 0.5 is consistent. The paper deserves credit for attempting to validate its key assumption, for providing an anonymous artifact link, and for including an 'ideal' watermarking ablation that strengthens the empirical case. The main weakness is that the formalization of distribution consistency does not, as written, entail the step used in the impossibility proof, and the direct validation of the assumption covers only the synthetic ideal scheme, not the three real schemes.","major_comments":[{"comment":"The formal Definition 5.1 does not imply the equality E[Detect_k(x,c_i) | c_i in Phi_j] = eps_pos used in the proof of Theorem 5.1. As written, the definition quantifies over distributions Q on C and requires that if c ~ Q gives Detect ~ D, then conditioning on any equivalence class Phi also gives Detect ~ D. When Q is chosen to be pi_j, which is supported entirely on Phi_j, the marginal and the conditional distributions are identical, so the definition imposes no constraint on the Detect distribution under pi_j. The third equality in the proof therefore silently uses a stronger property: for every Phi_j, a code drawn from Phi_j under the stationary distribution has the same Detect distribution as a code drawn from the whole space. Please state this stronger per-space property explicitly as the assumption, or revise the proof so that every step follows from the stated formal definition. This is load-bearing because the impossibility conclusion depends on it.","section":"Definition 5.1 and Theorem 5.1 proof"},{"comment":"The direct experimental validation of Assumption 5.1 is carried out only on the synthetic ideal watermarking scheme from Appendix E, using UglifyJS normalization and GPT-4o de-normalization (Algorithm 4). The three deployed schemes, SWEET, WLLM, and SynthID, are not subjected to this direct distribution-consistency test. The observed AUROC collapse for these schemes is consistent with distribution consistency but is also consistent with weaker explanations, such as obfuscators destroying the specific N-gram features without inducing the full distribution-matching property. Moreover, the Anderson-Darling test checks only whether z-scores inside each approximate equivalent space are marginally N(0,1); it does not directly test the conditional-independence statement of Definition 5.1. Please either run the Algorithm 4 test directly on SWEET, WLLM, and SynthID, or explicitly limit the theorem's practical claim and present the real-scheme AUROC results as indirect empirical support rather than direct validation.","section":"Subsection 6.4 and Appendix F"},{"comment":"The experiments use off-the-shelf obfuscators as one-shot, often deterministic transformations, whereas Theorem 5.1 applies to RandomWalk(., Gamma_ERG, t) for t at least the mixing time. The text says a fixed-output obfuscator 'can be seen as an implementation of our attack algorithm with a random seed fixed,' but fixing the random seed gives a deterministic length-one path, not a t-step random walk that has converged to the stationary distribution. Thus the empirical results do not directly instantiate Theorem 5.1's attacker. Please clarify the relationship: either show that the obfuscators used approximate a sufficiently long random walk, or present the empirical result as a separate, weaker claim that real obfuscators in practice break the schemes even without the theoretical mixing guarantee.","section":"Section 6 and Theorem 5.1 applicability"}],"minor_comments":[{"comment":"The acronym 'AUCROC' appears in several places where 'AUROC' is intended; please make the notation consistent.","section":"Throughout"},{"comment":"The benchmark name is misspelled as 'HumenEval-X-JS'; it should be 'HumanEval-X-JS'.","section":"Figure 6 caption"},{"comment":"Reference [52] is the original Llama paper, not a citation for LLaMA-3.1-8B-Instruct; please cite the appropriate model release and update the citation in the experimental setup.","section":"Section 6.1 and References"},{"comment":"The sentence '98.10% of the equivalent spaces take the null hypothesis' should explicitly state the significance level at which the Anderson-Darling test accepts the null hypothesis, since the preceding sentence already defines it but the summary line is easy to misread.","section":"Subsection 6.4"},{"comment":"The rendered text in the figure (the repeated '1.00' blocks in the temperature legends) appears garbled in the provided version; please check the figure generation and ensure the axis labels and legends are legible.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"This is a timely and potentially influential paper, but the central proof contains a formal gap between Definition 5.1 and the step used in Theorem 5.1, and the assumption validation covers only the ideal watermark. Both issues are fixable within the manuscript's scope: strengthen the assumption statement, revise the proof accordingly, and either add a direct test on the real schemes or temper the claim. The empirical results are broad and convincing enough that I would expect the paper to be acceptable after a careful revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical claim is solid and worth taking seriously; the theory section has a real but fixable formal gap.\n\nThe genuinely new thing here is the direct attack: off-the-shelf obfuscators (UglifyJS, JS Obfuscator, Python-Minifier, PyMinifier) push detection AUROC to roughly 0.5 for three N-gram-based code watermarks — SWEET, WLLM, and SynthID — across two LLMs, two languages, and four benchmarks. The results are consistent, and the effort looks reproducible. The control experiment with an \"ideal\" watermarking scheme is the best part: even with global N-gram selection and access to the benchmark test suite, the ideal scheme collapses from 0.97 to 0.51 AUROC after UglifyJS. That is real evidence the fragility is structural to N-gram features, not a flaw of a specific watermark implementation. The discussion of why the prior impossibility result's quality oracle is unreasonable for code (Rice's theorem) is also sound and clearly written.\n\nThe soft spots are in the formal layer, and the stress-test note holds up. Theorem 5.1's proof needs E[Detect | c ~ π_j] = ε_pos under the stationary distribution of the walk on each equivalence class, but Definition 5.1 as formalized only states a conditional-coherence property for distributions Q that have mass both inside and outside the class. Applied to Q = π_j — which lives entirely inside one class — the definition is vacuous, so it does not imply the proof step. The intended assumption is plainly that the detector is statistically independent of the rule-based partition; restated that way, the theorem goes through as a straightforward argument. But as written, Assumption 5.1 and Theorem 5.1 do not quite line up.\n\nSecond weakness: the direct validation of distribution consistency (the 98.10% Anderson-Darling figure) is run only on the synthetic ideal scheme, not on SWEET, WLLM, or SynthID. The AUROC collapse on the real schemes is consistent with the assumption but indirect; the paper's wording slightly oversells the support. I would not call the argument circular — the ideal-scheme check is an independent test of the mechanism — but it is one step removed from the actual targets.\n\nWho gets value: anyone working on LLM watermarking, particularly code watermarking or deployed provenance tools like SynthID. The attack alone should change how robustness claims are evaluated in this area.\n\nRecommendation: yes, send to serious referees. The theory section needs revision — restate the assumption, make the stationary-distribution step explicit, and validate distribution consistency on at least one real scheme — but the empirical result is strong enough to carry the paper through revision. This is not a desk reject.","headline":"Solid, broadly convincing obfuscation attack on N-gram code watermarks; the impossibility theorem needs a formal fix, but the empirical collapse of detection is real.","tokens_in":48479,"tokens_out":6153,"would_cite":true,"duration_ms":69540,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that semantics-preserving code obfuscation reduces any N-gram watermark detector to random guessing, with the attacked false negative rate rising to $1 - \\epsilon_{\\text{pos}}$, and shows three leading schemes collapsing…","keywords":["N-gram watermarking","code obfuscation","LLM watermarking robustness","random walk attack","distribution consistency","AI-generated code detection","semantics-preserving transformation","Markov chain mixing time"],"falsifier":"Measure the false negative rate at the scheme's original operating threshold after obfuscating watermarked code: the theorem predicts it will be statistically indistinguishable from $1 - \\epsilon_{\\text{pos}}$, with AUROC near 0.5. A concrete falsifier would be an N-gram watermark whose marked tokens are attached to features that obfuscators preserve, for instance the sequence of third-party API calls, together with obfuscated samples that retain test-suite passability while keeping detection AUROC well above 0.6; that would show distribution consistency fails for that rule set and the $1 - \\epsilon_{\\text{pos}}$ bound does not apply.","tokens_in":47611,"feed_emoji":"🎭","tokens_out":10820,"duration_ms":119735,"temperature":0.7,"pith_summary":"This paper tries to establish that N-gram-based watermarking, the dominant approach for marking AI-generated code, is not robust against code obfuscation. It models obfuscation as a uniformly random walk over reversible, semantics-preserving code edits and proves that any N-gram detector with false positive rate $\\epsilon_{\\text{pos}}$ will, after enough steps, fail on $1 - \\epsilon_{\\text{pos}}$ of the obfuscated watermarked code, meaning detection becomes no better than a coin flip. The only assumption needed, distribution consistency, is that inside every bucket of codes connected by such reversible edits the detector's score distribution matches its distribution over all code. If the claim holds, current robustness claims for code watermarking are overstated, because off-the-shelf obfuscators are cheap and preserve program behavior. The experiments show post-obfuscation AUROC tightly surrounding 0.5 across two models, two languages, four benchmarks, and four obfuscators.","feed_headline":"Obfuscation turns code watermark detection into a coin flip","feed_subtitle":"Proof plus experiments: SWEET, WLLM, and SynthID all drop to AUROC near 0.5 after obfuscation.","key_machinery":"The argument rides on a Markov random walk over a code-transformation graph. An ergodicity rule set is a collection of semantics-preserving edit rules that includes an empty rule and an inverse rule for every edit, which makes the graph irreducible and aperiodic; this guarantees a stationary distribution $\\pi$ over each equivalence space that is independent of where the walk starts (Theorem 4.1) and makes the rule set partition the high-quality code space into disjoint rule-based equivalence spaces (Definition 4.1). The decisive object is the distribution consistency assumption (Assumption 5.1): conditional on which equivalence space a code lands in, the watermark detector's decision is distributed exactly as over the whole code space. Combined with the stationary distribution, this makes the expected detection probability after a long random walk equal the global false positive rate $\\epsilon_{\\text{pos}}$, so the false negative rate on obfuscated watermarked code becomes $1 - \\epsilon_{\\text{pos}}$. The mixing-time bound in Appendix C shows the walk converges in $\\mathcal{O}(l^2 \\ln \\epsilon^{-1} + l^3)$ steps for code segments of length $l$, which the authors argue is cheaper than LLM generation.","core_discovery":"On the paper's own terms, the discovery is an impossibility result with a lightweight assumption: for every N-gram-based watermarking scheme, if distribution consistency (Assumption 5.1) holds, there exists an ergodicity rule set such that the RandomWalk attacker (Algorithm 2) achieves a false negative rate of $1 - \\epsilon_{\\text{pos}}$ on watermarked code, where $\\epsilon_{\\text{pos}}$ is the scheme's original false positive rate (Theorem 5.1). With a finite number of steps beyond the Markov chain's mixing time, the attacked false negative rate is at least $1 - \\epsilon - \\epsilon_{\\text{pos}}$ (Theorem 5.2), and the mixing time itself is shown to scale as $\\mathcal{O}(l^2 \\ln \\epsilon^{-1} + l^3)$ for segments of length $l$. The authors establish the graph machinery: an ergodicity rule set, containing an empty rule and an inverse for every edit, makes the code-transformation graph irreducible and aperiodic, so a stationary distribution exists (Theorem 4.1), and the rule set partitions the high-quality code space into disjoint equivalence spaces (Theorem 4.2). Empirically, only 9 of 1,280 obfuscated data points fall outside the AUROC band (0.4, 0.6), and even an idealized N-gram watermark with global selection and test-oracle assistance drops from 0.9747 to 0.5085 AUROC after UglifyJS. The distribution consistency assumption passes an Anderson-Darling test in 98.10% of the constructed equivalent spaces.","pith_inferences":["Editorial inference: the theory needs only a reversible rule set whose partition is independent of N-gram features, so the result should transfer to any token-rewriting pipeline such as formatters, minifiers, transpilers, or auto-refactoring tools, not just tools labeled obfuscators.","Editorial inference: the quantitative bound that the false negative rate becomes $1 - \\epsilon_{\\text{pos}}$ is stronger than the reported AUROC collapse and was not directly measured; testing threshold-level false negative rates on SWEET, WLLM, and SynthID would be a sharper confirmation of Theorem 5.1 than AUROC alone.","Editorial inference: providers deploying N-gram watermarks could run a distribution-consistency check on their intended adversary rule sets before rollout; the paper's construction procedure is a template, but applying it to the real schemes rather than the synthetic ideal scheme would be the needed audit.","Editorial inference: the same obfuscation likely weakens non-watermark statistical detectors of AI-generated code as well, since those detectors also rely on low-level token or lexeme statistics rather than semantic features."],"forward_implications":["Any scheme that keys detection on intact N-grams, including SWEET, WLLM, SynthID, and the broader green-red and Gumbel watermark families, can be evaded by running an off-the-shelf obfuscator with no loss in code quality.","The attack is low-cost: the mixing-time analysis gives a random walk whose step count grows roughly quadratically in code-segment length, which the paper argues is far cheaper than regenerating code with an LLM, so evasion scales to large workloads.","Switching N-gram length from 2 to 5 does not help a defender; the ablation study shows all settings collapse to AUROC around 0.5 after obfuscation.","A robust code watermark would have to target semantic features, such as sequences of third-party API calls, that reversible rule sets do not disturb, because that is what breaks distribution consistency.","The paper explains why earlier impossibility frameworks that assume a quality oracle do not transfer to code: verifying program behavior is undecidable in general, so attacks must rely on implementable transformation rules instead."],"supporting_citations":[{"why":"Defines SynthID, the industrially deployed 5-gram tournament-sampling watermark that is one of the three schemes the attack is demonstrated on.","marker":"[10]"},{"why":"Defines WLLM, the green-red watermark that is the distortionary baseline whose detector is attacked and whose hyperparameters set the experiment grid.","marker":"[25]"},{"why":"Defines SWEET, the code-specific low-entropy green-red watermark that is the central code-watermarking target of the attack.","marker":"[32]"},{"why":"Supplies the Markov chain and mixing-time theory, including stationary distributions, irreducibility, and aperiodicity, that Theorems 4.1, 5.2, and Appendix C build on.","marker":"[33]"},{"why":"Supplies the congestion and canonical-paths tools used to bound the random walk's mixing time in Theorem C.4.","marker":"[24]"},{"why":"The prior watermark-impossibility work whose quality-oracle and perturbation-oracle assumptions the paper argues are impossible and inefficient for code, replacing them with implementable rule sets.","marker":"[61]"},{"why":"Provides GPT-4o, used as the randomized de-normalizer in Algorithm 4 to construct approximate equivalent spaces for testing distribution consistency.","marker":"[21]"},{"why":"Describes DIPPER, the natural-language paraphrasing attack whose limitations motivate extending the idea to semantics-preserving code obfuscation.","marker":"[30]"}],"fun_headline_variants":["Obfuscation breaks N-gram watermarks: detection becomes coin flip","Obfuscation reduces watermark AUROC to 0.5, theoretically and empirically","Proof plus practice: code obfuscation flips watermark detection","Obfuscation defeats N-gram watermarks: AUROC falls to chance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire impossibility result rests on one premise: inside any bucket of codes connected by reversible, semantics-preserving edits, the watermark detector's scores are distributed exactly as they are across all code, so the bucket tells you nothing about the watermark score.","fun_headline_variants_meta":{"raw":{"variants":["Obfuscation breaks N-gram watermarks: detection becomes coin flip","Obfuscation reduces watermark AUROC to 0.5, theoretically and empirically","Proof plus practice: code obfuscation flips watermark detection","Obfuscation defeats N-gram watermarks: AUROC falls to chance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000414,"raw_usage":{"total_tokens":2284,"prompt_tokens":1232,"completion_tokens":1052,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":848,"completion_tokens_details":{"reasoning_tokens":968}},"tokens_in":848,"tokens_out":1052,"duration_ms":9907,"temperature":1.0,"reasoning_tokens":968,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:24:56.919834+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the false negative rate at the scheme's original operating threshold after obfuscating watermarked code: the theorem predicts it will be statistically indistinguishable from $1 - \\epsilon_{\\text{pos}}$, with AUROC near 0.5. A concrete falsifier would be an N-gram watermark whose marked tokens are attached to features that obfuscators preserve, for instance the sequence of third-party API calls, together with obfuscated samples that retain test-suite passability while keeping detection AUROC well above 0.6; that would show distribution consistency fails for that rule set and the $1 - \\epsilon_{\\text{pos}}$ bound does not apply.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines SynthID, the industrially deployed 5-gram tournament-sampling watermark that is one of the three schemes the attack is demonstrated on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines WLLM, the green-red watermark that is the distortionary baseline whose detector is attacked and whose hyperparameters set the experiment grid."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Markov chain and mixing-time theory, including stationary distributions, irreducibility, and aperiodicity, that Theorems 4.1, 5.2, and Appendix C build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the congestion and canonical-paths tools used to bound the random walk's mixing time in Theorem C.4."},{"cited_title":"Edelman, Danilo Francati, Daniele Venturi, Giuseppe Ateniese, and Boaz Barak","cited_arxiv_id":null,"evidence_quote":"The prior watermark-impossibility work whose quality-oracle and perturbation-oracle assumptions the paper argues are impossible and inefficient for code, replacing them with implementable rule sets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes DIPPER, the natural-language paraphrasing attack whose limitations motivate extending the idea to semantics-preserving code obfuscation."}],"review_version":1}