Pith. sign in

REVIEW 3 major objections 5 minor 72 references

Disappearing Ink: Obfuscation Breaks N-gram Code Watermarks in Theory and Practice

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that semantics-preserving code obfuscation reduces any N-gram watermark detector to random guessing, with the attacked false negative rate rising to $1 - \epsilon_{\text{pos}}$, and shows three leading schemes collapsing…

desk verdict Solid, broadly convincing obfuscation attack on N-gram code watermarks; the impossibility theorem needs a formal fix, but the empirical collapse of detection is real. read the letter →

arxiv 2507.05512 v1 pith:PMOGZTXO submitted 2025-07-07 cs.CR cs.AI

Gehao Zhang , Eugene Bagdasarian , Juan Zhai , Shiqing Ma This is my paper
classification cs.CRcs.AI
keywords N-gramwatermarkingcodeobfuscationLLMrobustnessrandomwalkattackdistributionconsistencyAI-generateddetectionsemantics-preservingtransformationMarkovchainmixingtime
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that N-gram-based watermarking, the dominant approach for marking AI-generated code, is not robust against code obfuscation. It models obfuscation as a uniformly random walk over reversible, semantics-preserving code edits and proves that any N-gram detector with false positive rate $\epsilon_{\text{pos}}$ will, after enough steps, fail on $1 - \epsilon_{\text{pos}}$ of the obfuscated watermarked code, meaning detection becomes no better than a coin flip. The only assumption needed, distribution consistency, is that inside every bucket of codes connected by such reversible edits the detector's score distribution matches its distribution over all code. If the claim holds, current robustness claims for code watermarking are overstated, because off-the-shelf obfuscators are cheap and preserve program behavior. The experiments show post-obfuscation AUROC tightly surrounding 0.5 across two models, two languages, four benchmarks, and four obfuscators.

What carries the argument

The argument rides on a Markov random walk over a code-transformation graph. An ergodicity rule set is a collection of semantics-preserving edit rules that includes an empty rule and an inverse rule for every edit, which makes the graph irreducible and aperiodic; this guarantees a stationary distribution $\pi$ over each equivalence space that is independent of where the walk starts (Theorem 4.1) and makes the rule set partition the high-quality code space into disjoint rule-based equivalence spaces (Definition 4.1). The decisive object is the distribution consistency assumption (Assumption 5.1): conditional on which equivalence space a code lands in, the watermark detector's decision is distributed exactly as over the whole code space. Combined with the stationary distribution, this makes the expected detection probability after a long random walk equal the global false positive rate $\epsilon_{\text{pos}}$, so the false negative rate on obfuscated watermarked code becomes $1 - \epsilon_{\text{pos}}$. The mixing-time bound in Appendix C shows the walk converges in $\mathcal{O}(l^2 \ln \epsilon^{-1} + l^3)$ steps for code segments of length $l$, which the authors argue is cheaper than LLM generation.

What would settle it

Measure the false negative rate at the scheme's original operating threshold after obfuscating watermarked code: the theorem predicts it will be statistically indistinguishable from $1 - \epsilon_{\text{pos}}$, with AUROC near 0.5. A concrete falsifier would be an N-gram watermark whose marked tokens are attached to features that obfuscators preserve, for instance the sequence of third-party API calls, together with obfuscated samples that retain test-suite passability while keeping detection AUROC well above 0.6; that would show distribution consistency fails for that rule set and the $1 - \epsilon_{\text{pos}}$ bound does not apply.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is an impossibility result with a lightweight assumption: for every N-gram-based watermarking scheme, if distribution consistency (Assumption 5.1) holds, there exists an ergodicity rule set such that the RandomWalk attacker (Algorithm 2) achieves a false negative rate of $1 - \epsilon_{\text{pos}}$ on watermarked code, where $\epsilon_{\text{pos}}$ is the scheme's original false positive rate (Theorem 5.1). With a finite number of steps beyond the Markov chain's mixing time, the attacked false negative rate is at least $1 - \epsilon - \epsilon_{\text{pos}}$ (Theorem 5.2), and the mixing time itself is shown to scale as $\mathcal{O}(l^2 \ln \epsilon^{-1} + l^3)$ for segments of length $l$. The authors establish the graph machinery: an ergodicity rule set, containing an empty rule and an inverse for every edit, makes the code-transformation graph irreducible and aperiodic, so a stationary distribution exists (Theorem 4.1), and the rule set partitions the high-quality code space into disjoint equivalence spaces (Theorem 4.2). Empirically, only 9 of 1,280 obfuscated data points fall outside the AUROC band (0.4, 0.6), and even an idealized N-gram watermark with global selection and test-oracle assistance drops from 0.9747 to 0.5085 AUROC after UglifyJS. The distribution consistency assumption passes an Anderson-Darling test in 98.10% of the constructed equivalent spaces.

Load-bearing premise

The entire impossibility result rests on one premise: inside any bucket of codes connected by reversible, semantics-preserving edits, the watermark detector's scores are distributed exactly as they are across all code, so the bucket tells you nothing about the watermark score.

Editorial extensions

If this is right

  • Any scheme that keys detection on intact N-grams, including SWEET, WLLM, SynthID, and the broader green-red and Gumbel watermark families, can be evaded by running an off-the-shelf obfuscator with no loss in code quality.
  • The attack is low-cost: the mixing-time analysis gives a random walk whose step count grows roughly quadratically in code-segment length, which the paper argues is far cheaper than regenerating code with an LLM, so evasion scales to large workloads.
  • Switching N-gram length from 2 to 5 does not help a defender; the ablation study shows all settings collapse to AUROC around 0.5 after obfuscation.
  • A robust code watermark would have to target semantic features, such as sequences of third-party API calls, that reversible rule sets do not disturb, because that is what breaks distribution consistency.
  • The paper explains why earlier impossibility frameworks that assume a quality oracle do not transfer to code: verifying program behavior is undecidable in general, so attacks must rely on implementable transformation rules instead.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the theory needs only a reversible rule set whose partition is independent of N-gram features, so the result should transfer to any token-rewriting pipeline such as formatters, minifiers, transpilers, or auto-refactoring tools, not just tools labeled obfuscators.
  • Editorial inference: the quantitative bound that the false negative rate becomes $1 - \epsilon_{\text{pos}}$ is stronger than the reported AUROC collapse and was not directly measured; testing threshold-level false negative rates on SWEET, WLLM, and SynthID would be a sharper confirmation of Theorem 5.1 than AUROC alone.
  • Editorial inference: providers deploying N-gram watermarks could run a distribution-consistency check on their intended adversary rule sets before rollout; the paper's construction procedure is a template, but applying it to the real schemes rather than the synthetic ideal scheme would be the needed audit.
  • Editorial inference: the same obfuscation likely weakens non-watermark statistical detectors of AI-generated code as well, since those detectors also rely on low-level token or lexeme statistics rather than semantic features.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that N-gram-based LLM code watermarks are fundamentally fragile under code obfuscation. It formalizes obfuscation as a random walk over a graph of semantics-preserving transformations, proves under a 'distribution consistency' assumption that after the walk mixes the detector's false negative rate rises to 1 minus its false positive rate (Theorem 5.1), and reports extensive experiments on SWEET, WLLM, and SynthID across two LLMs, two languages, four benchmarks, and four obfuscators, showing that post-obfuscation AUROC is tightly clustered around 0.5. The paper also constructs an 'ideal' N-gram watermarking scheme and shows that even that scheme collapses after obfuscation, and it reports a direct statistical test of its core assumption on that ideal scheme only.

Significance. If the formal gap in Theorem 5.1 is repaired, this is a significant negative result for a widely used family of code watermarks: it indicates that off-the-shelf, semantics-preserving obfuscators can defeat detection by deployed N-gram-based schemes. The empirical sweep is unusually broad for this area, covering multiple watermarking schemes, models, languages, and obfuscators, and the AUROC collapse to roughly 0.5 is consistent. The paper deserves credit for attempting to validate its key assumption, for providing an anonymous artifact link, and for including an 'ideal' watermarking ablation that strengthens the empirical case. The main weakness is that the formalization of distribution consistency does not, as written, entail the step used in the impossibility proof, and the direct validation of the assumption covers only the synthetic ideal scheme, not the three real schemes.

major comments (3)
  1. [Definition 5.1 and Theorem 5.1 proof] The formal Definition 5.1 does not imply the equality E[Detect_k(x,c_i) | c_i in Phi_j] = eps_pos used in the proof of Theorem 5.1. As written, the definition quantifies over distributions Q on C and requires that if c ~ Q gives Detect ~ D, then conditioning on any equivalence class Phi also gives Detect ~ D. When Q is chosen to be pi_j, which is supported entirely on Phi_j, the marginal and the conditional distributions are identical, so the definition imposes no constraint on the Detect distribution under pi_j. The third equality in the proof therefore silently uses a stronger property: for every Phi_j, a code drawn from Phi_j under the stationary distribution has the same Detect distribution as a code drawn from the whole space. Please state this stronger per-space property explicitly as the assumption, or revise the proof so that every step follows from the stated formal definition. This is load-bearing because the impossibility conclusion depends on it.
  2. [Subsection 6.4 and Appendix F] The direct experimental validation of Assumption 5.1 is carried out only on the synthetic ideal watermarking scheme from Appendix E, using UglifyJS normalization and GPT-4o de-normalization (Algorithm 4). The three deployed schemes, SWEET, WLLM, and SynthID, are not subjected to this direct distribution-consistency test. The observed AUROC collapse for these schemes is consistent with distribution consistency but is also consistent with weaker explanations, such as obfuscators destroying the specific N-gram features without inducing the full distribution-matching property. Moreover, the Anderson-Darling test checks only whether z-scores inside each approximate equivalent space are marginally N(0,1); it does not directly test the conditional-independence statement of Definition 5.1. Please either run the Algorithm 4 test directly on SWEET, WLLM, and SynthID, or explicitly limit the theorem's practical claim and present the real-scheme AUROC results as indirect empirical support rather than direct validation.
  3. [Section 6 and Theorem 5.1 applicability] The experiments use off-the-shelf obfuscators as one-shot, often deterministic transformations, whereas Theorem 5.1 applies to RandomWalk(., Gamma_ERG, t) for t at least the mixing time. The text says a fixed-output obfuscator 'can be seen as an implementation of our attack algorithm with a random seed fixed,' but fixing the random seed gives a deterministic length-one path, not a t-step random walk that has converged to the stationary distribution. Thus the empirical results do not directly instantiate Theorem 5.1's attacker. Please clarify the relationship: either show that the obfuscators used approximate a sufficiently long random walk, or present the empirical result as a separate, weaker claim that real obfuscators in practice break the schemes even without the theoretical mixing guarantee.
minor comments (5)
  1. [Throughout] The acronym 'AUCROC' appears in several places where 'AUROC' is intended; please make the notation consistent.
  2. [Figure 6 caption] The benchmark name is misspelled as 'HumenEval-X-JS'; it should be 'HumanEval-X-JS'.
  3. [Section 6.1 and References] Reference [52] is the original Llama paper, not a citation for LLaMA-3.1-8B-Instruct; please cite the appropriate model release and update the citation in the experimental setup.
  4. [Subsection 6.4] The sentence '98.10% of the equivalent spaces take the null hypothesis' should explicitly state the significance level at which the Anderson-Darling test accepts the null hypothesis, since the preceding sentence already defines it but the summary line is easy to misread.
  5. [Figure 5] The rendered text in the figure (the repeated '1.00' blocks in the temperature legends) appears garbled in the provided version; please check the figure generation and ensure the axis labels and legends are legible.

Circularity Check

1 steps flagged · score 6.0 of 10

Impossibility theorem is distribution consistency restated; the proof substitutes the assumption's conclusion into itself, though the empirical obfuscation results are independent.

  1. self definitional [Section 5.1, Definition 5.1, Assumption 5.1, and Theorem 5.1 proof]
    "That is, the observed distribution of Detectk(,)’s results within each equivalent space in Φ∈PΓERG remains the same as the distribution observed over the entire code space C. ... = 1 − Σ_{Φ_j∈PΓERG} (r_j · Σ_{c_i∈Φ_j} (π_i · E[Detectk(x,c_i)])) = 1 − Σ_{Φ_j∈PΓERG} (r_j · Σ_{c_i∈Φ_j} (π_i · ε_pos)) = 1 − ε_pos"

    Definition 5.1 says the detector's output distribution inside each equivalence class equals the whole-space distribution D=Bernoulli(ε_pos). Theorem 5.1's proof then substitutes E[Detect(c_i)]=ε_pos for every c_i and concludes the attacked FNR is 1−ε_pos; that substitution is exactly the assumption re-instantiated at the stationary distribution, so the 'impossibility' is contained in the premise. No watermark-specific or obfuscation-specific mechanism produces the ε_pos rate. As written, Definition 5.1 only constrains distributions Q whose marginal Detect law is D, so for Q=π_j supported on one Φ the premise and conclusion are the same object and the proof must silently import the stronger intended independence.

full rationale

The central theoretical claim is circular in the self-definitional sense: Assumption 5.1 defines distribution consistency as the detector output distribution inside every equivalence class being identical to the whole-space distribution, and the proof concludes that mixing to stationarity inside an equivalence class leaves the detector at the whole-space rate ε_pos, i.e., the assumption instantiated. The proof line Σ π_i E[Detect(c_i)] = Σ π_i ε_pos is the premise, not a derived fact; the formal definition as written does not even constrain the stationary distribution π_j, so the proof silently uses a stronger independence statement that is equivalent to the theorem's conclusion. This partial circularity warrants a 6. The empirical contribution remains independent: AUROC collapse on three external watermarking schemes with four external obfuscators is not fitted, and Appendix F directly tests the assumption on the ideal watermark via UglifyJS normalization plus GPT-4o de-normalization, with 98.10% of equivalent spaces passing the Anderson-Darling test. There is no load-bearing self-citation, and no fitted parameter is renamed as a prediction. The theorem is nevertheless a restatement of its key premise rather than an independent impossibility result.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted free parameters enter the central derivation; experimental hyperparameters are traversed rather than fit. The key axioms are standard Markov chain theory, implementability of reversible syntax rules, and, decisively, Assumption 5.1, which is close in content to the conclusion and only indirectly validated for the actual attacked schemes.

assumptions (4)
  • standard math Markov chain convergence and mixing-time theorems (irreducibility and aperiodicity imply a unique stationary distribution; congestion bounds mixing time).
    Invoked in Theorem 4.1 and Appendix C to assert stationarity and estimate t_c(epsilon).
  • domain assumption There exists an implementable ergodicity rule set: every rule has an inverse and an empty rule exists, and all transformations preserve code semantics (Table 1).
    Required for Theorem 4.1 and the partition result. The example rules (rename variables, add or delete comments, add or delete dead code) are plausible but not fully verified to preserve semantics in all contexts.
  • ad hoc to paper Distribution consistency (Assumption 5.1): for N-gram watermarking, Detect output is independent of the rule-based equivalent-space partition.
    This is the key premise of Theorem 5.1. It is argued from the low-level nature of N-gram features and tested on an ideal synthetic watermark in Appendix F, but not directly on the deployed schemes.
  • domain assumption The four off-the-shelf obfuscators preserve the functional behavior of generated code (Section 6.1).
    The attack claims quality preservation. The paper reports Python-Minifier, JS Obfuscator, and UglifyJS maintain passability, while PyMinifier has a 0.21 percent failure rate, a partial violation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Disappearing Ink: Obfuscation Breaks N-gram Code Watermarks in Theory and Practice." pith.science (2026). https://pith.science/paper/PMOGZTXO

@misc{pith2026250705512,
  author       = {Pith},
  title        = {Pith review of: Disappearing Ink: Obfuscation Breaks N-gram Code Watermarks in Theory and Practice},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PMOGZTXO}},
  note         = {Machine review of arXiv:2507.05512}
}
read the original abstract

Distinguishing AI-generated code from human-written code is becoming crucial for tasks such as authorship attribution, content tracking, and misuse detection. Based on this, N-gram-based watermarking schemes have emerged as prominent, which inject secret watermarks to be detected during the generation. However, their robustness in code content remains insufficiently evaluated. Most claims rely solely on defenses against simple code transformations or code optimizations as a simulation of attack, creating a questionable sense of robustness. In contrast, more sophisticated schemes already exist in the software engineering world, e.g., code obfuscation, which significantly alters code while preserving functionality. Although obfuscation is commonly used to protect intellectual property or evade software scanners, the robustness of code watermarking techniques against such transformations remains largely unexplored. In this work, we formally model the code obfuscation and prove the impossibility of N-gram-based watermarking's robustness with only one intuitive and experimentally verified assumption, distribution consistency, satisfied. Given the original false positive rate of the watermarking detection, the ratio that the detector failed on the watermarked code after obfuscation will increase to 1 - fpr. The experiments have been performed on three SOTA watermarking schemes, two LLMs, two programming languages, four code benchmarks, and four obfuscators. Among them, all watermarking detectors show coin-flipping detection abilities on obfuscated codes (AUROC tightly surrounds 0.5). Among all models, watermarking schemes, and datasets, both programming languages own obfuscators that can achieve attack effects with no detection AUROC higher than 0.6 after the attack. Based on the theoretical and practical observations, we also proposed a potential path of robust code watermarking.

Figures

Figures reproduced from arXiv: 2507.05512 by the authors.

Figure 1
Figure 1. The performance of Green-Red watermarks [ [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Left: An outline of our attack schema. We consider the set of all high-quality outputs (High-quality Space). Our attack randomly modifies codes while preserving semantics by a set of equivalent transformation rules. (i.e., perform code obfuscation). We prove that starting from a high-quality watermarked code sample, the obfuscated code can be independently distributed within a subset of the high-quality space (Theor… view at source ↗
Figure 3
Figure 3. Comparison between Natural Language and Pro [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: A full version of SWEET and WLLM results among two [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 4
Figure 4. Figure 4: Watermarking performances of WLLM and SWEET [ [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Left: LLaMA 3.1 Pass@1 changes with temperature. Comparison between SynthID [10] watermarked and non￾watermarked code generation. Right: SynthID detection AUROC changes with temperature. Comparison between original SynthID watermarked code and obfuscated code. For resu…
Figure 6
Figure 6. Figure 6: Watermarking performance under the N-gram length from two to five, on WLLM watermarking [ [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Left: Distribution of the Anderson-Darling test statistic used to assess whether the z-scores in each approximate rule-based equivalent space follow a normal distribution (i.e., under the null hypothesis). The blue vertical line indicates the critical value at a signif…
Figure 8
Figure 8. Figure 8: Watermarking performance of WLLM and SWEET [ [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Left: DeepSeek Coder Pass@1 changes with temperature. Comparison between SynthID [10] watermarked and non-watermarked code generation. Right: SynthID detection AUROC changes with temperature. Comparison between original SynthID watermarked code and obfuscated code. For…
Figure 10
Figure 10. Figure 10: Left: Frequency distribution chart for z-scores of generated code from ideal watermarking scheme, before/after UglifyJS-based attack. Right: Frequency distribution chart for p-values. whether directly sampled from the distribution of the whole code space C or not. We …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 51 canonical work pages

  1. [1]

    Scott Aaronson. 2023. Watermarking of Large Language Models. https://simons. berkeley.edu/talks/scott-aaronson-ut-austin-openai-2023-08-17 Accessed: 2024- 12-12

    work page 2023

  2. [2]

    Sebastian Banescu and Alexander Pretschner. 2018. A tutorial on software obfuscation. Advances in Computers 108 (2018), 283–353

    work page 2018

  3. [3]

    Chandan Kumar Behera and D Lalitha Bhaskari. 2015. Different obfuscation techniques for code protection. Procedia Computer Science 70 (2015), 757–763

    work page 2015

  4. [4]

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps- Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. 2023. MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation. IEEE Trans. Softw. Eng. 49, 7 (July 2023), 3675–369...

    arXiv 2023

  5. [5]

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023. CodeT: Code Generation with Generated Tests. In The Eleventh International Conference on Learning Representations. https: //openreview.net/forum?id=ktrw68Cmu9c

    work page 2023

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

  7. [7]

    Minhao Cheng, Jinfeng Yi, Pin-Yu Chen, Huan Zhang, and Cho-Jui Hsieh. 2020. Seq2sick: Evaluating the robustness of sequence-to-sequence models with adver- sarial examples. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 3601–3608

    work page 2020

  8. [8]

    Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019. Robust Neural Ma- chine Translation with Doubly Adversarial Inputs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Anna Korho- nen, David Traum, and Lluís Màrquez (Eds.). Association for Computational Linguistics, Florence, Italy, 4324–4333. doi:10.18653/v1/P19-1425

    doi:10.18653/v1/p19-1425 2019

Show all 72 references
  1. [9]

    Jean-Pierre Corriveau, Vojislav Radonjic, and Wei Shi. 2014. Requirements ver- ification: Legal challenges in compliance testing. In 2014 IEEE International Conference on Progress in Informatics and Computing. IEEE, 451–454

    2014

  2. [10]

    Sumanth Dathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejovicova, et al. 2024. Scalable watermarking for identifying large language model outputs. Nature 634, 8035 (2024), 818–823

    2024

  3. [11]

    Ayan Dey, Sukriti Bhattacharya, and Nabendu Chaki. 2019. Software watermark- ing: Progress and challenges. INAE Letters 4 (2019), 65–75

    2019

  4. [12]

    Javid Ebrahimi, Daniel Lowd, and Dejing Dou. 2018. On Adversarial Examples for Character-Level Neural Machine Translation. In Proceedings of the 27th International Conference on Computational Linguistics, Emily M. Bender, Leon Derczynski, and Pierre Isabelle (Eds.). Associatio...

    2018

  5. [13]

    Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite gener- ation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416–419

    2011

  6. [14]

    Jiayi Fu, Xuandong Zhao, Ruihan Yang, Yuansen Zhang, Jiangjie Chen, and Yanghua Xiao. 2024. GumbelSoft: Diversified Language Model Watermarking via the GumbelMax-trick. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P...

    2024 doi

  7. [15]

    GPTZero. [n. d.]. AI Detector - the Original AI Checker for ChatGPT & More. https://gptzero.me/ Accessed: 2024-12-12

    2024

  8. [16]

    Batu Guan, Yao Wan, Zhangqian Bi, Zheng Wang, Hongyu Zhang, Pan Zhou, and Lichao Sun. 2024. CodeIP: A Grammar-Guided Multi-Bit Watermark for Large Language Models of Code. In Findings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal,...

    2024

  9. [17]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence.arXiv preprint arXiv:2401.14196 (2024)

    2024 arXiv

  10. [18]

    Venkatesan Guruswami. 2000. Rapidly mixing markov chains: A comparison of techniques. Available: cs. washington. edu/homes/venkat/pubs/papers. html (2000)

    2000

  11. [19]

    Zhengmian Hu, Lichang Chen, Xidong Wu, Yihan Wu, Hongyang Zhang, and Heng Huang. 2024. Unbiased Watermark for Large Language Models. In The Twelfth International Conference on Learning Representations. https: //openreview.net/forum?id=uWVC5FVidc

    2024

  12. [20]

    Mingjia Huo, Sai Ashish Somayajula, Youwei Liang, Ruisi Zhang, Farinaz Koushanfar, and Pengtao Xie. 2024. Token-specific watermarking with enhanced detectability and semantic coherence for large language models. In Proceedings of the 41st International Conference on Machine Le...

    2024

  13. [21]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024)

    2024 arXiv

  14. [22]

    Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez

    Md. Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024. Map- Coder: Multi-Agent Code Generation for Competitive Problem Solving. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Ma...

    2024 doi

  15. [23]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Live- CodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In The Thirteenth International Conference ...

    2025

  16. [24]

    Mark Jerrum. 2003. Counting, sampling and integrating: algorithms and complexity. Springer Science & Business Media

    2003

  17. [25]

    John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models. In International Conference on Machine Learning. PMLR, 17061–17084

    2023

  18. [26]

    John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. 2024. On the Reliability of Watermarks for Large Language Models. In The Twelfth International Conference on Learning Repres...

    2024

  19. [27]

    Stephen Cole Kleene. 1952. Introduction to metamathematics. (1952)

    1952

  20. [28]

    Evgenios Konstantinou and Stefen Wolthusen. 2008. Metamorphic virus: Analysis and detection. Royal Holloway University of London 15 (2008), 15

    2008

  21. [29]

    Kalpesh Krishna, Yapei Chang, John Wieting, and Mohit Iyyer. 2022. Rankgen: Improving text generation with large ranking models. arXiv preprint arXiv:2205.09726 (2022)

    2022 arXiv

  22. [30]

    Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. 2023. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. Advances in Neural Information Processing Systems 36 (2023), 27469–27500

    2023

  23. [31]

    Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang

  24. [32]

    Taehyun Lee, Seokhee Hong, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, and Gunhee Kim. 2024. Who Wrote this Code? Watermark- ing for Code Generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

    2024 doi

  25. [33]

    David A Levin and Yuval Peres. 2017. Markov chains and mixing times. Vol. 107. American Mathematical Soc

    2017

  26. [34]

    Boquan Li, Mengdi Zhang, Peixin Zhang, Jun Sun, and Xingmei Wang. 2024. Resilient watermarking for llm-generated codes. arXiv e-prints (2024), arXiv– 2402

    2024

  27. [35]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. Science 378, 6624 (2022), 1092–1097

    2022

  28. [36]

    Aiwei Liu, Leyi Pan, Xuming Hu, Shuang Li, Lijie Wen, Irwin King, and Philip S. Yu. 2024. An Unforgeable Publicly Verifiable Watermark for Large Language Models. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=gMLQwKDY3N

    2024

  29. [37]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in Neural Information Processing Systems 36 (2023), 21558–21572

    2023

  30. [38]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in Neural Information Processing Systems 36 (2024)

    2024

  31. [39]

    Yepeng Liu and Yuheng Bu. 2024. Adaptive text watermark for large language models. arXiv preprint arXiv:2401.13927 (2024)

    2024 arXiv

  32. [40]

    Elizabeth M Merkhofer, John Henderson, Abigail S Gertner, Michael Doyle, and Lily Wong. 2022. Practical attacks on machine translation using paraphrase. In Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Researc...

    2022

  33. [41]

    Meta. [n. d.]. Introducing Llama 3.1: Our most capable models to date. https: //ai.meta.com/blog/meta-llama-3-1/ Accessed: 2024-12-13

    2024

  34. [42]

    Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability curvature. In International Conference on Machine Learning. PMLR, Conference’17, July 2017, Washington, DC, US...

    2023

  35. [43]

    Kaiwen Ning, Jiachi Chen, Qingyuan Zhong, Tao Zhang, Yanlin Wang, Wei Li, Yu Zhang, Weizhe Zhang, and Zibin Zheng. 2024. Mcgmark: An encodable and robust online watermark for llm-generated malicious code. arXiv preprint arXiv:2408.01354 (2024)

    2024 arXiv

  36. [44]

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2025. Asleep at the keyboard? assessing the security of github copilot’s code contributions. Commun. ACM 68, 2 (2025), 96–105

    2025

  37. [45]

    Yanru Peng, Yuting Chen, and Beijun Shen. 2019. An adaptive approach to recommending obfuscation rules for Java bytecode obfuscators. In2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), Vol. 1. IEEE, 97–106

    2019

  38. [46]

    Henry Gordon Rice. 1953. Classes of recursively enumerable sets and their decision problems. Transactions of the American Mathematical society 74, 2 (1953), 358–366

    1953

  39. [47]

    Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2023. Can AI-generated text be reliably detected?arXiv preprint arXiv:2303.11156 (2023)

    2023 arXiv

  40. [48]

    Gustavo Sandoval, Hammond Pearce, Teo Nys, Ramesh Karri, Siddharth Garg, and Brendan Dolan-Gavitt. 2023. Lost at c: A user study on the security implications of large language model code assistants. In 32nd USENIX Security Symposium (USENIX Security 23). 2205–2222

    2023

  41. [49]

    Yash Shah, Jimil Shah, and Krishna Kansara. 2018. Code obfuscating a Kotlin- based App with Proguard. In2018 Second International Conference on Advances in Electronics, Computers and Communications (ICAECC). IEEE, 1–5

    2018

  42. [50]

    Michael A Stephens. 1974. EDF statistics for goodness of fit and some comparisons. Journal of the American statistical Association 69, 347 (1974), 730–737

    1974

  43. [51]

    Zhensu Sun, Xiaoning Du, Fu Song, and Li Li. 2023. Codemark: Impercepti- ble watermarking for code datasets against neural code completion models. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Enginee...

    2023

  44. [52]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  45. [53]

    Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal Adversarial Triggers for Attacking and Analyzing NLP. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on ...

    2019 doi

  46. [54]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering (2024)

    2024

  47. [55]

    Wing Wong and Mark Stamp. 2006. Hunting for metamorphic engines. Journal in Computer Virology 2 (2006), 211–229

    2006

  48. [56]

    Scott Wu. 2024. Introducing Devin, the first AI software engineer. https://www. cognition.ai/blog/introducing-devin Accessed: 2024-12-11

    2024

  49. [57]

    Yihan Wu, Zhengmian Hu, Junfeng Guo, Hongyang Zhang, and Heng Huang

  50. [58]

    Zhiyi Xue, Liangguo Li, Senyue Tian, Xiaohong Chen, Pingping Li, Liangyu Chen, Tingting Jiang, and Min Zhang. 2024. LLM4Fin: Fully Automating LLM-Powered Test Case Generation for FinTech Software Acceptance Testing. InProceedings of the 33rd ACM SIGSOFT International Symposium...

    2024

  51. [59]

    Borui Yang, Wei Li, Liyao Xiang, and Bo Li. 2024. Srcmarker: Dual-channel source code watermarking via scalable code transformations. In 2024 IEEE Symposium on Security and Privacy (SP). IEEE, 4088–4106

    2024

  52. [60]

    In Proceedings of the 41st International Conference on Machine Learning (Vienna, Austria) (ICML’24)

    A resilient and accessible distribution-preserving watermark for large lan- guage models. In Proceedings of the 41st International Conference on Machine Learning (Vienna, Austria) (ICML’24). JMLR.org, Article 2190, 28 pages

  53. [61]

    Edelman, Danilo Francati, Daniele Venturi, Giuseppe Ateniese, and Boaz Barak

    Hanlin Zhang, Benjamin L. Edelman, Danilo Francati, Daniele Venturi, Giuseppe Ateniese, and Boaz Barak. 2024. Watermarks in the sand: impossibility of strong watermarking for language models. In Proceedings of the 41st International Conference on Machine Learning (Vienna, Aust...

    2024

  54. [62]

    Xinze Zhang, Junzhe Zhang, Zhenhua Chen, and Kun He. 2021. Crafting ad- versarial examples for neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language...

    2021

  55. [63]

    Ilsun You and Kangbin Yim. 2010. Malware obfuscation techniques: A brief survey. In 2010 International conference on broadband, wireless computing, communication and applications. IEEE, 297–300

    2010

  56. [64]

    Xuandong Zhao, Sam Gunn, Miranda Christ, Jaiden Fairoze, Andres Fabrega, Nicholas Carlini, Sanjam Garg, Sanghyun Hong, Milad Nasr, Florian Tramer, et al. 2024. SoK: Watermarking for AI-Generated Content. arXiv preprint arXiv:2411.18479 (2024)

    2024 arXiv

  57. [65]

    Xuandong Zhao, Lei Li, and Yu-Xiang Wang. 2025. Permute-and-Flip: An optimally stable and watermarkable decoder for LLMs. In The Thirteenth International Conference on Learning Representations. https://openreview.net/ forum?id=YyVVicZ32M

    2025

  58. [66]

    Xuandong Zhao, Prabhanjan Vijendra Ananth, Lei Li, and Yu-Xiang Wang

  59. [67]

    In The Twelfth International Conference on Learning Representations

    Provable Robust Watermarking for AI-Generated Text. In The Twelfth International Conference on Learning Representations. https://openreview.net/ forum?id=SsmT8aO45L

  60. [68]

    Impossibility of Impossibility

    Tong Zhou, Xuandong Zhao, Xiaolin Xu, and Shaolei Ren. 2024. Bileve: Securing Text Provenance in Large Language Models Against Spoofing with Bi-level Signa- ture. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=vj...

    2024

  61. [70]

    Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating Natural Ad- versarial Examples. In International Conference on Learning Representations (ICLR)

    2018

  62. [71]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, et al. 2023. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowle...

    2023

  63. [2024]

    Transactions on Machine Learning Research (2024)

    Robust Distortion-free Watermarks for Language Models. Transactions on Machine Learning Research (2024). https://openreview.net/forum?id= FpaCL1MO2C

    2024

  64. [9258]

    doi:10.18653/v1/2024.findings-emnlp.541

    2024 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.