Pith. sign in

REVIEW 3 major objections 7 minor 41 references

Grammar-guided watermarking lets LLMs mark generated code more detectably without wrecking its quality.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 13:28 UTC pith:ZE7PI7TH

load-bearing objection GDW is a clean, inference-only packaging of CFG eligibility + soft role-scaled bias that really does push the code-watermark trade-off frontier; the hand-built role lists are the main soft spot but do not sink the result. the 3 major comments →

arxiv 2607.10210 v1 pith:ZE7PI7TH submitted 2026-07-11 cs.CR cs.CL

Toward Stronger Code Watermarking: A Grammar-Driven Approach to Optimizing the Trade-off Between Quality and Detectability

Licheng Yu , Aiwei Liu , Songze Li This is my paper
classification cs.CR cs.CL
keywords code watermarkinglarge language modelsgrammar-guided generationlogits biasstructural role modulationdetectability–quality trade-offvariable renaming robustness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models now write a lot of code, so owners need a way to prove a snippet came from their model. Existing logit watermarks work poorly on code because code is low-entropy: forcing green tokens often breaks syntax or function. This paper shows that a grammar-driven scheme can fix the trade-off. At each step an incremental parser keeps only syntactically legal tokens, a static role mask then applies a stronger bias to identifiers and literals than to keywords and operators, and detection reweights the same way. Across Python, Java and Go, several model sizes and both sampling and beam search, the resulting curves sit above prior methods, and the watermark survives simple variable renaming.

Core claim

Grammar-Driven Watermark (GDW) establishes a stronger quality–detectability frontier than existing logit, entropy and code-specific watermarks: by restricting bias to CFG-admissible tokens and scaling that bias according to structural role, it raises the area under the Pass@1–F1 curve while preserving functional correctness.

What carries the argument

The three-level masking mechanism (green-list mask × syntactic-validity mask × structural-role mask) together with the role function r(v) that multiplies the base bias δ by λ>1 only for content-bearing tokens; detection uses the identical weights in a weighted z-score.

Load-bearing premise

That a fixed, hand-curated split of the vocabulary into syntax-critical versus content-bearing tokens, plus a CFG incremental parser, faithfully tells which tokens can absorb stronger bias without breaking real program executability.

What would settle it

Replace the static role lists with a random or inverted partition (or disable the parser) and re-measure AUTC and pass@1; if the claimed superiority disappears or functional correctness collapses, the central claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Model providers can watermark code assistants at inference time with no extra training and still keep competitive pass rates.
  • Detection of machine-written code becomes statistically stronger under the same quality budget, aiding provenance and copyright checks.
  • Variable-renaming attacks no longer erase the watermark advantage of the method relative to baselines.
  • The same role-aware bias schedule can be reused for other low-entropy structured languages once a CFG and role lists exist.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the role lists are language-specific, extending GDW to C++ or Rust will require fresh manual curation or an automatic role classifier.
  • The modest inference overhead of incremental parsing may become the practical bottleneck for very long generations or multi-file projects.
  • A learned, entropy-conditioned role weight could further tighten the trade-off once more training data are available.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes Grammar-Driven Watermark (GDW), an inference-time logits-based watermark for LLM code generation. To mitigate the quality–detectability trade-off under low-entropy code, GDW combines a three-level masking mechanism (greenlist, CFG/EBNF syntactic-validity, and eligibility; §3.2.1, Eq. 1) with structural role-aware modulation that applies base bias δ to syntax-critical tokens and λ·δ to content-bearing tokens (§3.2.2–3.2.3, Eqs. 2–3, Algorithm 1). Detection uses a matching role-weighted z-score (Eq. 6, Algorithm 2). Across Python/Java/Go, several code LLMs, and sampling vs beam search, GDW reports the highest AUTC (Tables 1–2, Fig. 4; Appendix A) and remains strongest after variable-renaming (Table 5). Ablations (Table 4, Fig. 5b) attribute gains to the role mask, syntactic mask, and weighted detector.

Significance. If the empirical frontier holds under fuller documentation and statistical checks, GDW is a useful advance for code provenance: it is training-free, multi-language, and explicitly couples generation and detection via token roles. Strengths include systematic multi-model/multi-decoder evaluation, a clear ablation of the three design pieces (Table 4), and an explicit quality–detectability metric (AUTC, Eq. 7) rather than a single operating point. The grammar-guided eligibility idea and role-aligned detector are concrete methodological contributions relative to uniform KGW and entropy-threshold methods (SWEET/EWD) and to prior code-specific schemes (STONE, CodeIP).

major comments (3)
  1. §3.2.2 and Eqs. (2)–(3): The structural role partition (V_syntax vs V_content) and role function r(v) are load-bearing for both hierarchical modulation and the weighted detector (Eq. 6). The manuscript only states that the pools come from a “detailed analysis” of Python/Java/Go; it supplies neither the lists, construction criteria, coverage over the tokenizer vocabulary, nor any tokenizer-alignment check. Figure 2 uses the same partition and therefore cannot independently validate it. Table 4 only ablates the role mask entirely (AUTC 0.860→0.800), not noisy, entropy-derived, or alternative partitions. Without a reproducible construction procedure and a sensitivity study, the claim that role-aware modulation is what drives the AUTC gains in Tables 1–2 is under-supported and hard to reimplement.
  2. §3.2.1 (Syntactic Validity Mask / incremental parser): The method assumes an incremental EBNF parser yields an admissible set A_t ⊆ V that correctly intersects the model’s BPE/subword vocabulary. The paper does not specify how multi-token keywords, operators, or partial identifiers are mapped into token-level admissibility, nor what happens when the parser and tokenizer disagree. Misalignment would either waste green-list signal or still allow strong bias onto syntax-critical fragments, undermining both the quality claim and the interpretation of M_elig,t in Eq. (1). A concrete description (and preferably a small diagnostic of |A_t| and parse failures) is needed for the three-level mask to be credible.
  3. Tables 1–2 and Eq. (7): AUTC is the primary evidence for a “stronger trade-off frontier,” yet no multi-seed variance, error bars, or confidence intervals are reported for pass@1, F1, or AUTC. Several reported margins are small (e.g., HumanEval sample Qwen2.5-Coder-3B: GDW 0.811 vs CodeIP 0.805). The shared integration interval [Q_start, Q_end] is defined only as “common” and is not given numerically per plot. Without variance estimates and an explicit interval protocol, it is difficult to judge whether GDW’s ranking is stable or sensitive to sampling noise and curve truncation.
minor comments (7)
  1. §4.1 / Eq. (7): State the numerical [Q_start, Q_end] used for each AUTC entry and how methods with non-overlapping quality ranges are handled.
  2. §4.3 / Table 3: λ sensitivity is shown only for HumanEvalPack-Java with beam search; a short note on whether λ=2 remains preferred under sampling and on Python would strengthen the hyperparameter claim.
  3. Limitations: The incremental-parser overhead is acknowledged but not quantified (latency or tokens/s vs KGW). Even a single-model timing table would help readers assess practicality.
  4. Figure 2 caption and §3.2.2: Clarify whether “syntax-critical” / “content-bearing” labels are assigned by static token-type lists or by runtime parse roles; the text suggests static pools, which should be stated explicitly.
  5. Related Work (§5): A short, direct comparison of computational cost and training requirements vs CodeIP (type predictor) would better position the inference-only claim.
  6. Algorithm 2 / detection: Note that the detector must share the same V_syntax / V_content partition as generation; this is fine if the lists are published with the method.
  7. Presentation: Several figure panels in the manuscript source appear as Unicode glyph dumps rather than readable axes; ensure final figures have legible labels for Pass@1 and F1.

Circularity Check

0 steps flagged

No significant circularity: empirical watermark method with intentional generation–detection alignment, evaluated on external benchmarks against external baselines.

full rationale

GDW is an inference-time systems method, not a first-principles derivation that claims to predict a quantity from inputs that already encode that quantity. Watermark injection (three-level masks, role function r(v), Eq. 3) and detection (weighted z-score, Eq. 6) deliberately share the same structural-role weights; that is standard matched-statistic design, not a self-definitional reduction of a claimed prediction. Hyperparameters (γ, δ, λ, τ) are swept or fixed by the authors and performance is reported as Pass@1–F1 trade-off curves and AUTC on HumanEval, MBPP+, and HumanEvalPack against external baselines (KGW, SWEET, EWD, STONE, CodeIP, SynthID-Text). No uniqueness theorem is imported from overlapping authors to force the design; self-citations (e.g., EWD / Liu-related entropy work) appear as baselines or related work, not as load-bearing premises that make the AUTC claim true by construction. Fig. 2 motivates the role split via measured entropy under that same hand partition—it is design motivation, not a fitted identity renamed as prediction. Concerns about the hand-built V_syntax / V_content lists and CFG–tokenizer alignment are validity/assumption risks, not circularity under the stated patterns. The paper is self-contained against external benchmarks; score 0 is appropriate.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 4 invented entities

Central claim is empirical superiority of GDW’s quality–detectability frontier. It rests on standard KGW-style hashing, the assumption that CFG incremental parsing plus a static syntax/content vocabulary split is a good proxy for safe bias placement, and free knobs (γ, δ, λ, τ) that define watermark strength and detection. No new physical entities; invented constructs are algorithmic modules and the AUTC summary metric.

free parameters (4)
  • green-list ratio γ
    Fixed to 0.5 for red-green methods; directly scales expected green hits and z-score null model.
  • base watermark bias δ
    Swept in (0, 5.0] to trace quality–detectability curves; primary strength knob for all red-green methods including GDW.
  • role scaling factor λ
    Set to 2.0 after a small sweep (Table 3, λ∈{2,3,4,5}); multiplies bias on content-bearing tokens and detection weights.
  • detection threshold τ
    z-score threshold fixed at 4.0; F1 reported at 1% FPR following KGW practice—threshold choice affects detectability points on the curve.
axioms (4)
  • domain assumption KGW-style pseudorandom green/red partition of the vocabulary seeded by the previous token is a valid carrier for a detectable statistical signal.
    Inherited from Kirchenbauer et al. (2023a) and used throughout §2.2 and §3.
  • domain assumption An EBNF/CFG incremental parser over the current prefix yields a token set whose restriction preserves syntactic validity of generated code for Python, Java, and Go under the model’s tokenizer.
    Core of the syntactic validity mask in §3.2.1; tokenizer–grammar mismatch is not proven away.
  • ad hoc to paper Tokens can be partitioned into syntax-critical vs content-bearing pools such that stronger bias on the latter and weaker bias on the former improves the quality–detectability trade-off.
    Motivated by mean-entropy measurements in Fig. 2 and formalized by M_role and r(v) in §3.2.2; partition is hand-designed per language.
  • domain assumption pass@1 and F1 at fixed FPR, summarized by AUTC over a shared quality interval, adequately represent the quality–detectability trade-off for code watermarking.
    Evaluation design in §4.1; AUTC defined in Eq. (7).
invented entities (4)
  • Three-level masking mechanism (M_wm, M_syn, M_elig) no independent evidence
    purpose: Restrict watermark bias to green-list tokens that are also CFG-admissible at the current step.
    Defined in §3.2.1; algorithmic construct, not an external physical entity.
  • Structural role-aware modulation and role function r(v) no independent evidence
    purpose: Apply δ to syntax-critical tokens and λ·δ to content-bearing tokens under the eligibility mask.
    Eq. (2)–(3); design choice of this paper.
  • Role-aware weighted z-score detector no independent evidence
    purpose: Align detection weights with generation-time role scaling to improve detectability under low entropy.
    Eq. (5)–(6), Algorithm 2.
  • AUTC (Area Under the Trade-off Curve) no independent evidence
    purpose: Scalar summary of Pass@1–F1 trade-off curves for method ranking.
    Eq. (7) in §4.1; evaluation metric invented for the paper’s comparisons.

pith-pipeline@v1.1.0-grok45 · 35386 in / 3725 out tokens · 39364 ms · 2026-07-14T13:28:20.981635+00:00 · methodology

0 comments
read the original abstract

With the rapid development of Large Language Models (LLMs), text watermarking has emerged as a crucial technique for identifying machine-generated content. However, directly applying existing logits-based watermarking methods to code generation remains challenging, since the low-entropy nature of code exacerbates the trade-off between code quality and watermark detectability. In this paper, we propose a novel code watermarking approach called Grammar-Driven Watermark (GDW) for LLMs. GDW preserves syntactic validity through a grammar-guided three-level masking mechanism and injects watermark signals via structural role-aware modulation, assigning a stronger bias to content-bearing tokens while applying a more conservative bias to syntax-critical tokens. Aligning with the generation process, we further design a role-aware weighted detection statistic to improve detectability. Experiments across multiple programming languages, models, and decoding strategies show that GDW establishes a stronger quality-detectability trade-off frontier than existing methods, while maintaining robustness against variable-renaming attacks.

Figures

Figures reproduced from arXiv: 2607.10210 by Aiwei Liu, Licheng Yu, Songze Li.

Figure 1
Figure 1. Figure 1: Comparison among existing logits-based wa [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Mean token entropy of syntax-critical tokens [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: A Grammar-Driven Watermark Approach for LLM-Generated Code [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Pass@1-F1 curves for watermarked code generation. We used two different decoding strategies: sampling [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Subfigure (a) illustrates the trade-off curves of GDW for different [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Pass@1-F1 trade-off curves on HumanEvalPack for Qwen2.5-Coder-3B and StarCoder2-3B under two [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Trade-off curves for representative baselines under sampling and beam search decoding strategies. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Impact of Decoding strategies on the quality-detectability trade-off of GDW. [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 12 linked inside Pith

  1. [1]

    2025 , note =

    Anthropic , title =. 2025 , note =

    2025

  2. [2]

    2021 , note =

    Microsoft and OpenAI , title =. 2021 , note =

    2021

  3. [3]

    2023 , note =

    Anysphere , title =. 2023 , note =

    2023

  4. [4]

    INAE Letters , volume=

    Software watermarking: Progress and challenges , author=. INAE Letters , volume=. 2019 , publisher=

    2019

  5. [5]

    International Conference on Machine Learning , pages=

    A watermark for large language models , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  6. [6]

    International Conference on Learning Representations , year=

    An Unforgeable Publicly Verifiable Watermark for Large Language Models , author=. International Conference on Learning Representations , year=

  7. [7]

    Proceedings of the 26th ACM SIGPLAN-SIGACT symposium on Principles of programming languages , pages=

    Software watermarking: Models and dynamic embeddings , author=. Proceedings of the 26th ACM SIGPLAN-SIGACT symposium on Principles of programming languages , pages=

  8. [8]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Who wrote this code? watermarking for code generation , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  9. [9]

    arXiv preprint arXiv:2403.13485 , year=

    An entropy-based text watermarking detection method , author=. arXiv preprint arXiv:2403.13485 , year=

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:2402.19173 , year=

    Starcoder 2 and the stack v2: The next generation , author=. arXiv preprint arXiv:2402.19173 , year=

    Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:2409.12186 , year=

    Qwen2.5-coder technical report , author=. arXiv preprint arXiv:2409.12186 , year=

    Pith/arXiv arXiv

  12. [12]

    ArXiv , year=

    Qwen3 Technical Report , author=. ArXiv , year=

  13. [13]

    ArXiv , year=

    Evaluating Large Language Models Trained on Code , author=. ArXiv , year=

  14. [14]

    Is Your Code Generated by Chat

    Liu, Jiawei and Xia, Chunqiu Steven and Wang, Yuyao and Zhang, Lingming , booktitle =. Is Your Code Generated by Chat. 2023 , url =

    2023

  15. [15]

    First Conference on Language Modeling , year =

    Evaluating Language Models for Efficient Code Generation , author =. First Conference on Language Modeling , year =

  16. [16]

    ArXiv , year=

    Marking Code Without Breaking It: Code Watermarking for Detecting LLM-Generated Code , author=. ArXiv , year=

  17. [17]

    arXiv preprint arXiv:2307.15992 , year=

    Towards codable watermarking for injecting multi-bits information to LLMs , author=. arXiv preprint arXiv:2307.15992 , year=

    Pith/arXiv arXiv

  18. [18]

    arXiv preprint arXiv:2311.09535 , year=

    FunctionMarker: Watermarking language datasets via knowledge injection , author=. arXiv preprint arXiv:2311.09535 , year=

    Pith/arXiv arXiv

  19. [19]

    2024 IEEE Symposium on Security and Privacy (SP) , pages=

    Promptcare: Prompt copyright protection by watermark injection and verification , author=. 2024 IEEE Symposium on Security and Privacy (SP) , pages=. 2024 , organization=

    2024

  20. [20]

    33rd USENIX Security Symposium (USENIX Security 24) , pages=

    \ REMARK-LLM \ : A robust and efficient watermarking framework for generative large language models , author=. 33rd USENIX Security Symposium (USENIX Security 24) , pages=

  21. [21]

    arXiv preprint arXiv:2506.21602 , year=

    Bimark: Unbiased multilayer watermarking for large language models , author=. arXiv preprint arXiv:2506.21602 , year=

    Pith/arXiv arXiv

  22. [22]

    34th USENIX Security Symposium (USENIX Security 25) , pages=

    Provably robust multi-bit watermarking for \ AI-generated \ text , author=. 34th USENIX Security Symposium (USENIX Security 25) , pages=

  23. [23]

    Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security , pages=

    Protecting intellectual property of large language model-based code generation apis via watermarks , author=. Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security , pages=

    2023

  24. [24]

    arXiv preprint arXiv:2310.08920 , year=

    Embarrassingly simple text watermarks , author=. arXiv preprint arXiv:2310.08920 , year=

    Pith/arXiv arXiv

  25. [25]

    arXiv preprint arXiv:2405.14604 , year=

    Watermarking Low-entropy Generation for Large Language Models: An Unbiased and Low-risk Method , author=. arXiv preprint arXiv:2405.14604 , year=

    Pith/arXiv arXiv

  26. [26]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Morphmark: Flexible adaptive watermarking for large language models , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  27. [27]

    arXiv preprint arXiv:2310.00833 , year=

    Necessary and sufficient watermark for large language models , author=. arXiv preprint arXiv:2310.00833 , year=

    Pith/arXiv arXiv

  28. [28]

    arXiv preprint arXiv:2306.04634 , year=

    On the reliability of watermarks for large language models , author=. arXiv preprint arXiv:2306.04634 , year=

    Pith/arXiv arXiv

  29. [29]

    Dipmark: A stealthy, efficient and resilient watermark for large language models , author=

  30. [30]

    Findings of the Association for Computational Linguistics: NAACL 2024 , pages=

    A robust semantics-based watermark for large language model against paraphrasing , author=. Findings of the Association for Computational Linguistics: NAACL 2024 , pages=

    2024

  31. [31]

    arXiv preprint arXiv:2402.18059 , year=

    Token-specific watermarking with enhanced detectability and semantic coherence for large language models , author=. arXiv preprint arXiv:2402.18059 , year=

    Pith/arXiv arXiv

  32. [32]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    CodeIP: A grammar-guided multi-bit watermark for large language models of code , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    2024

  33. [33]

    Journal of Computer Security , volume=

    Software watermarking in the frequency domain: implementation, analysis, and attacks , author=. Journal of Computer Security , volume=. 2005 , publisher=

    2005

  34. [34]

    2014 Eighth International Conference on Complex, Intelligent and Software Intensive Systems , pages=

    Function level control flow obfuscation for software security , author=. 2014 Eighth International Conference on Complex, Intelligent and Software Intensive Systems , pages=. 2014 , organization=

    2014

  35. [35]

    International Conference on Advanced Intelligent Systems and Informatics , pages=

    Software watermarking for java program based on method name encoding , author=. International Conference on Advanced Intelligent Systems and Informatics , pages=. 2017 , organization=

    2017

  36. [36]

    ArXiv , year=

    OctoPack: Instruction Tuning Code Large Language Models , author=. ArXiv , year=

  37. [37]

    M ark LLM : An Open-Source Toolkit for LLM Watermarking

    Pan, Leyi and Liu, Aiwei and He, Zhiwei and Gao, Zitian and Zhao, Xuandong and Lu, Yijian and Zhou, Binglin and Liu, Shuliang and Hu, Xuming and Wen, Lijie and King, Irwin and Yu, Philip S. M ark LLM : An Open-Source Toolkit for LLM Watermarking. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations....

    doi:10.18653/v1/2024.emnlp-demo.7 2024

  38. [38]

    arXiv preprint arXiv:2403.17983 , year=

    Is The Watermarking Of LLM-Generated Code Robust? , author=. arXiv preprint arXiv:2403.17983 , year=

    Pith/arXiv arXiv

  39. [39]

    , author=

    The meaning and use of the area under a receiver operating characteristic (ROC) curve. , author=. Radiology , volume=

  40. [40]

    Computational linguistics , volume=

    A statistical approach to machine translation , author=. Computational linguistics , volume=

  41. [41]

    Nature , volume=

    Scalable watermarking for identifying large language model outputs , author=. Nature , volume=. 2024 , publisher=

    2024