Pith. sign in

REVIEW 3 major objections 3 minor 27 references

Optimizing Token Choice for Code Watermarking: An RL Approach

T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that an RL-trained policy for biasing token choices can embed detectable watermarks in generated code without degrading what the code does.

desk verdict CodeTracer is a plausible RL-based approach to code watermarking, but with only a readable abstract in front of me, the key reward/detector separation and the empirical claims are unverified. read the letter →

arxiv 2508.11925 v3 pith:RPSSJYZT submitted 2025-08-16 cs.CR cs.CLcs.LG

Zhimeng Guo , Huaisheng Zhu , Siyuan Xu , Hangfan Zhang , Teng Xiao , Minhao Cheng This is my paper
classification cs.CRcs.CLcs.LG
keywords codewatermarkingreinforcementlearningnext-tokenpredictionLLM-generatedGumbelTop-kexecutionfeedbackintellectualpropertyprotectionstructuredgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that code watermarking can be improved by treating it as a learning problem rather than a fixed decoding rule. It proposes CodeTracer, a framework in which a parameterized policy learns where and how strongly to bias token choices inside an LLM, guided by a reinforcement-learning reward that combines whether the code executes correctly with a statistical watermark signal. The payoff, if true, is that LLM-generated code can carry an ownership mark that a detector can spot from token statistics without breaking compilation, runtime behavior, or tests. The paper reports that CodeTracer beats existing watermarking methods on both detectability and preservation of code functionality.

What carries the argument

The load-bearing machinery is the trained token-selection policy and its composite reward. At each next-token step, the policy reweights the LLM's candidate tokens; the reward sums execution feedback, which asks whether the generated code runs and passes behavior checks, with a watermark-embedding signal, which measures how strongly the resulting token distribution deviates in the intended statistical direction. Gumbel Top-k reparameterization makes the discrete token decision differentiable so the whole pipeline can be trained end to end by gradient-based reinforcement learning.

What would settle it

Take CodeTracer's trained policy and generate code on a new benchmark of prompts, then run two detectors: the one used in the training reward and a second statistical detector never used in training. If watermarked code is flagged strongly by the first but only at chance level by the second, the claim that the watermark generalizes is falsified. Likewise, if watermarked code passes the training-time execution checks but fails hidden tests on fresh problems, the functionality-preservation claim is falsified.

Watch

Extended reading notes

Core claim

CodeTracer's central claim is that watermarking code can be made adaptive: rather than applying a fixed bias to token probabilities, a parameterized policy decides which tokens to favor at each generation step so the output carries a statistically detectable pattern. The policy is trained with a reward combining outcome-level execution feedback, so the code still does what it should, and a process-level watermark-embedding signal, so the token deviations are strong enough to be detected. Gumbel Top-k reparameterization lets gradients flow through the discrete token choices. The paper contends this joint optimization yields watermarks that are both more detectable and less harmful to code fun

Load-bearing premise

The central bet is that the watermark signal used in the training reward produces watermarks that are detectable by independent detectors and on new code, rather than merely matching the detector or code samples seen during training.

Editorial extensions

If this is right

  • Code generators can embed ownership marks during generation itself, without needing a separate post-hoc watermarking pass.
  • Watermark detectability can be quantified from token-distribution statistics, giving a measurable way to audit provenance of LLM output.
  • Putting execution feedback in the reward means watermarked code is held to a functionality standard during training, not only evaluated afterward.
  • The same policy-trained biasing approach could apply to any structured token sequence with a checkable outcome, not just code.
  • Using Gumbel Top-k reparameterization makes watermark strength and placement learnable per context rather than fixed by a hand-designed rule.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test the paper leaves implicit: if the watermark-embedding reward is computed with the same detector used in evaluation, the reported detectability gain may partly reflect overfitting; testing on held-out detectors would separate robust watermarking from reward hacking.
  • The same learning setup could be adapted to structured generation tasks with verifiable outcomes, such as SQL query correctness or proof validity, where functional preservation matters as much as the embedded signal.
  • The trade-off between detectability and functionality could be plotted as a rate-distortion curve, letting users choose watermark strength before training; the paper does not derive such a curve.
  • Because the watermark is a learned distributional bias, an adversary might try to clean it by re-sampling or paraphrasing tokens; robustness to such attacks is not settled by this paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes CodeTracer, a code watermarking framework that trains a parameterized reinforcement-learning policy to bias token choices during next-token prediction, aiming to embed watermarks while preserving code functionality. The training reward combines execution feedback with watermark embedding signals, and Gumbel Top-k reparameterization is used for discrete optimization. The abstract claims significant superiority over state-of-the-art baselines in both detectability and functionality preservation, and the code is publicly available. The body of the manuscript, however, is supplied only as a corrupted byte stream; apart from the abstract and a header line, no technical content, experiments, or results are readable. Consequently, the paper's central claims cannot currently be verified.

Significance. If the claims are correct, CodeTracer would offer a conceptually interesting contribution: using RL to adapt watermark embedding to code's syntactic constraints, with execution feedback as an independent grounding signal. The availability of source code is a positive step toward reproducibility. However, the unreadable full text and the unspecified relationship between the reward's watermark signal and the evaluation detector leave the central claims unsubstantiated. The potential significance is real, but the submitted manuscript does not provide enough verifiable evidence to judge it.

major comments (3)
  1. [Full text] The main body of the manuscript is an unreadable corrupted byte stream; only the abstract and the arXiv header line are legible. No equations, algorithms, experimental protocols, baselines, tables, or results are accessible. This is load-bearing for the central claim of 'significant superiority' because the supporting evidence cannot be inspected. The authors must resubmit a readable PDF or otherwise provide a machine-readable version. Until then, the technical content is unverifiable.
  2. [Abstract] The reward is described as 'seamlessly integrat[ing] execution feedback with watermark embedding signals,' but the identity of the watermark embedding signal is not specified. If this signal is the same statistic (or an approximation of the same detector) used in the reported detectability evaluation, the reported gains would be an in-sample optimization artifact. The text must state explicitly whether the training signal and evaluation detector are distinct, and ideally report cross-detector and robustness results (e.g., against semantics-preserving transformations).
  3. [Abstract] No quantitative results, dataset descriptions, baseline names, error bars, or statistical tests appear in the abstract. The phrase 'extensive comparative evaluations' cannot be assessed without the full text. At minimum, the abstract should name the datasets and metrics or the paper should include a reproducible appendix. Given the unreadable full text, this concern is currently unresolvable.
minor comments (3)
  1. [Header] The header line displays 'arXiv:2508.11923v1 [cs.LG]' while the paper is arXiv:2508.11925 (cs.CR). This appears to be a typo or a copy-paste error; please correct.
  2. [Abstract] The terms 'process-level' and 'outcome-level' rewards are introduced without explanation. A sentence or two defining these would help readers.
  3. [Abstract] The abstract would benefit from a precise statement of the threat model and the watermark detection setup (e.g., known/unknown key, white-box/black-box).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity can be exhibited from the available text; the reward/detector relationship is unspecified rather than shown to be identical.

full rationale

The only usable text is the abstract, and the full manuscript is an unreadable corrupted byte stream. The abstract says the reward system 'seamlessly integrates execution feedback with watermark embedding signals,' but it does not define the embedding signal, the evaluation detector, or any equation linking the two. Under the hard rule that circularity may only be claimed when the paper itself can be quoted to exhibit a reduction or fitted-parameter-as-prediction, there is no quotable step here that demonstrates the training reward is the same statistic as the evaluation detector. The skeptical concern—that the policy may be optimizing the exact detector it is later scored on—is a verifiability worry, not an exhibited circular step. Likewise, no self-citation chain, imported uniqueness theorem, or ansatz-smuggling citation appears in the available text. The abstract's comparison against external baselines and use of execution feedback as an independent outcome-level reward also give the derivation potential non-circular content. Therefore the honest finding is no significant circularity, score 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

No new physical entities are introduced. The free parameters reflect tuning choices in the reward and Gumbel reparameterization that the central claim depends on but that are not specified in the abstract.

free parameters (4)
  • WM reward weight
    Abstract says reward integrates watermark embedding signals; the relative weight is not stated and would need tuning.
  • Execution reward weight
    The balance between execution feedback and watermark signals is a tuning choice not specified in the abstract.
  • Gumbel Top-k k
    The number of candidate tokens considered in the Gumbel Top-k reparameterization affects exploration and watermark strength; not reported.
  • Detector threshold
    The decision threshold for declaring a watermark present changes detectability and false positive rates; not specified.
assumptions (3)
  • standard math Gumbel Top-k reparameterization gives valid gradient estimates for discrete token choices.
    Invoked in the abstract as the mechanism for gradient-based optimization of discrete watermarking decisions.
  • domain assumption Execution feedback is a reliable proxy for code functionality and is available at training time.
    The reward integrates execution feedback; if tests or interpreters are unavailable or imperfect, functionality preservation may not generalize.
  • domain assumption Watermark detectability is well captured by statistical deviation from the base token distribution.
    The abstract describes watermarks as 'subtle yet statistically detectable deviations'; not all detectors or attacks are covered.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimizing Token Choice for Code Watermarking: An RL Approach." pith.science (2026). https://pith.science/paper/RPSSJYZT

@misc{pith2026250811925,
  author       = {Pith},
  title        = {Pith review of: Optimizing Token Choice for Code Watermarking: An RL Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RPSSJYZT}},
  note         = {Machine review of arXiv:2508.11925}
}
read the original abstract

Protecting intellectual property on LLM-generated code necessitates effective watermarking systems that can operate within code's highly structured, syntactically constrained nature. In this work, we introduce CodeTracer, an innovative adaptive code watermarking framework underpinned by a novel reinforcement learning training paradigm. At its core, CodeTracer features a policy-driven approach that utilizes a parameterized model to intelligently bias token choices during next-token prediction. This strategy ensures that embedded watermarks maintain code functionality while exhibiting subtle yet statistically detectable deviations from typical token distributions. To facilitate policy learning, we devise a comprehensive reward system that seamlessly integrates execution feedback with watermark embedding signals, balancing process-level and outcome-level rewards. Additionally, we employ Gumbel Top-k reparameterization to enable gradient-based optimization of discrete watermarking decisions. Extensive comparative evaluations demonstrate CodeTracer's significant superiority over state-of-the-art baselines in both watermark detectability and the preservation of generated code's functionality. Our code is available at https://github.com/TimeLovercc/CodeTracer.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 11 canonical work pages

  1. [1]

    GPT-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shyamal Anadkat. GPT-4 technical report. arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  2. [2]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, and Quoc Le. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 , 2021

    arXiv 2021

  3. [3]

    Evaluating L arge L anguage M odels trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde Pinto, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Wi...

    work page 2021

  4. [4]

    Scalable watermarking for identifying large language model outputs

    Sumanth Dathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, and Tatiana Matejovicova. Scalable watermarking for identifying large language model outputs. Nature , 634(8035):818--823, 2024

    work page 2024

  5. [5]

    GLTR : Statistical detection and visualization of generated text

    Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. GLTR : Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations , pages 111--116, Florence, Italy, July 2019. Association for Computational Linguistics

    work page 2019

  6. [6]

    Codeip : A grammar-guided multi-bit watermark for large language models of code

    Batu Guan, Yao Wan, Zhangqian Bi, Zheng Wang, Hongyu Zhang, Pan Zhou, and Lichao Sun. Codeip : A grammar-guided multi-bit watermark for large language models of code. arXiv preprint arXiv:2404.15639 , 2024

    arXiv 2024

  7. [7]

    Deepseek-r1 : Incentivizing reasoning capability in LLMs via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, and Xiao Bi. Deepseek-r1 : Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025

    arXiv 2025

  8. [8]

    A survey of static software watermarking

    James Hamilton and Sebastian Danicic. A survey of static software watermarking. In 2011 World Congress on Internet Security ( WorldCIS -2011) , pages 100--107. IEEE, IEEE, February 2011

    work page 2011

Show all 27 references
  1. [9]

    A watermark for large language models

    John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, and Tom Goldstein. A watermark for large language models. In International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 17251--17274. PMLR, 2023

    2023

  2. [10]

    Paraphrasing evades detectors of AI -generated text, but retrieval is an effective defense

    Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. Paraphrasing evades detectors of AI -generated text, but retrieval is an effective defense. Advances in Neural Information Processing Systems , 36, 2024

    2024

  3. [11]

    Robust distortion-free watermarks for language models

    Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion-free watermarks for language models. arXiv preprint arXiv:2307.15593 , 2023

    2023 arXiv

  4. [12]

    Tulu 3 : Pushing frontiers in open language model post-training

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, and Shane Lyu. Tulu 3 : Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124 , 2024

    2024 arXiv

  5. [13]

    Who wrote this code? watermarking for code generation

    Taehyun Lee, Seokhee Hong, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, and Gunhee Kim. Who wrote this code? watermarking for code generation. arXiv preprint arXiv:2305.15060 , 2023

    2023 arXiv

  6. [14]

    ACW : Enhancing traceability of AI -generated codes based on watermarking

    Boquan Li, Mengdi Zhang, Peixin Zhang, Jun Sun, Xingmei Wang, and Zirui Fu. ACW : Enhancing traceability of AI -generated codes based on watermarking. arXiv preprint arXiv:2402.07518 , 2024

    2024 arXiv

  7. [15]

    Protecting intellectual property of large language model-based code generation APIs via watermarks

    Zongjie Li, Chaozheng Wang, Shuai Wang, and Cuiyun Gao. Protecting intellectual property of large language model-based code generation APIs via watermarks. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security , pages 2336--2350, 2023

    2023

  8. [16]

    Aiwei Liu, Leyi Pan, Yijian Lu, Jingjing Li, Xuming Hu, Lijie Wen, Irwin King, and Philip S. Yu. A survey of text watermarking in the era of large language models. arXiv preprint arXiv:2312.07913 , 2024

    2024 arXiv

  9. [17]

    Xmark : Dynamic software watermarking using collatz conjecture

    Haoyu Ma, Chunfu Jia, Shijia Li, Wantong Zheng, and Dinghao Wu. Xmark : Dynamic software watermarking using collatz conjecture. IEEE Transactions on Information Forensics and Security , 14(11):2859--2874, November 2019

    2019

  10. [18]

    DetectGPT : Zero-shot machine-generated text detection using probability curvature

    Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. DetectGPT : Zero-shot machine-generated text detection using probability curvature. arXiv preprint arXiv:2301.11305 , 2023

    2023 arXiv

  11. [19]

    Are you copying my model? protecting the copyright of large language models for EaaS via backdoor watermark

    Wenjun Peng, Jingwei Yi, Fangzhao Wu, Shangxi Wu, Bin Zhu, Lingjuan Lyu, Binxing Jiao, Tong Xu, Guangzhong Sun, and Xing Xie. Are you copying my model? protecting the copyright of large language models for EaaS via backdoor watermark. arXiv preprint arXiv:2305.10036 , 2023

    2023 arXiv

  12. [20]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019

    2019

  13. [21]

    GPTZero : An open-source initiative for AI -generated text detection

    Edward Tian, Yiting Wang, and Zhengyuan Dai. GPTZero : An open-source initiative for AI -generated text detection. arXiv preprint arXiv:2303.08217 , 2023

    2023 arXiv

  14. [22]

    Reparameterizable subset sampling via continuous relaxations

    Sang Michael Xie and Stefano Ermon. Reparameterizable subset sampling via continuous relaxations. In International Joint Conference on Artificial Intelligence ( IJCAI ) , 2019

    2019

  15. [23]

    Learning to watermark LLM -generated text via reinforcement learning

    Xiaojun Xu, Yuanshun Yao, and Yang Liu. Learning to watermark LLM -generated text via reinforcement learning. arXiv preprint arXiv:2403.10553 , 2024

    2024 arXiv

  16. [24]

    Towards code watermarking with dual-channel transformations

    Borui Yang, Wei Li, Liyao Xiang, and Bo Li. Towards code watermarking with dual-channel transformations. arXiv preprint arXiv:2309.00860 , 2023

    2023 arXiv

  17. [25]

    Simplerl-zoo : Investigating and taming zero reinforcement learning for open base models in the wild

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo : Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892 , 2025

    2025 arXiv

  18. [26]

    SoK : Watermarking for AI -generated content

    Xuandong Zhao, Sam Gunn, Miranda Christ, Jaiden Fairoze, Andres Fabrega, Nicholas Carlini, Sanjam Garg, Sanghyun Hong, Milad Nasr, and Florian Tramer. SoK : Watermarking for AI -generated content. arXiv preprint arXiv:2411.18479 , 2024

    2024 arXiv

  19. [27]

    A survey of large language models for code: Evolution, benchmarking, and future trends

    Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, and Jiachi Chen. A survey of large language models for code: Evolution, benchmarking, and future trends. arXiv preprint arXiv:2311.10372 , 2023

    2023 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.