{"id":"321a77d2-9b2f-441f-a80c-4d8f7308e8e0","arxiv_id":"2502.02068","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"RoSeMary embeds recoverable signatures in LLM-generated code and uses zero-knowledge proofs to verify ownership without revealing the signature.","lead":"This paper presents RoSeMary, a watermarking system that embeds secret signatures into code generated by large language models while preserving functionality and verifying ownership with zero-knowledge proofs. It reports high detection accuracy on code benchmarks and fast private verification, but the proof system does not bind the watermark extractor or the disputed code to the owner.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's proof is vacuously satisfiable: because Re and M are uncommitted private inputs, a prover can choose Re to map any public embedding to an arbitrary M, so the ZKP does not bind the verdict to the disputed code or to the owner's fixed watermarking system.","rationale":"The reader's REJECT verdict is well-founded. My stress-test focuses on Section 3.4, Algorithm 1. The printed circuit relation is not an ownership claim: the only public input is an embedding produced by the owner's private encoder, and the witness consists of the uncommitted decoder Re and the uncommitted signature M. In a zk-SNARK, the verifier is convinced only that there exists a witness satisfying the circuit, but here a witness can be manufactured for any input. Choosing an affine Re that maps the public embedding to an arbitrary M makes the BER condition hold trivially; Table 8 confirms Re is a Linear layer, so such a map exists. Thus any snippet, watermarked or not, admits a valid proof. The problem is not missing cryptographic details; the advertised property that a proof demonstrates generation by a proprietary LLM is logically absent. The ML-side results are plausible incremental contributions, but the central novelty is the ML/Crypto codesign for secure verification, and that part fails as stated. A fix would require committing to Re (e.g., via a public key or Merkle root) and making the raw disputed code a public input with Se inside the circuit, or otherwise cryptographically binding the public embedding to the snippet; none of this is present. Consequently, no experimental correction would rescue the central security claim, and the paper should remain rejected.","tokens_in":15356,"tokens_out":5341,"duration_ms":57086,"concrete_test":"Run an adversarial proof experiment using the authors' EZKL/Halo2 pipeline: choose an arbitrary public embedding x (e.g., the embedding of a human-written, non-watermarked snippet, or a random vector), pick a 4-bit signature M*, and construct a BF16-quantized affine decoder Re with Re(x)=M*. Feed x, theta, Re, and M* into Algorithm 1 and generate and verify the proof. If the proof verifies, the ZKP relation is vacuous and the Section 3.4 ownership claim fails. If that specific construction fails, repeat with Re fixed to the owner's decoder and M set equal to the actual decoder output on x; a verifying proof would still demonstrate vacuity through the uncommitted private signature.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.4 and Algorithm 1, the public input is only the embedding S(T,M)_embed, computed by the owner's private encoder Se, while the decoder Re and the signature M are private inputs. The circuit returns 1 iff there exists (Re, M) such that BER(Re(S(T,M)_embed), M) < theta. Because Re is an uncommitted witness and M is also private and unconstrained by any public commitment, this relation is trivially satisfiable for any public vector. For any embedding x and any target signature M*, a malicious prover can choose an affine Re with Re(x)=M* (the decoder in Table 8 is a Linear layer, so this is feasible) and set M=M*. The proof then verifies, even though x need not come from a watermarked snippet and Re need not be the owner's model. Even with a fixed Re, setting private M to the decoder output makes the BER condition true. Thus the proof does not establish that a particular disputed code was generated by the owner's LLM; it only proves an existential statement over unconstrained private values. The Section 3.4 claim of publicly verifiable proofs is therefore unsupported. This is a load-bearing flaw in the central advertised novelty, not an implementation detail.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RoSeMary, an end-to-end code watermarking framework for LLM-generated code. It trains a CodeT5-based insertion module and a shallow decoder to embed and extract short binary signatures via syntactic and variable-rename transformations, and uses Halo2-based zk-SNARKs to enable verification without revealing the signature. Experiments report high AUROC values, watermarked pass rates around 95-98%, robustness to variable renaming and refactoring, and a ZKP verification time of about 120 ms.","tokens_in":15585,"tokens_out":5723,"duration_ms":52507,"significance":"If the ZKP construction were sound, the ML/Crypto codesign would be a valuable step toward practical code watermarking. The ML component is plausible and the empirical comparison to existing watermarking methods is useful, though the evaluation is incomplete in several respects. However, the central security claim—that RoSeMary provides publicly verifiable proof that a code snippet was generated by a proprietary LLM—is not supported by the described ZKP relation. This flaw is load-bearing because it concerns the paper's main advertised novelty.","major_comments":[{"comment":"The zero-knowledge relation in Algorithm 1 is trivially satisfiable. The private inputs are the decoder Re and the signature M, with no commitment or public binding to the owner's fixed watermarking system. For any public embedding x and any chosen signature M*, a malicious prover can choose an affine decoder Re satisfying Re(x)=M*, yielding BER=0. The circuit therefore accepts even if x is not derived from a watermarked snippet and even if Re is not the owner's model. The proof does not establish that the disputed code was watermarked by the owner, nor that the embedding corresponds to the code under dispute. Consequently, the Section 3.4 claim of publicly verifiable proofs is unsupported. This is a fundamental soundness problem in the central contribution.","section":"Section 3.4 and Algorithm 1"},{"comment":"The abstract claims \"unaltered watermarked code functionality,\" but Table 1 reports watermarked pass rates of 95.12% on HumanEval, 97.64% on MBPP, and 95.39% on EvalPlus. No pass rates for the original unwatermarked code are reported, so the stated \"less than 5% pass rate drop\" cannot be verified from the paper. The reported numbers themselves imply a 4-5% drop relative to perfect functionality, which contradicts the word \"unaltered.\" The authors should report original pass rates, provide error bars, and temper the fidelity claim accordingly.","section":"Section 4.2 and Table 1"},{"comment":"The evaluation uses only 4-bit messages. Appendix A, Table 6 shows that increasing the message length to 8 bits reduces AUROC from 0.97 to 0.88 and TPR from 0.99 to 0.73 on MBPP. A 4-bit message space admits only 16 distinct signatures, which is too small for meaningful owner identification in practice. The paper's claims of high detectability and practical usability therefore hinge on an unreasonably short signature. The authors should evaluate longer message lengths or provide a clear security and usability justification for the 4-bit choice.","section":"Appendix A and Table 6"},{"comment":"The public input S(T,M)_embed is produced by the owner's private encoder Se. The paper assumes this embedding reveals nothing sensitive, but no analysis is provided; because the embedding is computed by a trained encoder, it may encode recoverable information about the signature M. Moreover, the verifier cannot check that the public embedding was actually computed from the disputed code snippet, since neither the original code nor the encoder is part of the circuit's public inputs. This breaks the link between the proof and the code being disputed. The protocol needs to bind the proof to the code snippet, for example by committing to the code or recomputing the embedding inside the circuit.","section":"Section 3.4 and Algorithm 1"}],"minor_comments":[{"comment":"The table caption lists \"DS-1000\" as a dataset, but the column headers and the text in Section 4.1 refer to EvalPlus; these labels should be reconciled.","section":"Table 1 caption"},{"comment":"The sentence beginning \"Compared to KGW (Kirchenbauer et al., 2023b) KGW watermarks...\" contains a duplicated phrase and should be rewritten.","section":"Section 4.2"},{"comment":"References Chen et al. 2021a and Chen et al. 2021b appear to describe the same arXiv report; the authors should consolidate or clearly differentiate them.","section":"References"},{"comment":"The verification time is reported as an average over 25 examples, but no standard deviation or per-example distribution is given; the same lack of error bars affects the robustness results in Figure 3.","section":"Section 4.3 and Table 2"},{"comment":"Line 4 uses both S(T,M)_embed and Se(S(T,M)) without clarifying whether the public embedding itself is the circuit input or is recomputed inside the circuit; this ambiguity should be resolved.","section":"Algorithm 1"},{"comment":"The text describes Re as composed of batch normalization, fully-connected, ReLU, and dropout layers, but Algorithm 1 calls it a \"shallow linear decoder\"; the architecture used in the ZKP should be stated precisely.","section":"Section 3.4"},{"comment":"The claim of \"first-of-its-kind ML/Crypto codesign\" is overstated given the existence of ZKROWNN, which is cited for ZK verification of DNN watermarks; the novelty should be positioned more carefully.","section":"Section 2"}],"recommendation":"reject","confidential_remarks":"The ML watermarking component may be salvageable and could form the basis of a revised submission, but the ZKP construction is unsound as described. A revision would require redesigning the circuit to commit to Re and M, binding the proof to the specific disputed code via public inputs or in-circuit recomputation, and demonstrating that the public embedding does not leak the signature. The authors should also report original pass rates, error bars, and results for longer message lengths. As submitted, the central advertised contribution is not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper's central advertised contribution—zero-knowledge verification of code watermarks—doesn't hold up. The ZKP circuit in Algorithm 1 has public input only an embedding S(T,M)_embed, and private inputs the decoder Re and signature M. The relation it proves is: there exists (Re, M) such that BER(Re(embedding), M) < theta. For any public embedding, a prover can pick Re to be a linear map that outputs M, or simply set M = Re(embedding). The proof always verifies, and it says nothing about whether the embedding came from a particular code snippet or from the owner's watermarking system. Since the encoder is private, the verifier can't even check the embedding-to-code link. That's not a fixable implementation detail; it's the core of the 'secure verification' claim.\n\nWhat's genuinely useful is the ML part. Replacing SrcMarker's shallow transformer with a CodeT5 backbone and adding noise-based adversarial training is a sensible, incremental improvement. The results—0.97 AUROC across HumanEval, MBPP, and EvalPlus, with pass rates within a few points of the originals—are credible, though not startling. The robustness numbers under variable renaming and refactoring are also reasonable. The paper is clearly written and the evaluation is mostly standard.\n\nBut there are other problems. No error bars; single runs. The 4-bit message is a performance-critical choice; at 8 bits AUROC drops to 0.88, which undercuts the 'high detectability' claim. The abstract promises 'unaltered code functionality' while Table 1 shows 2-5% pass-rate drops. And there's no code release, which makes the ML claims harder to verify. The ZK benchmarks look fine but are attached to a vacuous relation.\n\nWho is this for? Someone working on code watermarking could use the ML findings as a baseline comparison. The paper deserves a serious referee—the ZKP flaw needs a proper airing, and the ML part is worth reviewing. But as is, the central security claim is unsupported; I would not accept it without major revision to the ZKP design. My vote: reject, but with an invitation to resubmit if they fix the ZKP relation with commitments to the decoder and a public binding of the code snippet to the embedding.","headline":"The ML watermarking is a credible incremental step, but the ZKP verification claim is vacuous as written: the proof relation in Algorithm 1 is trivially satisfiable for any embedding.","tokens_in":16163,"tokens_out":3128,"would_cite":false,"duration_ms":29772,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that watermarks can be embedded in LLM-generated code without breaking it, and that ownership can be proven without revealing the watermark.","keywords":["code watermarking","LLM-generated code","zero-knowledge proof","zk-SNARK","CodeT5","detectability-fidelity-robustness","adversarial robustness","intellectual property protection"],"falsifier":"Take a code snippet that was never watermarked, compute its embedding with the owner's public encoder, and try to find any decoder and signature that satisfy the bit-error-rate threshold inside the described zk-SNARK circuit; if such a witness exists and the proof verifies, the ownership claim fails.","tokens_in":15076,"feed_emoji":"🔐","tokens_out":9424,"duration_ms":81198,"temperature":0.7,"pith_summary":"This paper argues that code watermarking can be both strong and practical if the insertion and extraction networks are trained end-to-end on a large pretrained code model, and if verification is handled by a zero-knowledge proof that never reveals the signature. It presents RoSeMary, which hides a binary message by choosing syntactic transformations and variable renames, and reports near-perfect detection (0.97 AUROC) while keeping the watermarked code's pass rate within a few points of the original on HumanEval, MBPP, and EvalPlus. The paper also shows the same watermark survives variable-rename and refactor attacks, and that a third party can verify ownership in about 120 ms using a zk-SNARK proof of about 18.75 KB. A sympathetic reader would take the paper to be establishing that the detectability-fidelity-robustness tri-objective can be satisfied by combining a pretrained code encoder with adversarial training and cryptographic verification, rather than by sampling watermarked tokens at generation time.","feed_headline":"Near-perfect code watermarking, verified in 120 ms","feed_subtitle":"An end-to-end ML/Crypto system marks LLM-generated code, preserves its behavior, and proves ownership without revealing the signature.","key_machinery":"The load-bearing object is the end-to-end trained watermark insertion/extraction pair. The insertion module uses the CodeT5 encoder to fuse code features with the watermark message, and two decoders output distributions over syntactic transformations and over variable renames; executing the highest-probability choices yields the watermarked code. A second encoder-decoder pair recovers the message, and the whole system is trained with a weighted sum of a functionality loss, a detectability loss, and a robustness loss, where adversarial noise is added to the transformation probabilities during training. For verification, the decoder weights and the signature become private witnesses in a zk-SNARK circuit that checks the bit error rate between the recovered message and the owner's signature on the public embedding.","core_discovery":"The central discovery is that swapping the shallow from-scratch transformer of prior neural code watermarking for a pretrained CodeT5 encoder, and training the insertion and extraction heads jointly, produces watermarked code that keeps its functionality while letting the hidden message be recovered even after adversarial edits. The paper further discovers that the message extraction step can be moved inside a zk-SNARK circuit, so an owner can prove that a snippet's embedding decodes to a known signature within a chosen bit-error-rate threshold without exposing the signature or the decoder. On three code benchmarks the system reaches 0.97 AUROC, keeps pass-rate drops under five percentage points, and verifies proofs in under 120 ms.","pith_inferences":["Beyond the paper, the same end-to-end adversarial-training recipe could transfer to other low-entropy structured outputs like SQL queries, configuration files, or protocol messages, where inference-time watermarking has even less room to hide a signal.","Beyond the paper, the security of the ownership proof depends on the embedding being a faithful, collision-resistant representation of the snippet; if the embedding is not recomputed inside the circuit, a proof only shows that some decoder and signature match the embedding, not that the snippet itself was watermarked.","Beyond the paper, the reported 4-bit-to-8-bit message-length ablation (AUROC drops from 0.97 to 0.88) implies a practical capacity ceiling; scaling to longer signatures would require either more transformation channels or a larger code context."],"forward_implications":["A code-LLM provider could watermark snippets after generation in a single forward pass without altering the model's decoding behavior, since insertion takes about 0.027 seconds per sample and keeps pass rates within five points of the original.","One proof, about 18.75 KB, can be verified by any third party in roughly 120 ms, so the owner never needs to reveal or re-encode the signature after each verification.","Under a 50% variable-rename attack the detection AUROC stays above 0.93, and under full refactoring by Qwen2.5-Coder-32B it stays at 0.73, showing the watermark survives common code-rewriting attacks.","The loss-weight ablation shows that increasing the robustness weight can improve attack resistance while keeping the pass rate flat, at the cost of a small drop in detection TPR when using MBPP."],"supporting_citations":[{"why":"supplies the pre-trained CodeT5 encoder used as the watermark insertion backbone and as the shared feature extractor for extraction.","marker":"Wang et al. 2021"},{"why":"defines the dual-channel syntactic/variable-rename transformation space and the functionality approximation loss that RoSeMary extends, and serves as the main neural baseline.","marker":"Yang et al. 2024"},{"why":"provides the inference-based watermarking baseline and the variable-rename and refactor attack settings used in robustness evaluation.","marker":"Lee et al. 2024"},{"why":"introduces the green/red-list inference watermarking baseline for natural language that motivates the low-entropy code problem.","marker":"Kirchenbauer et al. 2023a"},{"why":"supplies the HumanEval benchmark and the pass-rate metric used to measure code functionality fidelity.","marker":"Chen et al. 2021b"},{"why":"supplies the MBPP benchmark used for both fidelity and robustness evaluations.","marker":"Austin et al. 2021"},{"why":"provides the Halo2 zk-SNARK proof system used to implement the private watermark verification circuit.","marker":"Zcash 2024"},{"why":"provides the EZKL tooling that translates the trained decoder and BER check into a zero-knowledge circuit.","marker":"Zkonduit 2024"},{"why":"demonstrates the feasibility of zero-knowledge watermark verification for neural networks, which RoSeMary adapts to generated code.","marker":"Sheybani et al. 2023"}],"fun_headline_variants":["RoSeMary: ML/Crypto code watermarking with zk-proofs","Watermark LLM code, prove ownership in 120 ms","Code watermarking with zero-knowledge verification","End-to-end code watermarking, verified in 120 ms","ML/Crypto codesign watermarks LLM code securely"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a proof whose public input is an embedding computed by the owner, with the decoder and signature as private inputs, establishes that a specific disputed code snippet came from the owner's system; if the embedding is not derived from the snippet inside the circuit, the proof only shows that some decoder and signature match the embedding, not that the snippet itself was watermarked.","fun_headline_variants_meta":{"raw":{"variants":["RoSeMary: ML/Crypto code watermarking with zk-proofs","Watermark LLM code, prove ownership in 120 ms","Code watermarking with zero-knowledge verification","End-to-end code watermarking, verified in 120 ms","ML/Crypto codesign watermarks LLM code securely"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000264,"raw_usage":{"total_tokens":1573,"prompt_tokens":881,"completion_tokens":692,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":608}},"tokens_in":497,"tokens_out":692,"duration_ms":6640,"temperature":1.0,"reasoning_tokens":608,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T13:27:26.690842+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a code snippet that was never watermarked, compute its embedding with the owner's public encoder, and try to find any decoder and signature that satisfy the bit-error-rate threshold inside the described zk-SNARK circuit; if such a witness exists and the proof verifies, the ownership claim fails.","supporting_citations":[{"cited_title":"Srcmarker: Dual- channel source code watermarking via scalable code transformations","cited_arxiv_id":null,"evidence_quote":"defines the dual-channel syntactic/variable-rename transformation space and the functionality approximation loss that RoSeMary extends, and serves as the main neural baseline."},{"cited_title":"Zkrownn: Zero knowledge right of ownership for neural networks","cited_arxiv_id":null,"evidence_quote":"demonstrates the feasibility of zero-knowledge watermark verification for neural networks, which RoSeMary adapts to generated code."}],"review_version":1}