{"id":"cf4afafe-6d78-48d2-a08f-df0e65ce6ed8","arxiv_id":"2506.07403","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A context-aware plug-in for LLM watermarking that skips or weakens watermarks on semantically critical tokens, improving task accuracy at similar detection rates.","lead":"This paper presents CAW, a plug-in that decides where a text watermark should be embedded by estimating how important each token is using surrounding word probabilities, then skips or weakens watermarks on critical tokens. It reports that this reduces the accuracy damage caused by watermarking on math and knowledge tasks while keeping detection rates high.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The capacity evaluator's GPT-4o-derived labels may not measure the actual quality impact the method relies on, so the claimed trade-off gains could come from soft-watermark tuning rather than contextual state awareness.","rationale":"The reader's weakest assumption is that the GPT-4o-labeled ground truth identifies tokens that must be preserved and that the evaluator generalizes. This is indeed the load-bearing premise. The paper's argument requires that high predicted capacity corresponds to large accuracy loss under watermark substitution, but the reported evaluator metrics only validate prediction of GPT-4o annotations on human-written references. The additional distribution shift to model-generated chain-of-thought outputs makes the link even more fragile. If the link fails, CAW's quality gains could be explained by the additional tunable strength parameters θ and β effectively acting as a softer watermark, not by contextual generation-state awareness. The proposed counterfactual relabeling test directly targets this premise, so the conditional verdict remains appropriate: accept only after the evaluator's labels are validated against actual task-accuracy impact.","tokens_in":18309,"tokens_out":5493,"duration_ms":76468,"concrete_test":"Hold out 100 GSM8K and MMLU-sociology questions. For each base method on Llama2-13b and Vicuna-13b, generate responses under the original watermark, and for a random sample of token positions perform one counterfactual generation that forces the watermark-selected substitute at that position. Label a position capacity-critical if that substitution changes the final answer accuracy. Then compare the GPT-4o-trained evaluator's predicted capacities against these direct counterfactual labels using AUROC or rank correlation. If the evaluator has near-chance discrimination (AUROC around 0.5-0.6), the proxy-label premise fails; if it shows strong discrimination, the central mechanism is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 (Eq. 1) defines watermark capacity as the impact of embedding at a token position, but the training labels in Appendix C are GPT-4o annotations of 'quality-critical segments' in human-written reference responses. The central mechanism requires these labels to coincide with counterfactual task-accuracy damage caused by watermark-induced token substitution, and to transfer from human references to Llama2/Vicuna chain-of-thought outputs. That equivalence is asserted, not measured. Figure 3 only shows that the learned evaluator predicts GPT-4o annotations, not that those annotations predict answer flips in GSM8K/MMLU. If the labels are weakly correlated with true per-token capacity, CAW's θ/β strength schedule may simply soften the watermark, and the Table 1/Figure 6 improvements would not validate the contextual-state mechanism. The missing training-data details and absent error bars deepen this risk rather than create a separate one.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes CAW, a plug-and-play watermarking-quality-preservation framework for LLMs. CAW trains a three-layer neural 'watermark capacity evaluator' that takes the probability distributions of the current token and its immediate left and right contexts as input and predicts whether the token is quality-critical; the prediction is then used to decide whether to watermark the position and how strongly, using a linear strength mapping with threshold θ and scaling coefficient β. For sampling-based watermarks the top-K pool is narrowed; for reweighting watermarks the logit bias δ is scaled. To avoid the latency of the one-step lookahead, CAW pre-generates all candidate branches in a single tree-attention forward pass. Experiments integrate CAW with KGW, Unigram, and EXP on GSM8K and MMLU sociology for Llama2/Vicuna-7B/13B, comparing against hard/soft baselines and an entropy-based plugin on accuracy, AUROC, and F1, plus robustness and latency analyses.","tokens_in":18485,"tokens_out":8565,"duration_ms":99890,"significance":"If the mechanism works as described, CAW is a practically useful contribution: it is model-agnostic, applies to both reweighting- and sampling-based watermark families, and explicitly targets task accuracy rather than perplexity when measuring quality degradation. The tree-attention pre-generation mechanism is a sensible way to bound the latency cost of contextual lookahead, and the paper makes an honest case that perplexity is insufficient for watermark-quality evaluation. However, the evidence as presented does not yet establish that the contextual capacity evaluator is the source of the gains: the GPT-4o labels are not validated against the counterfactual quantity of interest, and the experimental comparisons lack variance information despite repeated runs.","major_comments":[{"comment":"The ground truth used to train the capacity evaluator is not shown to measure the quantity the method actually needs. The evaluator is trained on GPT-4o annotations of 'quality-critical segments' in human-written reference answers, while the central mechanism requires these labels to predict the counterfactual task-accuracy damage that watermarking causes at each token of model-generated chain-of-thought responses. Figure 3 validates the evaluator only against the GPT-4o labels; there is no experiment linking those labels, or the learned evaluator, to answer flips on GSM8K or MMLU. If the labels are noisy or task-specific, the improvements in Table 1 could come from the θ/β strength schedule rather than from contextual state awareness. Please add a direct validation (e.g., oracle capacity scores obtained by perturbing individual tokens and measuring answer changes, or an ablation with random capacity scores), and report the training-data provenance: source datasets, split, number of examples, the GPT-4o annotation prompt, and agreement statistics.","section":"§3.2, Eq. (1), and Appendix C"},{"comment":"The central claim that CAW 'consistently outperforms' baselines is made without variance information, even though Appendix F states that every setup was run five times. Several reported differences are small relative to the plausible run-to-run noise (e.g., GSM8K Llama2-7b: KGW+ours accuracy 0.1105 vs KGW-Soft 0.1140; EXP+ours 0.1084 vs EXP+entropy 0.1060), and AUC/F1 differences between CAW and hard baselines are often under 0.03. Please report standard deviations or confidence intervals and, where the same generated outputs are compared, paired significance tests; without these, 'consistently' and the Pareto-dominance statements are not supported.","section":"Table 1 and Figure 6"},{"comment":"The claim of 'full Pareto optimality' is based on only three manually selected strength settings for the baselines, with no error bars and no specification of how the soft/mid/hard tiers were calibrated across methods. If the baseline points are placed at unfavorable strengths, CAW can appear Pareto-dominant without providing a better algorithmic trade-off. Please provide denser trade-off curves, a transparent strength-calibration rule, and variance bands, or restrict the claim to the specific configurations tested.","section":"§4.3 and Figure 6"},{"comment":"The comparison does not isolate the contribution of the contextual capacity evaluator from the effect of simply applying a softer watermark. CAW is free to choose a per-token strength schedule, while the baselines are fixed hard/soft strengths; an ablation that matches the average watermark strength, green-token rate, or detection F1 of the baselines is needed to show that the per-token contextual selection, not just a lower effective embedding rate, drives the accuracy gains. The entropy-based plugin is the right kind of control, but it is also a heuristic threshold; a randomized-capacity control with the same θ/β schedule would strengthen the causal claim.","section":"§4.2/§4.3 and Table 1"}],"minor_comments":[{"comment":"The training objective as written is not a valid cross-entropy loss: it uses \\(\\hat C_i\\) both as the ground-truth label and inside \\(\\log(E(p(t_i); \\Theta))\\), so the expression collapses to \\(\\log(E(\\cdot))\\) regardless of the label. Please rename the label (e.g., \\(y_i\\)) and write the standard binary cross-entropy form \\(y_i \\log(E) + (1-y_i) \\log(1-E)\\).","section":"Eq. (5)"},{"comment":"The term 'watermark capacity' is used oppositely to its usual meaning: tokens with higher predicted capacity are left unmodified, while tokens with lower capacity receive the watermark. This will confuse readers; consider renaming the quantity to something like 'importance' or 'sensitivity', or explicitly define the inverse relationship more carefully.","section":"§3.2"},{"comment":"It should be stated explicitly that the \\(p(t_{i+1})\\) appearing in the evaluator input is \\(p(t_{i+1} \\mid t_i'^{(ori)})\\), the distribution conditioned on the original candidate token, not on the eventual selected token. As written, the notation \\([p(t_{i+k})]\\) in Eq. (1) is ambiguous about which candidate branch is used.","section":"§3.1, Step 4, and Eq. (1)"},{"comment":"The definitions \\(K' = \\beta (\\theta - \\hat C_i)/\\theta \\cdot K\\) and \\(\\delta' = \\beta (\\theta - \\hat C_i)/\\theta \\cdot \\delta\\) introduce \\(\\beta\\) and \\(\\theta\\) as empirical parameters, but no values or tuning ranges are reported anywhere in the paper; please provide these values or state how they were selected.","section":"Appendix D"},{"comment":"The text states that memory overhead remains within 5% for KGW and UNI and around 10% for EXP, but these numbers are not directly readable from the figure; please add explicit labels or a small table with the normalized values and their standard deviations.","section":"Figure 8"},{"comment":"The robustness section says each data point is the mean of five repeated trials with variance, but only means are plotted; please add error bars or explicitly state that the variance is omitted for visual clarity.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The framework and the motivation are solid, and the paper is readable. The main risk I see is that the learned capacity evaluator is trained on GPT-4o annotations whose relationship to the true quantity of interest is asserted rather than demonstrated; the missing variance reporting compounds this risk. I would expect the requested validations and statistical details to be achievable within the scope of a revision, so I am not recommending rejection. Editorial note: the dependence on GPT-4o for labeling has reproducibility and cost implications; the authors should disclose the annotation data and prompt in the supplementary material."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nWhat you should know: CAW is a genuinely new plug-and-play approach to reducing watermark-induced quality loss, and the empirical package is decent; the weak spot is that the capacity evaluator's supervision is underspecified and may not measure what the mechanism claims.\n\nThe core idea is simple: instead of using isolated token entropy to decide where to watermark, feed the probability distributions of the previous, current, and next token through a small learned network, get a per-token 'capacity' score, and scale the watermark strength accordingly. That's a real departure from the cited entropy-based and fixed-partition baselines. The paper integrates with KGW, Unigram, and EXP, covering both reweighting and sampling schemes, which is useful. It also evaluates with task accuracy on GSM8K and MMLU rather than perplexity, which is the right call for this problem. The tree-attention trick for pre-generating next-token distributions is a sensible engineering contribution, and the latency numbers look plausible.\n\nNow the soft spots. The main table has no error bars or variance despite five runs, so the reported gains are hard to gauge. More importantly, the capacity evaluator is trained on GPT-4o annotations of 'quality-critical segments' in human-written references. That's a proxy for watermark capacity, not a measurement of it. The paper never shows that those annotations align with counterfactual task-accuracy damage from token substitution. Figure 3 only shows the evaluator predicts the GPT-4o labels well, not that protecting those tokens actually preserves accuracy. So there is a real risk that the adaptive strength schedule is just a more selective way of softening the watermark, and the 'contextual state awareness' story is oversold. The missing training-set details (how many examples, what prompt, inter-annotator agreement) deepen this. No code or trained weights either. A clean ablation that isolates the evaluator (e.g., random or confidence-based adaptive strength) would have spoken to this.\n\nThat said, the paper is honest about its limits in the appendix, and the central claim holds up as a reasonable engineering statement. It's a solid subfield-level advance, not a paradigm shift. People working on watermarking quality or deploying watermarks in quality-sensitive applications will get value from it.\n\nI'd send it out for review. The evaluator training and ablations need substantial strengthening, but the idea and empirical direction are worth referee time.","headline":"Useful plug-and-play idea for watermark quality preservation, but the evaluator's training story is thin and the gains may partly reflect a learned softness schedule.","tokens_in":19042,"tokens_out":6892,"would_cite":true,"duration_ms":53240,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A plug-and-play watermarking framework that adapts watermark strength to each token's contextual importance can improve both detection rate and output quality.","keywords":["LLM watermarking","contextual generation states","watermark capacity evaluator","adaptive watermark strength","tree attention","generation quality","AIGT detection"],"falsifier":"Take a task where the token that determines correctness is low-probability and isolated, such as a single digit in a long arithmetic chain, and compare CAW's accuracy-detectability frontier against the hard baseline at matched detection rate; if CAW's accuracy advantage over the entropy-based plugin disappears or reverses when the evaluator is trained on a different token-importance distribution, the claim of general quality preservation would be falsified.","tokens_in":18090,"feed_emoji":"🛡️","tokens_out":5206,"duration_ms":58149,"temperature":0.7,"pith_summary":"The paper claims that LLM watermarking can degrade generated text because existing methods decide where and how strongly to watermark using heuristics such as token confidence or entropy, which do not capture true semantic importance. It introduces a contextual generation states-aware watermarking framework, CAW, that first predicts each token's watermark capacity from the probability distributions of the token and its immediate context, then leaves important tokens untouched and scales watermark strength for the rest. A tree-attention pre-generation mechanism makes this adaptive strategy fast enough for practical use. The result, evaluated on reasoning and knowledge benchmarks across four model variants, is a better trade-off between detectability and task accuracy than hard, soft, and entropy-based watermarking baselines.","feed_headline":"Watermarking that adapts strength per token preserves LLM accuracy","feed_subtitle":"CAW predicts which tokens matter from neighboring generation probabilities, then waters only the rest to keep answers correct.","key_machinery":"The key mechanism is the watermark capacity evaluator, a three-layer fully connected network that takes the ranked top-100 probability values of $p(t_{i-1})$, $p(t_i)$, and $p(t_{i+1}\\,|\\,t_i')$ as input and outputs a capacity score $\\hat{C}_i \\in (0,1)$: $\\hat{C}_i = E([p(t_{i+k})]_{k=-N^-}^{N^+}; \\Theta^*)$ with $N^-=N^+=1$. The network is trained with cross-entropy on segments marked by a strong external judge model as quality-critical. Tokens with $\\hat{C}_i > \\theta$ are protected from watermarking; for other tokens the watermark strength is scaled as $K' = \\beta(\\theta-\\hat{C}_i)/\\theta\\,K$ for sampling-based methods and $\\delta' = \\beta(\\theta-\\hat{C}_i)/\\theta\\,\\delta$ for reweighting methods. To avoid the latency of looking ahead at future tokens, the framework uses tree attention to pre-generate all candidate next-token distributions in a single forward pass, so the adaptive decision comes at only a small overhead.","core_discovery":"The central claim is that watermark-induced quality loss can be minimized by making the watermark injection itself aware of contextual generation states. The paper defines watermark capacity as a token's semantic importance and tolerance to perturbation, learns an evaluator that predicts this capacity from the probability distributions of the previous, current, and next token positions, and then adaptively watermarks: tokens predicted to be critical are generated without watermarking, while tokens with spare capacity receive watermark strength scaled to that capacity. Integrated with three base watermarking methods (KGW, Unigram, and EXP), CAW achieves higher task accuracy at comparable detection rates on GSM8K and MMLU across Llama2 and Vicuna models at 7B and 13B scales. The paper also argues that perplexity is an inadequate quality metric because watermarking can change a critical digit or fact while leaving fluency intact, and therefore uses task-level accuracy as the quality measure.","pith_inferences":["The paper leaves the capacity evaluator tied to its two evaluation tasks; a natural extension is to test whether the same evaluator transfers to code, dialogue, or other domains whose critical tokens differ sharply.","If the learned capacity scores track human-judged informational importance, the evaluator could serve as a reusable token-importance signal beyond watermarking, though this is not examined.","A testable extension is training the evaluator on multi-bit or sentence-level watermarks, where the strength variable is richer than a single scalar and the capacity prediction may need to be aggregated over longer spans."],"forward_implications":["Existing single-bit watermarking methods can be wrapped by CAW without changing their detection procedure, gaining accuracy at matched detectability; the paper demonstrates this for KGW, Unigram, and EXP.","Perplexity-based claims of quality preservation become suspect, because task-level accuracy is what reveals watermark harm in reasoning and knowledge tasks.","The multi-branch pre-generation mechanism keeps added latency below about 15% and memory overhead within roughly 5-10%, so adaptive watermarking remains practical.","On GSM8K, CAW reaches accuracy close to the soft baseline while holding detection rates near the hard baseline, indicating users do not have to choose between detectability and answer correctness."],"supporting_citations":[{"why":"Introduces the KGW reweighting watermark that serves as one of the base methods CAW is integrated with.","marker":"[5]"},{"why":"Introduces the Unigram watermark with a fixed vocabulary partition, used as a second base method.","marker":"[7]"},{"why":"Introduces the EXP sampling-based watermark, the third base method, and distortion-free watermarking ideas.","marker":"[21]"},{"why":"Supplies the entropy-based token-selection plugin that CAW is compared against as the main quality-enhancement baseline.","marker":"[8]"},{"why":"Provides the tree attention technique that enables multi-branch pre-generation and keeps the adaptive watermarking latency low.","marker":"[14]"},{"why":"Provides the open-source watermarking implementation used to build the baseline and reproduction setup.","marker":"[28]"},{"why":"Defines the Llama2 models used to evaluate CAW across two model scales.","marker":"[31]"},{"why":"Defines the Vicuna-v1.5 models used to evaluate CAW across a second model family.","marker":"[32]"},{"why":"Provides the MMLU sociology subset used to measure watermark impact on knowledge memory.","marker":"[33]"},{"why":"Provides the GSM8K math word problems used to measure watermark impact on reasoning accuracy.","marker":"[34]"}],"fun_headline_variants":["Adaptive watermarking preserves LLM text quality by targeting low-impact tokens","Context-aware watermarking picks the right tokens to keep output accurate","State-aware watermarking: preserve key tokens while embedding watermarks","Watermarking that learns which words are critical to avoid degrading answers","Target watermarks to low-impact tokens to keep LLM answers accurate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the training labels produced by the external judge model on the training prompts correctly identify the tokens that must stay untouched for the answer to remain correct, and that this mapping carries over to unseen prompts and models.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive watermarking preserves LLM text quality by targeting low-impact tokens","Context-aware watermarking picks the right tokens to keep output accurate","State-aware watermarking: preserve key tokens while embedding watermarks","Watermarking that learns which words are critical to avoid degrading answers","Target watermarks to low-impact tokens to keep LLM answers accurate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000642,"raw_usage":{"total_tokens":2956,"prompt_tokens":948,"completion_tokens":2008,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":1918}},"tokens_in":564,"tokens_out":2008,"duration_ms":16456,"temperature":1.0,"reasoning_tokens":1918,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:35:13.691930+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a task where the token that determines correctness is low-probability and isolated, such as a single digit in a long arithmetic chain, and compare CAW's accuracy-detectability frontier against the hard baseline at matched detection rate; if CAW's accuracy advantage over the entropy-based plugin disappears or reverses when the evaluator is trained on a different token-importance distribution, the claim of general quality preservation would be falsified.","supporting_citations":[{"cited_title":"A Watermark for Large Language Models","cited_arxiv_id":null,"evidence_quote":"Introduces the KGW reweighting watermark that serves as one of the base methods CAW is integrated with."},{"cited_title":"SpecInfer: Accelerating Large Language Model Serving with Tree-Based Speculative Inference and Verification","cited_arxiv_id":null,"evidence_quote":"Provides the tree attention technique that enables multi-branch pre-generation and keeps the adaptive watermarking latency low."},{"cited_title":"Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality, 2023.https://vicuna.lmsys.org","cited_arxiv_id":null,"evidence_quote":"Defines the Vicuna-v1.5 models used to evaluate CAW across a second model family."}],"review_version":1}