{"id":"b0be6b15-3c27-49a3-9e38-e6f0c549d2ab","arxiv_id":"2506.22623","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A watermark that seeds each token's sampling temperature from a hash of the previous h tokens is claimed to beat the Aaronson watermark under a 30% BERT paraphrase attack, based on a single ROC curve without error bars.","lead":"This paper proposes a text watermarking method that varies the sampling temperature of a language model based on a hash of recent tokens, and reports it detects machine-generated text more reliably than a prior watermarking method even after 30% of words are paraphrased. The claim matters because robust watermarks could help trace AI-generated content, but the paper provides no code, no parameter values, and no error bars, so the result is not yet reproducible.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control: the score may detect machine-generated text generally, not the temperature watermark; paraphrased-text scores are also computed under mismatched temperatures.","rationale":"The reader's REJECT is sound, but the most load-bearing gap is broader than the temperature-mismatch issue. The paper's detection statistic, Eq. (3), is an average of per-token softmax probabilities computed under a temperature schedule that is itself a deterministic function of the text. This is close to a standard likelihood-based machine-text detector, not a cryptographic watermark. The evaluation's negative class appears to be human-written text only; no unwatermarked Vicuna outputs are tested. If unwatermarked Vicuna text also scores high, then the method's discriminative power has nothing to do with the temperature watermark, and the claimed advantage over the Aaronson baseline would be a comparison of detectors rather than watermarks. The temperature-mismatch concern raised by the reader is real and important for the paraphrase claim, but it is secondary to this missing control: without the control, even the non-paraphrased result is uninterpretable. The concrete tests I propose would settle both issues. I therefore leave the reader's REJECT verdict unchanged.","tokens_in":4908,"tokens_out":8933,"duration_ms":99814,"concrete_test":"Generate 1,000 unwatermarked Vicuna-7B responses to the same Alpaca prompts using the same base sampling temperature T0 (no per-token temperature variation). Score them with Eq. (3) exactly as in the paper. If the ROC for unwatermarked-vs-human matches the reported watermarked-vs-human ROC (TPR≈90% at FPR=2%), the temperature schedule is not the source of detectability. Additionally, for the paraphrase test, save the generation-time temperatures T*_t at each position; on the BERT-paraphrased texts, compare the detection score computed with the paper's re-derived T_t against the score computed with the saved T*_t. If the re-derived score is substantially closer to the human-text distribution, the mismatch is the reason the 30% robustness result cannot generalize.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the temperature schedule in Eq. (1)/(3) creates a detectable watermark that survives paraphrasing. The evaluation in Section 3.3 compares watermarked Vicuna text only against human-written Alpaca answers (as stated: 'comparing them with human-written ones'). Figure 3's 'non-watermarked' class is human text, not unwatermarked Vicuna text. Therefore the observed separation may be entirely a generic model-likelihood effect: Eq. (3) is an average of per-token model probabilities under a temperature schedule derived from the text itself, so any fluent Vicuna output may score high. Without a control of unwatermarked Vicuna outputs, the experiment cannot attribute the TPR=90% result to the watermark. Second, under the BERT paraphrase (30% token replacement, Section 3.2), the detector re-computes T_t from the modified context via the hash of the last h tokens, so for roughly 1-(1-0.3)^h tokens the temperature differs from the generation-time value. The paper gives no analysis or ablation of this mismatch; if the score separation disappears when correct temperatures are replaced by mismatched ones, the claimed paraphrase robustness is an artifact of the particular 30% BERT attack. Both gaps are testable, and together they leave the central claim unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a temperature-based watermark for LLM text: at each generated token t, a hash of the previous h tokens seeds a pseudo-random temperature T_t (Eq. (1)), and the logits are scaled by 1/T_t before softmax sampling (Alg. 1). Detection computes the average conditional probability of the text's tokens under the same temperature schedule (Eq. (3)). The authors replicate an earlier watermarking baseline on Vicuna-7B, then evaluate their method against human-written Alpaca answers, with and without a BERT-based 30% token-substitution paraphrase attack (Section 3). They report that at FPR = 2% the baseline TPR is 15% while their method reaches 90% TPR, and that this advantage survives the paraphrase attack (Section 4). The paper concludes that the temperature watermark is more robust than the Aaronson et al. baseline.","tokens_in":5205,"tokens_out":5119,"duration_ms":57340,"significance":"If the reported result were supported, a watermark that remains detectable after 30% token substitution would be a practically significant contribution, and the failed replication of the baseline on an open-source model would also be worth reporting. The paper also has genuine strengths: it targets a real attack, uses an open-source model, and states a concrete, testable detection score. However, the experimental evidence as presented does not support the central claim. The non-watermarked comparison class is human text rather than unwatermarked Vicuna output, the detection-time temperature schedule is recomputed from paraphrased text without any analysis of the resulting mismatch, and no watermarking hyperparameters are reported. These gaps are load-bearing for the claimed TPR=90% result, so the contribution is currently unsubstantiated.","major_comments":[{"comment":"The control condition is not an unwatermarked LLM baseline: Section 3.3 states that the generated texts are 'comparing them with human-written ones' from the Alpaca dataset. Eq. (3) is an average per-token likelihood under the Vicuna distribution after a temperature schedule, so any fluent Vicuna output, watermarked or not, may score substantially higher than human text. Without a control in which the same prompts are used to generate unwatermarked Vicuna text, the TPR=90% at FPR=2% reported in Section 4 cannot be attributed to the watermark rather than to a generic model-likelihood effect.","section":"Section 3.3; Figure 3"},{"comment":"Under the paraphrase attack, Eq. (3) recomputes T_t from the hash of the last h tokens of the paraphrased sequence. With 30% token replacement, for a large fraction of positions the detection-time temperature differs from the generation-time temperature, yet the paper provides no analysis or ablation of this mismatch. The robustness claim (H2) depends entirely on the score remaining discriminative under mismatched temperatures; without a controlled comparison of matched versus mismatched schedules, the claimed paraphrase invariance is unsupported.","section":"Section 2.4; Section 3.2"},{"comment":"The watermarking parameters h, T0, m, and M are never specified, and no sensitivity study is provided. Because the separation in Figure 3 could depend critically on these values, the omission prevents reproducibility and leaves open the possibility that the reported operating point is a favorable or even degenerate choice.","section":"Section 2.3; Eq. (1)"},{"comment":"The quantitative evaluation is based on a single run over 1k Alpaca samples, with no error bars, no multiple seeds, and no statistical significance testing. Given that the baseline distributions in Figure 3 overlap heavily, it is impossible to assess whether the reported TPR/FPR values are stable or the result of a particular draw.","section":"Section 3.3; Section 4"}],"minor_comments":[{"comment":"There is a typo: 'asses' should be 'assess'.","section":"Abstract"},{"comment":"The word 'reprocude' should be 'reproduce'.","section":"Section 3.1"},{"comment":"The phrase 'random 1k samples fro Alpaca' contains a typo: 'fro' should be 'from'.","section":"Section 3.3"},{"comment":"The meaning of the 'synonym' rows in the figure is not defined in the caption; the caption should explain that 'synonym = True' means the BERT paraphrase attack was applied.","section":"Figure 3"},{"comment":"The sentence after Eq. (2) states that temperature 'forces the model to take a specific path during generation,' but the mechanism is not spelled out; a more precise description of how temperature variation makes generated tokens more probable under Eq. (3) would help the reader.","section":"Section 2.3"},{"comment":"No link to code or data is provided, which limits reproducibility and verification of the reported numbers.","section":"General"}],"recommendation":"reject","confidential_remarks":"The paper appears to be an early-stage project report rather than a complete research paper. The main issue is not stylistic but evidentiary: the central empirical claim lacks the necessary control (unwatermarked Vicuna output), the paraphrase-robustness claim lacks any treatment of the temperature mismatch, and the key hyperparameters are undisclosed. I would encourage the authors to add the missing control, report parameters and multiple runs, and include an ablation of mismatched temperature schedules before resubmitting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know first: the temperature-seeded watermark is a plausible idea, but the paper's headline claim is not supported by its own experiments. The biggest problem is a missing control. The detection comparison is between watermarked Vicuna text and human-written Alpaca text, not between watermarked and unwatermarked Vicuna text. So the reported 90% TPR at 2% FPR may simply reflect that model-generated text scores higher under the model's own likelihood, not that the watermark itself is doing work. The stress-test note is right about this, and it lands on the central claim.\n\nSecond: the paraphrase robustness claim bumps into a temperature-mismatch problem. Under the 30% BERT token replacement, the detector recomputes temperatures from the paraphrased text, so for a large fraction of positions the detection-time temperature differs from the generation-time value. The paper gives no analysis or ablation of this mismatch. If the score separation survives only when temperatures match, the claimed robustness is an artifact of the particular attack.\n\nWhat the paper does well: it replicates the Aaronson watermark on Vicuna and shows that the published detection results do not transfer cleanly to a different base model—a modest but real negative result. It also spells out a concrete new watermark variant in Algorithm 1, a natural extension of the hash-based line. The writing is clear enough that a reader can see exactly what was done, which is not always true of workshop-level preprints.\n\nWhere it falls short: beyond the missing control, the evaluation is thin. Single run on 1k Alpaca samples, no code, no data, no hyperparameters (h, T0, m, M), no error bars, no significance tests. These are not minor quibbles; they make the core quantitative claim impossible to check.\n\nWho this is for: a workshop audience might appreciate the negative replication and the idea sketch, but a serious venue needs the control and the mismatch analysis before the robustness result can be taken seriously.\n\nRecommendation: I would not send this to peer review in its current form. The right move is to desk-reject with clear instructions to the authors to add unwatermarked Vicuna outputs as a control, analyze the temperature mismatch, and provide code and hyperparameters. If they redo the evaluation and the separation survives, the idea could be worth a look.","headline":"A plausible watermarking variant whose central empirical claim is unsupported by the missing control and the temperature-mismatch problem.","tokens_in":5702,"tokens_out":2631,"would_cite":false,"duration_ms":30794,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Temperature-seeded watermarking keeps machine text detectable after paraphrasing.","keywords":["LLM watermarking","temperature scaling","paraphrasing attack","synthetic text detection","hash-seeded randomness","watermark robustness","false positive rate"],"falsifier":"Paraphrase the watermarked outputs at several replacement rates, recompute the detection score using the paper's temperature re-derivation, and measure the true positive rate at the 2% false positive threshold; if the true positive rate on attacked text falls well below 90%, the robustness claim is refuted. A sharper check is to compare, token by token, the temperature derived from the original context with the temperature derived from the paraphrased context and locate at what divergence the score separation collapses.","tokens_in":4743,"feed_emoji":"🌡️","tokens_out":6258,"duration_ms":61811,"temperature":0.7,"pith_summary":"This paper tries to establish that watermarking generated text by varying the sampling temperature from token to token, with each temperature chosen deterministically from a hash of the preceding h tokens, is a workable and robust way to detect machine-written text. The authors reproduce a baseline watermarking method and find its detection sharply weakens on a different open model, then propose the temperature scheme and test it against a paraphrasing attack that rewrites 30% of the tokens. Their headline numbers are a true positive rate of 90% at a 2% false positive rate for the proposed method, versus 15% for the baseline, with the advantage surviving the attack. If true, this matters because paraphrase rewriting is one of the most practical ways to strip watermarks from AI text before it spreads.","feed_headline":"Temperature watermark beats paraphrase attacks: 90% vs 15%","feed_subtitle":"By hashing the last h tokens into a per-token temperature, machine text stays detectable after 30% of words are rewritten.","key_machinery":"The load-bearing object is the per-token temperature schedule: a deterministic map from the hash of the previous h tokens to a temperature in a range around a base value, applied as logit scaling before the softmax. Because the same hash is recomputable at detection time, the temperature schedule functions as a private code shared between generation and detection; it changes the entropy of the output distribution token by token, leaving a trace in the model's self-assigned probabilities. The detection statistic is the average of those recomputed token probabilities, which the paper argues is higher for watermarked text than for human text and remains higher after partial paraphrase.","core_discovery":"The central claim is that a watermark can be encoded in the temperature trajectory of generation rather than in word choices. At each step the generator hashes the last h tokens, uses that hash to seed a pseudo-random temperature, and divides the logits by that temperature before sampling. To detect, one runs the same model over the text, recomputes the same temperatures from the observed context, and averages the probability the model assigned to the observed token. The paper claims this score separates human and machine text clearly, whereas the baseline method's score distributions overlap, and that the separation remains after a masked-language paraphrase attack replaces 30% of the tokens. The quantitative anchor is the ROC comparison: at a fixed 2% false positive rate, the method yields a 90% true positive rate versus the baseline's 15%.","pith_inferences":["The paper does not report how the robustness degrades as the paraphrase rate rises above 30%; a natural extension is to map the true positive rate against attack rate to find the breaking point.","Because the watermark lives in the temperature schedule rather than in a green/red vocabulary split, it could in principle be combined with other watermarks; the paper does not test such combinations.","The scheme's discriminative power probably depends on how often high-entropy positions occur, where a low or high temperature flips the probabilities; this dependence is not isolated in the paper.","If the hash window h is short, a paraphraser changing one word can alter several downstream temperature draws; the paper does not analyze the window-size effect on robustness."],"forward_implications":["If the claim holds, machine-generated text can be marked without changing the set of words the model is allowed to use, only the sharpness of the distribution they are sampled from.","At a 2% false positive rate the method's 90% true positive rate, even after 30% token replacement, suggests paraphrase-only attacks are not sufficient to wash out the watermark.","The paper's failure to reproduce the baseline on a different open model implies watermark performance is model-dependent and must be re-validated when the generator changes.","The detection score is cheap to compute: one forward pass over the text and an average, so the method could be applied at scale to monitor generated content."],"supporting_citations":[{"why":"Supplies the baseline watermarking scheme and its reported detection results that the paper reproduces and compares against.","marker":"[1]"},{"why":"Introduces the paraphrasing attack as a threat to LLM watermarks, motivating the paper's robustness goal.","marker":"[8]"},{"why":"Defines the green/red-list watermark and the claim of model-agnosticism that Hypothesis H1 tests; the paper contrasts its temperature approach with this family.","marker":"[9]"},{"why":"Provides the masked-language model used to replace 30% of tokens in the paraphrase attack.","marker":"[11]"},{"why":"Supplies the open-source 7B model on which generation, detection, and the ROC comparison are run.","marker":"[12]"},{"why":"Documents a significant drop in detection performance on paraphrased text, which the paper's Hypothesis H2 aims to overcome.","marker":"[14]"}],"fun_headline_variants":["Temperature watermark survives paraphrasing: 90% vs 15%","Paraphrase-proof watermark: temperature encoding hits 90%","Temperature watermark resists paraphrase attacks: 90% vs 15%","Temperature encoding watermark beats rewriting: 90% vs 15%","Watermark via temperature resists 30% paraphrase: 90% vs 15%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The detection score is found by deriving temperatures from the text under examination, and the paper assumes that when a paraphraser swaps in words the generator never saw, the temperature sequence re-derived from the modified text still matches the original closely enough that watermarked text keeps scoring above human text, yet no analysis demonstrates this mismatch tolerance.","fun_headline_variants_meta":{"raw":{"variants":["Temperature watermark survives paraphrasing: 90% vs 15%","Paraphrase-proof watermark: temperature encoding hits 90%","Temperature watermark resists paraphrase attacks: 90% vs 15%","Temperature encoding watermark beats rewriting: 90% vs 15%","Watermark via temperature resists 30% paraphrase: 90% vs 15%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000668,"raw_usage":{"total_tokens":3007,"prompt_tokens":866,"completion_tokens":2141,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":2044}},"tokens_in":482,"tokens_out":2141,"duration_ms":15216,"temperature":1.0,"reasoning_tokens":2044,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:02:12.274977+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Paraphrase the watermarked outputs at several replacement rates, recompute the detection score using the paper's temperature re-derivation, and measure the true positive rate at the 2% false positive threshold; if the true positive rate on attacked text falls well below 90%, the robustness claim is refuted. A sharper check is to compare, token by token, the temperature derived from the original context with the temperature derived from the paraphrased context and locate at what divergence the score separation collapses.","supporting_citations":[{"cited_title":"Aaronson and H","cited_arxiv_id":null,"evidence_quote":"Supplies the baseline watermarking scheme and its reported detection results that the paper reproduces and compares against."}],"review_version":1}