REVIEW 4 major objections 6 minor 14 references
Temperature Matters: Enhancing Watermark Robustness Against Paraphrasing Attacks
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Temperature-seeded watermarking keeps machine text detectable after paraphrasing.
desk verdict A plausible watermarking variant whose central empirical claim is unsupported by the missing control and the temperature-mismatch problem. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the per-token temperature schedule: a deterministic map from the hash of the previous h tokens to a temperature in a range around a base value, applied as logit scaling before the softmax. Because the same hash is recomputable at detection time, the temperature schedule functions as a private code shared between generation and detection; it changes the entropy of the output distribution token by token, leaving a trace in the model's self-assigned probabilities. The detection statistic is the average of those recomputed token probabilities, which the paper argues is higher for watermarked text than for human text and remains higher after partial paraphrase.
What would settle it
Paraphrase the watermarked outputs at several replacement rates, recompute the detection score using the paper's temperature re-derivation, and measure the true positive rate at the 2% false positive threshold; if the true positive rate on attacked text falls well below 90%, the robustness claim is refuted. A sharper check is to compare, token by token, the temperature derived from the original context with the temperature derived from the paraphrased context and locate at what divergence the score separation collapses.
Extended reading notes
Core claim
The central claim is that a watermark can be encoded in the temperature trajectory of generation rather than in word choices. At each step the generator hashes the last h tokens, uses that hash to seed a pseudo-random temperature, and divides the logits by that temperature before sampling. To detect, one runs the same model over the text, recomputes the same temperatures from the observed context, and averages the probability the model assigned to the observed token. The paper claims this score separates human and machine text clearly, whereas the baseline method's score distributions overlap, and that the separation remains after a masked-language paraphrase attack replaces 30% of the tokens. The quantitative anchor is the ROC comparison: at a fixed 2% false positive rate, the method yields a 90% true positive rate versus the baseline's 15%.
Load-bearing premise
The detection score is found by deriving temperatures from the text under examination, and the paper assumes that when a paraphraser swaps in words the generator never saw, the temperature sequence re-derived from the modified text still matches the original closely enough that watermarked text keeps scoring above human text, yet no analysis demonstrates this mismatch tolerance.
Editorial extensions
If this is right
- If the claim holds, machine-generated text can be marked without changing the set of words the model is allowed to use, only the sharpness of the distribution they are sampled from.
- At a 2% false positive rate the method's 90% true positive rate, even after 30% token replacement, suggests paraphrase-only attacks are not sufficient to wash out the watermark.
- The paper's failure to reproduce the baseline on a different open model implies watermark performance is model-dependent and must be re-validated when the generator changes.
- The detection score is cheap to compute: one forward pass over the text and an average, so the method could be applied at scale to monitor generated content.
Reading between the lines
- The paper does not report how the robustness degrades as the paraphrase rate rises above 30%; a natural extension is to map the true positive rate against attack rate to find the breaking point.
- Because the watermark lives in the temperature schedule rather than in a green/red vocabulary split, it could in principle be combined with other watermarks; the paper does not test such combinations.
- The scheme's discriminative power probably depends on how often high-entropy positions occur, where a low or high temperature flips the probabilities; this dependence is not isolated in the paper.
- If the hash window h is short, a paraphraser changing one word can alter several downstream temperature draws; the paper does not analyze the window-size effect on robustness.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a temperature-based watermark for LLM text: at each generated token t, a hash of the previous h tokens seeds a pseudo-random temperature T_t (Eq. (1)), and the logits are scaled by 1/T_t before softmax sampling (Alg. 1). Detection computes the average conditional probability of the text's tokens under the same temperature schedule (Eq. (3)). The authors replicate an earlier watermarking baseline on Vicuna-7B, then evaluate their method against human-written Alpaca answers, with and without a BERT-based 30% token-substitution paraphrase attack (Section 3). They report that at FPR = 2% the baseline TPR is 15% while their method reaches 90% TPR, and that this advantage survives the paraphrase attack (Section 4). The paper concludes that the temperature watermark is more robust than the Aaronson et al. baseline.
Significance. If the reported result were supported, a watermark that remains detectable after 30% token substitution would be a practically significant contribution, and the failed replication of the baseline on an open-source model would also be worth reporting. The paper also has genuine strengths: it targets a real attack, uses an open-source model, and states a concrete, testable detection score. However, the experimental evidence as presented does not support the central claim. The non-watermarked comparison class is human text rather than unwatermarked Vicuna output, the detection-time temperature schedule is recomputed from paraphrased text without any analysis of the resulting mismatch, and no watermarking hyperparameters are reported. These gaps are load-bearing for the claimed TPR=90% result, so the contribution is currently unsubstantiated.
major comments (4)
- [Section 3.3; Figure 3] The control condition is not an unwatermarked LLM baseline: Section 3.3 states that the generated texts are 'comparing them with human-written ones' from the Alpaca dataset. Eq. (3) is an average per-token likelihood under the Vicuna distribution after a temperature schedule, so any fluent Vicuna output, watermarked or not, may score substantially higher than human text. Without a control in which the same prompts are used to generate unwatermarked Vicuna text, the TPR=90% at FPR=2% reported in Section 4 cannot be attributed to the watermark rather than to a generic model-likelihood effect.
- [Section 2.4; Section 3.2] Under the paraphrase attack, Eq. (3) recomputes T_t from the hash of the last h tokens of the paraphrased sequence. With 30% token replacement, for a large fraction of positions the detection-time temperature differs from the generation-time temperature, yet the paper provides no analysis or ablation of this mismatch. The robustness claim (H2) depends entirely on the score remaining discriminative under mismatched temperatures; without a controlled comparison of matched versus mismatched schedules, the claimed paraphrase invariance is unsupported.
- [Section 2.3; Eq. (1)] The watermarking parameters h, T0, m, and M are never specified, and no sensitivity study is provided. Because the separation in Figure 3 could depend critically on these values, the omission prevents reproducibility and leaves open the possibility that the reported operating point is a favorable or even degenerate choice.
- [Section 3.3; Section 4] The quantitative evaluation is based on a single run over 1k Alpaca samples, with no error bars, no multiple seeds, and no statistical significance testing. Given that the baseline distributions in Figure 3 overlap heavily, it is impossible to assess whether the reported TPR/FPR values are stable or the result of a particular draw.
minor comments (6)
- [Abstract] There is a typo: 'asses' should be 'assess'.
- [Section 3.1] The word 'reprocude' should be 'reproduce'.
- [Section 3.3] The phrase 'random 1k samples fro Alpaca' contains a typo: 'fro' should be 'from'.
- [Figure 3] The meaning of the 'synonym' rows in the figure is not defined in the caption; the caption should explain that 'synonym = True' means the BERT paraphrase attack was applied.
- [Section 2.3] The sentence after Eq. (2) states that temperature 'forces the model to take a specific path during generation,' but the mechanism is not spelled out; a more precise description of how temperature variation makes generated tokens more probable under Eq. (3) would help the reader.
- [General] No link to code or data is provided, which limits reproducibility and verification of the reported numbers.
Circularity Check
No significant circularity: detection score is a likelihood-based soft watermark, and the claimed robustness results are empirical comparisons, not derivations from the method's own output.
full rationale
The paper's derivation chain is self-contained: no prediction is identical to an input by construction. The watermarking generation (Alg. 1) uses a hash of the previous h tokens to seed a temperature T_t, and detection (Eq. 3) averages the conditional probabilities recomputed under that same temperature schedule. This is the standard construction of a soft watermark: watermarked text is scored under the distribution from which it was sampled. That does not by itself prove the claimed TPR=90% at FPR=2%; the score separation is an empirical ROC measurement against human-written Alpaca answers and against the Aaronson baseline, and the paraphrase robustness is measured under a 30% BERT masked-token replacement. The evaluation does lack a control of unwatermarked Vicuna text, and paraphrase changes the context so detection-time temperatures can differ from generation-time values, but these are experimental validity gaps rather than circular reductions. There are no load-bearing self-citations: references [1], [8], [9], and [14] are external prior work, and the method does not invoke a uniqueness theorem or a fitted parameter renamed as prediction. Hence no circularity score above 0 is warranted.
Assumptions & free parameters
free parameters (4)
- h
- T0
- m
- M
assumptions (5)
- domain assumption The probability scores P(st | s1..st-1; Tt) from a single Vicuna-7B forward pass are well-calibrated enough to separate human and watermarked text.
- domain assumption The BERT masked-language-model procedure that replaces 30% of tokens sampled one-by-one is a faithful proxy for paraphrasing attacks.
- domain assumption The hash of the last h tokens can be recomputed by the detector to recreate the temperature sequence; no secret key or adversarial model is considered.
- ad hoc to paper The temperature sampling law Tt = T0(m + (M - m)Ut) with unstated m, M, T0, h is a valid watermarking device.
- standard math The softmax with temperature in Eq. (2) yields a proper probability distribution, and the average of conditional probabilities in Eq. (3) defines a scalar score.
Cite this review
Pith. "Pith review of Temperature Matters: Enhancing Watermark Robustness Against Paraphrasing Attacks." pith.science (2026). https://pith.science/paper/MFM22LEW
@misc{pith2026250622623,
author = {Pith},
title = {Pith review of: Temperature Matters: Enhancing Watermark Robustness Against Paraphrasing Attacks},
year = {2026},
howpublished = {\url{https://pith.science/paper/MFM22LEW}},
note = {Machine review of arXiv:2506.22623}
}
read the original abstract
In the present-day scenario, Large Language Models (LLMs) are establishing their presence as powerful instruments permeating various sectors of society. While their utility offers valuable support to individuals, there are multiple concerns over potential misuse. Consequently, some academic endeavors have sought to introduce watermarking techniques, characterized by the inclusion of markers within machine-generated text, to facilitate algorithmic identification. This research project is focused on the development of a novel methodology for the detection of synthetic text, with the overarching goal of ensuring the ethical application of LLMs in AI-driven text generation. The investigation commences with replicating findings from a previous baseline study, thereby underscoring its susceptibility to variations in the underlying generation model. Subsequently, we propose an innovative watermarking approach and subject it to rigorous evaluation, employing paraphrased generated text to asses its robustness. Experimental results highlight the robustness of our proposal compared to the~\cite{aarson} watermarking method.
Figures
Reference graph
Works this paper leans on
-
[1]
S. Aaronson and H. Kirchner. Watermarking gpt outputs. https://www.scottaaronson.com/talks/ watermark.ppt, 2023
work page 2023
-
[2]
Guiding the release of safer e2e conversational ai through value sensitive design
A Stevie Bergman, Gavin Abercrombie, Shannon Spruit, Dirk Hovy, Emily Dinan, Y-Lan Boureau, Verena Rieser, et al. Guiding the release of safer e2e conversational ai through value sensitive design. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue . Association for Computational Linguistics, 2022
work page 2022
-
[3]
The threat of offensive ai to organizations
Yisroel Mirsky, Ambra Demontis, Jaidip Kotak, Ram Shankar, Deng Gelei, Liu Yang, Xiangyu Zhang, Maura Pintor, Wenke Lee, Yuval Elovici, et al. The threat of offensive ai to organizations. Computers & Security, page 103006, 2022
work page 2022
-
[4]
Katarina Kertysova. Artificial intelligence and disinformation: How ai changes the way disinformation is produced, disseminated, and can be countered. Security and Human Rights , 29(1-4):55–81, 2018. 5 A PREPRINT
work page 2018
-
[5]
On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623, 2021
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623, 2021
2021
-
[6]
Machine-generated text: A comprehensive survey of threat models and detection methods
Evan Crothers, Nathalie Japkowicz, and Herna L Viktor. Machine-generated text: A comprehensive survey of threat models and detection methods. IEEE Access, 2023
work page 2023
-
[7]
The ethical need for watermarks in machine-generated language
Alexei Grinbaum and Laurynas Adomaitis. The ethical need for watermarks in machine-generated language. arXiv preprint arXiv:2209.03118, 2022
arXiv 2022
-
[8]
On the reliability of watermarks for large language models
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of watermarks for large language models. arXiv preprint arXiv:2306.04634, 2023
arXiv 2023
Show all 14 references
-
[9]
A watermark for large language models
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. arXiv preprint arXiv:2301.10226, 2023
2023 arXiv
-
[10]
Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust
Yuxin Wen, John Kirchenbauer, Jonas Geiping, and Tom Goldstein. Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust. arXiv preprint arXiv:2305.20030, 2023
2023 arXiv
-
[11]
Bidirectional encoder representations from transformers (bert): A sentiment analysis odyssey
Shivaji Alaparthi and Manit Mishra. Bidirectional encoder representations from transformers (bert): A sentiment analysis odyssey. arXiv preprint arXiv:2007.01127, 2020
2007
-
[12]
Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , 2023
2023
-
[13]
Llama: Open and efficient foundation language models, 2023
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...
2023
-
[14]
Three bricks to consolidate watermarks for large language models
Pierre Fernandez, Antoine Chaffin, Karim Tit, Vivien Chappelier, and Teddy Furon. Three bricks to consolidate watermarks for large language models. CoRR, abs/2308.00113, 2023. 6
2023 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.