RECORD
Baselines for Identifying Watermarked Large Language Models
Machine Learning · Submitted 29 May 2023
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Notably, watermarked LLMs tend to produce distributions that diverge qualitatively and identifiably from standard models.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
We consider the emerging problem of identifying the presence and use of watermarking schemes in widely used, publicly hosted, closed source large language models (LLMs). We introduce a suite of baseline algorithms for identifying watermarks in LLMs that rely on analyzing distributions of output tokens and logits generated by watermarked and unmarked LLMs. Notably, watermarked LLMs tend to produce distributions that diverge qualitatively and identifiably from standard models. Furthermore, we investigate the identifiability of watermarks at varying strengths and consider the tradeoffs of each of our identification mechanisms with respect to watermarking scenario. Along the way, we formalize the specific problem of identifying watermarks in LLMs, as well as LLM watermarks and watermark detection in general, providing a framework and foundations for studying them.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | We consider the emerging problem of identifying the presence and use of watermarking schemes in widely used, publicly hosted, closed source large language models (LLMs). | background 0.95 | 0.12 | 0.04 | asserted 0.76 |
| s1 | We introduce a suite of baseline algorithms for identifying watermarks in LLMs that rely on analyzing distributions of output tokens and logits generated by watermarked and unmarked LLMs. | contribution 0.84 | 0.94 | 0.13 | asserted 0.99 |
| s2 | Notably, watermarked LLMs tend to produce distributions that diverge qualitatively and identifiably from standard models. | result 0.99 | 0.71 | 0.35 | asserted 0.75 |
| s3 | Furthermore, we investigate the identifiability of watermarks at varying strengths and consider the tradeoffs of each of our identification mechanisms with respect to watermarking scenario. | contribution 0.97 | 0.92 | 0.10 | asserted 0.76 |
| s4 | Along the way, we formalize the specific problem of identifying watermarks in LLMs, as well as LLM watermarks and watermark detection in general, providing a framework and foundations for studying them. | contribution 1.00 | 0.93 | 0.10 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
We consider the emerging problem of identifying the presence and use of watermarking schemes in widely used, publicly hosted, closed source large language models (LLMs). We introduce a suite of baseline algorithms for identifying watermarks in LLMs that rely on analyzing distributions of output tokens and logits generated by watermarked and unmarked LLMs. Notably, watermarked LLMs tend to produce distributions that diverge qualitatively and identifiably from standard models. Furthermore, we investigate the identifiability of watermarks at varying strengths and consider the tradeoffs of each of our identification mechanisms with respect to watermarking scenario. Along the way, we formalize the specific problem of identifying watermarks in LLMs, as well as LLM watermarks and watermark detection in general, providing a framework and foundations for studying them.
Human review
References
How the literature uses it
1 arXiv paper cite this in 1 sentence
Papers citing it
Cites it as background 1 paper
-
This includes the design of benchmark tasks and metrics to test the quality of watermarks [ref] , some with a special focus on text quality [ref] or robustness [ref] .
Optimizing watermarks for large language models · Related work · 0.99
What it cites · 5 sentences without a confident label
Uses the method or tool of 1 paper
-
Consider the watermark $W_s$ from [ref] that generates a pseudorandom number $r ∈[0,1]$ by applying $f_s$ to the previous tokens.
A Watermark for Large Language Models · Watermarks on Large Language Models · 0.91
Cites as support 1 paper
-
Though other watermarks are less sensitive to changes to the text, all known watermarks are vulnerable to attacks that preserve generated text quality while evading detection [ref] .
Can AI-Generated Text be Reliably Detected? · Watermarks on Large Language Models · 0.97
Cites as background 2 papers
-
More recently, [ref] propose a watermarking scheme that minimizes degradation in the quality of generated text, while being efficient to detect in text.
A Watermark for Large Language Models · Related Work · 0.81
-
Generated Text Detection Via Statistical Discrepancies Recent methods such as DetectGPT and GPTZero distinguish between machine-generated and human-written text by analyzing their statistical discrepancies [ref] .
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature · Related Work · 0.97
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.