Machine Learning · arXiv:2611.17035 v1 · submitted 12 Nov 2026 · 29 references
Prevalence Bounds for Edited Watermarked Text: Power Envelopes and Calibration Sensitivity
The paper says
“Watermark detectors decide whether one document came from a watermarked language model. Auditors often need a different number: the share of a corpus that did.”
more of the abstract
We study this share for Gumbel-max and green-list (KGW) watermarks when every document may have been edited by an unknown amount. Without restrictions on the edits, the upper end of the identified set is 1. Combining a known least-favourable next-token distribution with a sequential coupling, we derive a power envelope indexed by three budgets: a retention floor, an entropy budget, and a budget for in-place substitutions, which for Gumbel-max make edited positions look less watermarked than human text. The envelope gives an identified set that is sharp for the alternatives it dominates, finite-sample bounds on the realized share of a fixed corpus with either the exact null or a null calibrated on an independent human sample, and confidence sequences for streams. In Gumbel-max simulations where the envelope binds, the bounds cover, and the threshold that maximizes the gap between envelope and null is generally as tight as twelve thresholds. On text from OPT-1.3B and Qwen3-1.7B, 12,000 fresh human documents scored under three keys show no robust departure from the exact null, and all bounds covered in resampling trials from finite pools of documents with random edits, substitutions and paraphrases; at a share of 0.2 the exact-null bound is about [0.14,0.38]. These results are fragile in two ways we make precise. LLM-paraphrased KGW pools fall below the envelope, yet at 2000 documents the bounds still covered in these trials. A simulated null error of 0.01, comparable to the largest deviation we observed at our thresholds, is likewise harmless at 2000 documents but breaks exact-null coverage at 100,000; a null calibrated on an independent human sample keeps coverage there, but needs a large reference sample to stay narrow.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: major revision.
A mechanism-derived power envelope that actually delivers finite-sample prevalence bounds for edited watermarks; the theory holds, the honesty about limits is exemplary, and the KGW-envelope stress-test concern dissolves on inspection. This line is wrong
6 major objections · 16 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 29 references say.
The paper says, at §3.1, Lemma 2 (KGW part)
The review did not quote this passage.
Pith says: Objection 1 of 6 · major
The KGW envelope (Proposition 2) relies on a mean-only bound for the green probability at a substituted position (Lemma 2). The envelope then assumes that the pivot is Bernoulli with probability at least ¯μ, which is a stochastic dominance claim.
more of objection 1
However, Lemma 2 only provides a lower bound on the mean (green probability), not on the entire distribution. For the stochastic dominance argument (Lemma 3) to hold, one needs that the distribution function of the substituted pivot is pointwise ≤ that of Bernoulli(¯μ). The paper does not prove this; it only proves that the mean is at least ¯μ. This is a gap in the theoretical foundation for the KGW envelope when σmax > 0. The paper acknowledges that 'the KGW paraphrase pools fall below the envelope at moderate thresholds' (abstract, §6), but the issue is not just empirical: the envelope itself may not be valid under the stated assumptions. The authors should either prove the required stochastic dominance or state the envelope as a heuristic bound that is not guaranteed to be conservative.
The paper says, at §3.1, Lemma 2 (KGW part)
The review did not quote this passage.
Pith says: Objection 2 of 6 · major
The proof of Lemma 2 for KGW uses a pairing argument and claims that the posterior odds of the replacement token being green are at least gsub(q). However, the derivation assumes that the emission probability under the list in A is at least 1/(1+cq) times that under its partner.
more of objection 2
This inequality is not fully justified: it relies on the fact that the green mass of the list in A exceeds that of its partner by at most q, but the emission probability is a nonlinear function of the green mass (φ(S) = e^b S/(1+cS)). The paper states that 'the emission probability under the list in A is at least 1/(1+cq) times that under its partner', but this requires a proof that φ(S_A) / φ(S_B) ≥ 1/(1+cq) given S_A - S_B ≤ q. This is not obvious and is not proved. The Monte Carlo check (sim/substitution.py) is not a proof. This weakens the theoretical support for the KGW envelope.
The paper says, at §3.2, Theorem 1
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.91):
“Two processes also cannot show that the relaxed class is nearly lossless in general.”
Pith says: Objection 3 of 6 · major
The sharpness claim in Theorem 1 is with respect to the relaxed class Grel_n, not the actual mechanism class A(εmin, δ). The paper acknowledges this in §3.2 and Appendix B, but the practical relevance of sharpness is limited because the relaxed class is larger.
more of objection 3
The paper attempts to quantify the loss via a linear programming diagnostic (Appendix C) but notes that 'two processes also cannot show that the relaxed class is nearly lossless in general' (line b575). This is a significant caveat: the bounds may be wider than necessary for the true mechanism class. The authors should discuss the implications for the interpretation of the bounds: they are valid but potentially conservative, and the degree of conservatism is unknown in general.
The paper says, at §4.1, Conservative envelope
The review did not quote this passage.
Pith says: Objection 4 of 6 · major
The conservative envelope h_low is computed using a lattice approximation and floating-point tolerance. The paper states that 'the floating-point error is measured, not bounded' and that 'at n = 100, 200, 300, the envelope with Δs = 0.01 is conservative by at most 0.024 relative to a lattice four times finer.'
more of objection 4
This is a numerical check, not a rigorous guarantee. For longer documents (n up to 1000) the check was not performed. The paper subtracts a tolerance of 10^{-8} and clips to [0,1], but this does not provide a formal guarantee that h_low ≤ h pointwise. Given that the bounds are finite-sample and rely on h_low being a lower bound, this is a concern. The authors should either provide a rigorous bound on the discretization error or state that the envelope is approximate and the bounds are valid only up to numerical error.
The paper says, at §5, Table 2
The review did not quote this passage.
Pith says: Objection 5 of 6 · major
Table 2 shows that when the assumption on δ is violated (top probability 0.9 with assumed δ=0.2, i.e., maxP ≤ 0.8), coverage drops to 0.00. This is a load-bearing issue: the envelope is not robust to violations of the entropy budget.
more of objection 5
The paper acknowledges this in Remark 1 and §6, but the practical implication is that the bounds are only valid if the user can guarantee that max_w P_t(w) ≤ 1-δ at every scored position. The paper shows that in real models, 10-19% of positions exceed this threshold (Table 6), so the per-position condition is routinely violated. The paper argues that the document-level condition may still hold, but this is not proved. The sensitivity analysis in Table 2 is for a specific violation (deterministic remainder), but other violations may also cause undercoverage. The authors should discuss the practical difficulty of verifying the entropy budget and the consequences of its violation.
The paper says, at §6, Table 6 and surrounding text
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.94):
“Coverage at N = 2000 therefore does not validate the envelope; violations that persist in the population can matter as the sampling slack shrinks.”
Pith says: Objection 6 of 6 · major
The paper reports that for KGW paraphrase pools, the normalized ratio r_s(t) falls below 1 in every stratum, down to 0.39 for OPT with KGW. This indicates that the envelope is violated for these pools. Yet the bounds still covered at N=2000.
more of objection 6
The paper explains this by noting that the sampling slack (β_s) can mask violations. This is a crucial point: coverage in moderate-sized corpora does not validate the envelope. The paper should emphasize more strongly that the bounds are only guaranteed under the assumptions, and that empirical coverage in experiments does not imply the assumptions hold. The current discussion in §7 ('What the slack hides') is good, but it should be elevated to a major caveat in the abstract or introduction.
16 minor objections
The paper says, at Abstract
Pith says: minor objection 1 of 16
The abstract states 'the threshold that maximizes the gap between envelope and null is generally as tight as twelve thresholds.' This is unclear: does 'as tight as twelve thresholds' mean that using twelve thresholds is no better than using the single best threshold? Please rephrase for clarity.
The paper says, at §1, Introduction
Pith says: minor objection 2 of 16
The phrase 'the upper end of the identified set is 1' appears twice (abstract and §1). It is correct but could be stated more precisely: the upper end is 1 without restrictions on the alternative.
show 14 more
The paper says, at §2, Equation (1)
Pith says: minor objection 3 of 16
The notation F_s,it is used but not defined until Lemma 2. Consider defining it earlier or adding a forward reference.
The paper says, at §3.1, Lemma 2
Pith says: minor objection 4 of 16
The Gumbel-max part of Lemma 2 gives F_sub(x;q) = x^{1/q} + x(1 - x^{1/q-1})/(1-q). Please verify the expression: the second term should be x(1 - x^{1/q-1})/(1-q) or x(1 - x^{1/q-1})/(1-q)? The paper writes 'x(1−x 1/q−1)1−q' which is ambiguous. Please use parentheses.
The paper says, at §3.1, Lemma 2
Pith says: minor objection 5 of 16
The KGW bound g_sub(q) = g / {g + (1-g)(1+cq)} is derived under the assumption that the replacement token is drawn independently of the key. The paper should state this assumption explicitly in the lemma statement.
The paper says, at §3.2, Theorem 1
Pith says: minor objection 6 of 16
The proof sketch says 'When F_n ≠ U we have π_{n,L} > 0, so π > 0 throughout'. This is not strictly necessary for the argument; it is fine.
The paper says, at §3.3, Proposition 2
Pith says: minor objection 7 of 16
The expression for Δ_KGW(δ) uses p_{10} = γ|V|(|V| - γ|V|) / (|V|(|V|-1)). This simplifies to γ(1-γ) * |V|/(|V|-1). Please state the simplified form for clarity.
The paper says, at §4.2, Proposition 5
Pith says: minor objection 8 of 16
The bound uses β_s = {N_s log(2KJ/α)/2}^{1/2}. This is from Azuma-Hoeffding. The paper should note that this bound can be improved with a time-uniform or empirical-Bernstein bound, but it is fine for the fixed-N setting.
The paper says, at §4.5, Theorem 2
Pith says: minor objection 9 of 16
The boundary bd_k uses a mixture over λ ~ N(0, 1/ρ). The paper should cite the source of this boundary (Robbins 1970, Howard et al. 2021) more explicitly in the theorem statement.
The paper says, at §5, Table 3
Pith says: minor objection 10 of 16
The column 'maxh−t' is the single threshold maximizing h(t)-t. The paper should state whether this threshold is chosen with knowledge of the true parameters or from the envelope alone. It appears to be from the envelope alone, which is fine.
The paper says, at §5, Table 4
Pith says: minor objection 11 of 16
The table shows coverage for η = +0.005, +0.010, etc. The paper should clarify that η is the deviation from uniformity at t=0.5, and that the distribution is H(t) = t - 4η t(1-t). This is a specific parametric deviation; the results may not generalize to other forms of null deviation.
The paper says, at §6, Table 5
Pith says: minor objection 12 of 16
The table reports average lower and upper ends. The paper should also report the range or standard deviation of the bounds across corpora to give a sense of variability.
The paper says, at §6, 'The earlier undercoverage'
Pith says: minor objection 13 of 16
The paper describes an earlier undercoverage that did not replicate on fresh data. This is important but the explanation is somewhat buried. Consider moving it to a more prominent location or summarizing the lesson in the discussion.
The paper says, at §7, Discussion
Pith says: minor objection 14 of 16
The paper recommends reporting the upper end over a range of budgets. This is good advice. The paper could also recommend reporting the lower end from the Storey-type estimator, which is more robust.
The paper says, at Appendix A, Proof of Lemma 2
Pith says: minor objection 15 of 16
The proof for KGW is sketchy. The paper says 'Exhaustive enumeration over all green lists for |V|=10 and 300 random NTPs confirmed the bound'. This is not a proof. The authors should either provide a rigorous proof or state the bound as a conjecture supported by simulations.
The paper says, at Appendix E, Theorem 3
Pith says: minor objection 16 of 16
The betting confidence sequence assumes that P(Z_i = 1 | F_{i-1}) = π, i.e., the prevalence is constant. This is a stronger assumption than the stream setting of Theorem 2, which allows arbitrary drift. The paper should clarify this difference.
What would settle it
For a fixed KGW configuration (γ=0.5, b=2, large vocabulary), simulate the exact pivot distribution at a substituted position under the worst-case NTP P*=(1−δ,δ,0,...) and the worst-case substitution token (the one with highest probability under P* that is not the emitted token). Compute the exact distribution function of the green indicator at that position, and compare it pointwise to Bernoulli(¯μ) where ¯μ uses the bound g_sub(1−δ). If the true distribution function is pointwise ≤ Bernoulli(¯μ) (i.e., the true green probability is at least ¯μ and the distribution is stochastically dominated), then the envelope is valid. If not, the envelope is invalid for KGW with substitutions. This test can be done analytically or by exhaustive enumeration for small |V| (e.g., |V|=10) and Monte Carlo for larger |V|.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 13 Nov 2026
- Reader holds up
- Skeptic disagrees: says it holds up only with conditions The envelope's strictness condition for substitutions (σmax ≤ εmin δ/(1−δ)) is derived under the assumption that the substitution pivot bound Fsub(x;1−δ) is the worst case, but Lemma 2's KGW bound is not proved to be pointwise maximal; if the true worst-case substitution pivot distribution is…
- Editor wrote the verdict above
- Referee major revision: 6 major objections, 16 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The paper delivers what it promises: a mechanism-derived power envelope that turns three user-chosen budgets (εmin, σmax, δ) into finite-sample bounds on the share of watermarked documents in a corpus, with clean theory and honest empirical work. I read the proofs and the central claims hold up. The genuinely new piece is Lemma 2: an in-place substitution leaves the replacement under the original key draw, so for Gumbel-max edited positions look less watermarked than human text, not merely null — which corrects the usual hybrid-text model in a conservative direction and is why the substitution budget enters the envelope at all.
I do not think the stress-test concern about the KGW envelope survives reading. For a Bernoulli pivot the distribution function is determined by the success probability, so a lower bound on the conditional green probability is exactly a pointwise upper bound on the conditional distribution function — precisely Lemma 3's hypothesis — and the count then dominates Bin(n, ¯µ) by the standard quantile coupling. The KGW envelope with substitutions stands on the same footing as the Gumbel-max one, given Lemma 2's green-probability bound, whose proof is sketched but structurally right. The real gap is the class, not the envelope: LLM paraphrasing is outside A(εmin, δ), and the paper's Table 6 shows KGW paraphrase pools fall below the envelope at moderate thresholds. The paper owns this and warns that coverage at N = 2000 is too weak to validate the envelope.
Soft spots, in proportion. The Gumbel envelope's validity rests on a floating-point tolerance that is measured, not bounded (10−8); a purist referee will ask for interval arithmetic. Monotonicity of h in n is numerical, not proven, and the stratum-min envelope relies on it. Both are minor. Lemma 2's KGW bound deserves full detail in the journal version. The bounds are wide in the honest regime ([0.14, 0.38] at share 0.2), the envelope is informative only when σmax ≤ εmin δ/(1−δ), and the budgets cannot be estimated from the audited corpus, so the paper's advice to report over a budget range is right but limits the bite. The exact null is load-bearing at scale: their own simulation shows a null error of 0.01 breaks coverage at 100,000 documents, and the calibrated null is the right fix.
Credit where due: 12,000 fresh human documents, three keys, the earlier Qwen3 undercoverage reported rather than buried, the "what the slack hides" decomposition, and code plus data released for everything. Careful, reproducible work. The audience is statisticians working on two-groups partial identification and key holders who want to run real audits.
Send it to serious peer review; accept with minor revision. Expand the proof sketches, address the floating-point caveat, and the envelope, the identification theorem, and the calibration analysis all stand.
The strongest objection
The paper's central identification result (Theorem 1) and all downstream bounds rely on the power envelope h(t;n,εmin,δ) or h_KGW(t;n). For KGW with substitutions, the envelope uses a lower bound on the green probability at a substituted position: g_sub(q) ≥ g_sub(1−δ) (Lemma 2). However, Lemma 2 only proves that the green probability of a substituted token is at least g_sub(q), and that g_sub(q) is decreasing in q. It does not prove that this bound is pointwise maximal in the sense that the entire pivot distribution function is stochastically dominated by a worst-case distribution. For Gumbel-max, Lemma 2 gives the exact distribution function F_sub(x;q) and proves it is pointwise ≥ x, so the envelope uses a genuine worst-case distribution. For KGW, the envelope only uses a bound on the mean (green probability), not on the full distribution. Proposition 2 then constructs h_KGW by assuming that the pivot is Bernoulli with probability at least ¯μ, which is a mean-only bound. If the true distribution of a substituted KGW pivot is not stochastically dominated by Bernoulli(¯μ) (i.e., if its distribution function is not pointwise ≤ that of Bernoulli(¯μ)), then the stochastic dominance argument (Lemma 3) fails, and the envelope may not be valid. The paper does not address this gap. The KGW envelope is therefore on weaker theoretical ground than the Gumbel-max envelope when σmax > 0. The paper acknowledges that 'the KGW paraphrase pools fall below the envelope at moderate thresholds' (Section 6), which is consistent with this gap being real. The load-bearing concern is that the KGW envelope with substitutions may not be a valid lower bound on the alternative distribution, potentially invalidating the upper bound of the identified set for KGW when σmax > 0.
The reader's weakest_assumption was the exact null (Assumption 1). While that is indeed critical, the paper studies its sensitivity extensively (Section 5, Table 4) and provides a calibrated null alternative (Proposition 6). The reader's verdict of ACCEPT with high confidence seems to accept that the null sensitivity is adequately addressed. My concern is different: it targets the validity of the KGW envelope itself when substitutions are allowed (σmax > 0). The paper's Lemma 2 for KGW only bounds the mean green probability, not the full distribution. Proposition 2 then uses this mean bound to construct a stochastic dominance argument via Lemma 3, but Lemma 3 requires pointwise dominance of the distribution function, not just of the mean. For Gumbel-max, Lemma 2 provides the exact distribution function F_sub, so the envelope is on solid ground. For KGW, the paper does not prove that the pivot distribution at a substituted position is stochastically dominated by Bernoulli(¯μ). This is a gap in the proof of Proposition 2 for the case σmax > 0. The paper's own experiments show that KGW paraphrase pools fall below the envelope (Table 6), which is consistent with this gap being practically relevant. The paper acknowledges this as a limitation (Section 7: 'LLM paraphrasing is outside that model'), but the gap is more fundamental: even within the edit model (1), the KGW envelope with substitutions may not be valid. The concrete test I propose would settle whether this gap is real or can be closed. If the test shows the envelope is valid, then the paper's claims for KGW with substitutions are supported. If not, the KGW results with σmax > 0 are unreliable, and the paper should either restrict σmax=0 for KGW or derive a proper distributional bound. I therefore recommend CONDITIONAL acceptance, contingent on resolving this gap. I disagree with the reader's assessment that the weakest assumption is the exact null; the KGW envelope gap is more load-bearing because it is not studied in the paper and could invalidate a core theoretical contribution.
The referee's report
If the results hold, the paper makes a significant contribution to the statistical theory of watermark auditing. It moves beyond per-document detection to corpus-level prevalence estimation, a problem of practical importance for providers and auditors. The envelope is derived from the watermark mechanism and interpretable budgets, and the bounds are finite-sample and non-asymptotic. The paper also provides a thorough sensitivity analysis, including null calibration and the impact of assumption violations. The code and experiments are reproducible. The work is likely to be influential in the watermarking community and in applied statistics for partial identification.
Pith's reading of the paper (a machine's summary, not the authors')
Watermark detectors answer one text at a time; this paper asks what share of a whole corpus came from the watermarked model after users edited the documents. Its central claim is that this share is only partially identified — without edit restrictions the upper end of any honest interval is 1 — but the watermark mechanism itself supplies a power envelope that, under three budgets (a retention floor, an entropy cap, and a budget for in-place substitutions), yields finite-sample bounds on the realized share of a fixed corpus. A key discovery is that a substituted position keeps a key draw under which the replacement token lost, so Gumbel-max edits make edited text look less watermarked than human text rather than merely null. If the paper is right, a key-holding auditor can report an interval such as $[0.14, 0.38]$ when the true share is 0.2 on real-model text, monitor streams with anytime-valid sequences, and know precisely where the guarantee breaks: a null deviation of 0.01 is invisible at 2,000 documents yet destroys exact-null coverage at 100,000.
What carries the argument. The load-bearing object is the power envelope $h(t;n,\varepsilon_{\min},\delta)$, a pointwise lower bound on the p-value distribution of any watermarked document whose edits respect the three budgets, defined as $\mathbb{P}(\bar T_n \ge \bar\Gamma_n^{-1}(t))$ where $\bar T_n$ sums $-\log(1-\bar Y_t)$ for independent draws $\bar Y_t$ from the worst-case pivot distribution $\bar F(x)=(1-\varepsilon_{\min}-\sigma_{\max})x+\varepsilon_{\min}F^*(x;\delta)+\sigma_{\max}F_{\mathrm{sub}}(x;1-\delta)$. It is assembled from three parts: the least-favourable two-point next-token distribution $P^*=(1-\delta,\delta,0,\ldots)$ that maximizes the single-pivot distribution under the entropy cap (Lemma 1), the substitution lemma $F_{\mathrm{sub}}(x)\ge x$ showing in-place replacements subtract signal for Gumbel-max (Lemma 2), and a sequential coupling that lifts position-wise stochastic dominance to the full-document statistic (Lemma 3). The envelope converts per-position budgets into a restriction on the alternative in the two-groups mixture, so that Theorem 1's upper endpoint $\inf_t (F_n(t)-t)/(h_n(t)-t)$ is below 1 and sharp over the relaxed class. For inference the envelope is evaluated conservatively by lattice truncation and FFT convolution, then fed into martingale bounds (Azuma–Hoeffding for fixed corpora, mixture supermartingales for streams) to produce simultaneous finite-sample statements.
Core claim. The paper claims that the watermark-origin share of a corpus is a partial-identification problem whose identified set can be made informative by translating the watermark mechanism into a document-level power envelope $h(t;n,\varepsilon_{\min},\delta)$, the least probability that an edited watermarked document's p-value falls below $t$ over the mechanism class. Three results carry the argument. Lemma 2 shows that for Gumbel-max an in-place substitution leaves the key draw under which the replacement lost, so substituted pivots stochastically exceed null pivots ($F_{\mathrm{sub}}(x)\ge x$): edited text looks less watermarked than human text, and the usual edit model that treats edited positions as null is optimistic. Theorem 1 then shows the identified set $[\pi_L,\pi_U]$ with $\pi_U=\inf_t (F_n(t)-t)/(h_n(t)-t)$ is sharp over the relaxed class of alternatives that dominate the envelope, for both Gumbel-max and the green-list scheme (closed form in Proposition 2). On the inference side, Propositions 5 and 6 bound the realized share of a fixed, heterogeneous corpus under an exact or reference-calibrated null, and Theorem 2 gives a confidence sequence for drifting streams. The empirical claim is that on real OPT and Qwen3 text the exact null holds approximately (no coherent departure in 12,000 fresh human documents under three keys), the bounds cover in resampling trials with random edits, substitutions and paraphrases, and the fragility is quantitative: LLM-paraphrased KGW pools fall below the envelope, and a 0.01 null error that is harmless at 2,000 documents breaks coverage at 100,000.
The premise it rests on. The load-bearing premise is the exact null together with the three-budget edit model; of the two, the exact null is the one most likely to give way, because a deviation of 0.01 in the human p-value distribution — comparable to the largest the paper's own 12,000-document sample can detect — is absorbed at 2,000 documents but destroys coverage at 100,000.
If the paper is right (Pith's inferences, not the authors')
A key-holding auditor can replace the question 'is this text watermarked?' with 'what fraction of this corpus is?', and answer with an interval — about [0.14, 0.38] at a true share of 0.2 on real-model text under the exact null.
Detection counts with worst-case sensitivity are far wider in the low-power regime (upper ends of 0.56–0.75 versus about 0.38 for the envelope bound), so the envelope, not the null correction, is what makes prevalence statements useful.
Prevalence is unidentifiable when next-token distributions may be point masses (Proposition 3): nearly deterministic text such as code, templates, or lists cannot be audited for watermark origin, whatever the retention assumption.
Corpus size changes the meaning of empirical support: coverage at 2,000 documents says little, because the sampling slack shrinks like $N^{-1/2}$ while a null error or envelope violation does not; large audits need the calibrated null of Proposition 6 and a reference sample large enough to stay narrow.
Text streams with drifting adoption can be monitored at every stopping time by Theorem 2's confidence sequence, with no stationarity assumption and labels chosen adaptively so long as each label is fixed before its document is seen.
If these bounds were deployed on platform-scale corpora, the practical bottleneck would shift to reference-sample design: the calibrated null requires reference documents from the same population as the audited corpus, and the paper leaves open how to build such a sample when a corpus mixes domains.
Because Gumbel-max substitutions make edited text look less watermarked than human text, an adversarial editor who knows the budgets could edit at rates just inside the tolerance to keep the share unidentifiable; the paper's assumption that edits are made without regard to tokens is the natural target for an adversarial stress test.
A testable extension suggested by the KGW paraphrase failure is to replace the in-place edit restriction with a semantic-distance restriction on rewrites and check whether a corresponding envelope recovers the power that the paraphrase pools lose.
For instruction-tuned or low-temperature models the entropy budget $\delta$ should bind harder, since the paper itself notes that post-trained models produce lower-entropy text; running its per-stratum ratio check $r_s(t)$ on such models' own generations would show where the envelope starts to fail before any prevalence claim is made.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Watermark detectors decide whether one document came from a watermarked language model. Auditors often need a different number: the share of a corpus that did. We study this share for Gumbel-max and green-list (KGW) watermarks when every document may have been edited by an unknown amount. Without restrictions on the edits, the upper end of the identified set is 1. Combining a known least-favourable next-token distribution with a sequential coupling, we derive a power envelope indexed by three budgets: a retention floor, an entropy budget, and a budget for in-place substitutions, which for Gumbel-max make edited positions look less watermarked than human text. The envelope gives an identified set that is sharp for the alternatives it dominates, finite-sample bounds on the realized share of a fixed corpus with either the exact null or a null calibrated on an independent human sample, and confidence sequences for streams. In Gumbel-max simulations where the envelope binds, the bounds cover, and the threshold that maximizes the gap between envelope and null is generally as tight as twelve thresholds. On text from OPT-1.3B and Qwen3-1.7B, 12,000 fresh human documents scored under three keys show no robust departure from the exact null, and all bounds covered in resampling trials from finite pools of documents with random edits, substitutions and paraphrases; at a share of 0.2 the exact-null bound is about [0.14,0.38]. These results are fragile in two ways we make precise. LLM-paraphrased KGW pools fall below the envelope, yet at 2000 documents the bounds still covered in these trials. A simulated null error of 0.01, comparable to the largest deviation we observed at our thresholds, is likewise harmless at 2000 documents but breaks exact-null coverage at 100,000; a null calibrated on an independent human sample keeps coverage there, but needs a large reference sample to stay narrow.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-13
The authors' main claim, as the classifier picked it out 0.65
“On text from OPT-1.3B and Qwen3-1.7B, 12,000 fresh human documents scored under three keys show no robust departure from the exact null, and all bounds covered in resampling trials from finite pools of documents with random edits, substitutions and paraphrases; at a share of 0.2 the exact-null bound is about [0.14,0.38].”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.97 · human participants 0.39 · controlled comparison 0.85 · randomized assignment 0.06 · states quantitative result 0.98 · compares with prior results 0.71 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Watermark detectors decide whether one document came from a watermarked language model. | background 1.00 | 0.05 | 0.02 | asserted 1.00 |
| Auditors often need a different number: the share of a corpus that did. | background 0.79 | 0.07 | 0.03 | asserted 0.76 |
| We study this share for Gumbel-max and green-list (KGW) watermarks when every document may have been edited by an unknown amount. | contribution 0.99 | 0.84 | 0.05 | asserted 0.94 |
| Without restrictions on the edits, the upper end of the identified set is 1. | result 0.71 | 0.66 | 0.08 | asserted 1.00 |
| Combining a known least-favourable next-token distribution with a sequential coupling, we derive a power envelope indexed by three budgets: a retention floor, an entropy budget, and a budget for in-place substitutions, which for Gumbel-max make edited positions look less watermarked than human text. | method 0.79 | 0.94 | 0.21 | asserted 1.00 |
| The envelope gives an identified set that is sharp for the alternatives it dominates, finite-sample bounds on the realized share of a fixed corpus with either the exact null or a null calibrated on an independent human sample, and confidence sequences for streams. | contribution 0.53 | 0.94 | 0.27 | asserted 1.00 |
| In Gumbel-max simulations where the envelope binds, the bounds cover, and the threshold that maximizes the gap between envelope and null is generally as tight as twelve thresholds. | result 1.00 | 0.92 | 0.12 | asserted 0.94 |
| On text from OPT-1.3B and Qwen3-1.7B, 12,000 fresh human documents scored under three keys show no robust departure from the exact null, and all bounds covered in resampling trials from finite pools of documents with random edits, substitutions and paraphrases; at a share of 0.2 the exact-null bound is about [0.14,0.38]. | result 1.00 | 0.94 | 0.28 | asserted 1.00 |
| These results are fragile in two ways we make precise. | limitation 0.97 | 0.83 | 0.07 | asserted 0.99 |
| LLM-paraphrased KGW pools fall below the envelope, yet at 2000 documents the bounds still covered in these trials. | result 0.61 | 0.92 | 0.06 | asserted 1.00 |
| A simulated null error of 0.01, comparable to the largest deviation we observed at our thresholds, is likewise harmless at 2000 documents but breaks exact-null coverage at 100,000; a null calibrated on an independent human sample keeps coverage there, but needs a large reference sample to stay narrow. | limitation 0.77 | 0.90 | 0.12 | asserted 1.00 |
What kind of paper the classifier reports this is: theoretical result 0.99 · evidence from proof or derivation 0.65
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 8,523 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (3)
Figures are the authors', reproduced from the paper.
References (29)
- [1] So these n quantities determine the law, and they are exactly what is identified. The bound follows fromE Λ[ε]≥ε min. B Relaxed class versus mechanism class The sharpness in Theorem 1 is with respect to Greln . The class of alternatives actually produced by A(εmin, δ) is the c... on Pith
- [2] drift 0→0.4→0.1 500 1.00 [0.212,0.316], ¯π k = 0.224 periodic bursts 500 1.00 [0.311,0.449], ¯π k = 0.320 i.i.d. 0.2 500 1.00 [0.197,0.296], ¯π k = 0.207 The construction follows the hedged capital process of Waudby-Smith and Ramdas (2024). We use their predictable plug-in bet... on Pith
- [3] URLhttps://arxiv.org/abs/2608.14906. Shuwen Chai and Qiaosen Wang. Sample complexities of estimating Gumbel–Max watermark proportions with and without reduction to pivotal statistics. arXiv preprint arXiv:2607.00224, on PithPith's reviewarXiv
- [4] URLhttps://arxiv.org/abs/2607.00224. Sumanth Dathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejovicova, Jamie Hayes, Nidhi Vyas, Majd Al Merey, Jonah Brown-Cohen, Rudy Bunel, Borja ... on PithPith's reviewarXiv
- [6] doi: 10.1109/WIFS58808. 2023.10374576. Christopher Genovese and Larry Wasserman. A stochastic process approach to false discovery control.The Annals of Statistics, 32(3):1035–1061, on Pithdoi:10.1109/wifs58808
- [8] URL https://arxiv.org/abs/2611.06100. Joel L. Horowitz and Charles F. Manski. Identification and robustness with contaminated and corrupted data.Econometrica, 63(2):281–302, on PithPith's reviewarXiv
- [11] URLhttps://arxiv.org/abs/2602.17608. To appear. Guido W. Imbens and Charles F. Manski. Confidence intervals for partially identified parameters. Econometrica, 72(6):1845–1857, on PitharXiv
- [12] doi: 10.1111/j.1468-0262.2004.00555.x. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. InProceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of Machine L... on Pithdoi:10.1111/j.1468-0262.2004.00555.x
- [13] URL https://proceedings.iclr.cc/paper_files/paper/2024/ hash/d78e9e4316e1714fbb0f20be66f8044c-Abstract-Conference.html. Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. Paraphrasing evades detectors of AI-generated text, but retrieval is an effec... on Pith
- [14] doi: 10.52202/075280-1195. Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion-free watermarks for language models.Transactions on Machine Learning Research, on Pithdoi:10.52202/075280-1195
- [15] ISSN 2835-8856. URLhttps://openreview.net/forum?id=FpaCL1MO2C. Xiang Li, Feng Ruan, Huiyuan Wang, Qi Long, and Weijie J. Su. A statistical framework of watermarks for large language models: Pivot, detection efficiency and optimal rules.The Annals of Statistics, 53(1):322–351, ... on PithPith's reviewarXiv
- [17] URLhttps://arxiv.org/abs/2607.21958. Charles F. Manski.Partial Identification of Probability Distributions. Springer Series in Statistics. Springer, New York, on PithPith's reviewarXiv
- [22] doi: 10.1214/23-STS894. Herbert Robbins. Statistical methods related to the law of the iterated logarithm.The Annals of Mathematical Statistics, 41(5):1397–1409, on Pithdoi:10.1214/23-sts894
- [26] URL https://arxiv.org/abs/2502.17259. John D. Storey. A direct approach to false discovery rates.Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64(3):479–498, on PithPith's reviewarXiv
- [28] URLhttps://arxiv.org/abs/2602.14286. Jean Ville. ´Etude critique de la notion de collectif. Number 3 in Monographies des Probabilit´ es. Gauthier-Villars, Paris, on PithPith's reviewarXiv
- [30] URLhttps://arxiv.org/abs/2412.15115. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong... on PithPith's reviewarXiv
- [31] URL https://arxiv.org/abs/2505.09388. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar... on PithPith's reviewarXiv
- [34] If f = Qi(ai + biπ0) with positive factors and slopes of one sign, then f ′′ = f Pi̸=i′ cici′ ≥ 0 with ci = bi/(ai + biπ0), so each capital process is convex, and so is their average. 30 Table 8: Confidence sequences, N = 20,000 documents, 50 replications, assumed (εmin, δ) = ... on Pith
- [1970] doi: 10.1214/aoms/1177696786. Walter J. Rogan and Beth Gladen. Estimating prevalence from the results of a screening test. American Journal of Epidemiology, 107(1):71–76, on Pithdoi:10.1214/aoms/1177696786
- [1978] doi: 10.1093/oxfordjournals.aje.a112510. Tom Sander, Pierre Fernandez, Alain Durmus, Matthijs Douze, and Teddy Furon. Watermarking makes language models radioactive. InAdvances in Neural Information Processing Systems, volume 37, pages 21079–21113, on Pithdoi:10.1093/oxfordjournals.aje.a112510
show 9 more
- [1995] doi: 10.2307/2951627. Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, nonpara- metric, nonasymptotic confidence sequences.The Annals of Statistics, 49(2):1055–1080, on Pithdoi:10.2307/2951627
- [2003] doi: 10.1007/b97478. Nicolai Meinshausen and John Rice. Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses.The Annals of Statistics, 34(1):373–393, on Pithdoi:10.1007/b97478
- [2006] doi: 10.1214/009053605000000741. Francesca Molinari. Partial identification of probability distributions with misclassified data.Journal of Econometrics, 144(1):81–117, on Pithdoi:10.1214/009053605000000741
- [2008] doi: 10.1016/j.jeconom.2007.12.003. Julien Piet, Chawin Sitawarin, Vivian Fang, Norman Mu, and David Wagner. MARKMyWORDS: Analyzing and evaluating language model watermarks. In2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 68–91, on Pithdoi:10.1016/j.jeconom.2007.12.003
- [2021] doi: 10.1214/20-AOS1991. Baihe Huang, Eric Xu, Kannan Ramchandran, Jiantao Jiao, and Michael I. Jordan. Towards anytime-valid statistical watermarking. InAdvances in Neural Information Processing Systems (NeurIPS), on Pithdoi:10.1214/20-aos1991
- [2022] URLhttps://arxiv.org/abs/2205.01068. A Proofs Proof of Lemma 1.The result is due to Li et al. (2025a, Section 3.2); we give a short self- contained version. Fix x∈ (0, on PithPith's reviewarXiv
- [2023] URL https://simons. berkeley.edu/talks/scott-aaronson-ut-austin-openai-2023-08-17. August 17, on Pith
- [2024] doi: 10.1038/s41586-024-08025-4. Pierre Fernandez, Antoine Chaffin, Karim Tit, Vivien Chappelier, and Teddy Furon. Three bricks to consolidate watermarks for large language models. In2023 IEEE International Workshop on Information Forensics and Security (WIFS), pages 1–6. IEEE, on Pithdoi:10.1038/s41586-024-08025-4
- [2025] doi: 10.1109/SaTML64287.2025. 00012. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of Machine Learning Rese... on Pithdoi:10.1109/satml64287.2025
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment