pith. sign in

arxiv: 2605.30501 · v1 · pith:T65PZRQDnew · submitted 2026-05-28 · 💻 cs.CL

Linear Ensembles Wash Away Watermarks: On the Fragility of Distributional Perturbations in LLMs

Zhihao Wu , Gracia Gong , Qinglin Zhu , Yudong Chen , Runcong Zhao This is my paper

Pith reviewed 2026-06-29 07:33 UTC · model grok-4.3

classification 💻 cs.CL
keywords watermarkingLLMsensemble averagingAI text detectiondistributional perturbationsmodel fragilitystatistical signatures
0
0 comments X

The pith

Averaging the output distributions of a few independent LLMs recovers the original unwatermarked distribution up to a second-order error.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that watermarking works by shifting a model's output probability distribution in a detectable way, but these shifts are independent across different providers. When a user averages the distributions from several models, the shifts cancel and the result stays close to the unwatermarked distribution. This holds in theory with a small error bound and in practice when only three or five models are combined. The finding matters because detection systems rely on those statistical signatures remaining visible in generated text. The authors also supply a practical procedure for carrying out the averaging when the models differ in vocabulary and tokenisation.

Core claim

Averaging output probability distributions from multiple models recovers the unwatermarked distribution with at most a second-order error term. Simple averaging across three to five models suppresses watermark detection z-scores from the range 5-300 down below 2 and reduces true-positive rate at 5 percent false-positive rate to below 50 percent, while also raising output quality and speeding generation.

What carries the argument

Linear ensemble averaging of probability distributions across models

If this is right

  • Detection z-scores fall below the usual threshold of 4 after averaging three models.
  • True-positive rate at 5 percent false-positive rate drops below 50 percent.
  • Output quality rises by 27.5 percent and generation runs six times faster than the strongest single-model baseline.
  • Robust watermark-based detection would require coordination among competing model providers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Users could routinely defeat watermark detection by querying several public models and averaging their token probabilities.
  • Detection methods that assume a single fixed model distribution become unreliable once ensembling is common.
  • Alternative detection approaches would need to operate on properties that survive linear averaging.

Load-bearing premise

The distributional perturbations introduced by different providers' watermarks are independent of one another.

What would settle it

An experiment in which the averaged distribution from several independently watermarked models still produces a detection z-score above the threshold of 4.

Figures

Figures reproduced from arXiv: 2605.30501 by Gracia Gong, Qinglin Zhu, Runcong Zhao, Yudong Chen, Zhihao Wu.

Figure 1
Figure 1. Figure 1: Effect of Linear Ensembles. The ensemble average p¯N (Right) neutralises the independent watermark perturbations pi (Middle), effectively recovering the original consensus p ∗ (Left), which is calculated by averaging the unwatermarked models. and coherence. Frameworks that seek to approximate wa￾termark parameters via “random selection probing” (Chen et al., 2025a) require extensive iterative querying, cau… view at source ↗
Figure 2
Figure 2. Figure 2: WASH Framework Overview. The method ensembles N independent models to neutralise watermarks via probability averaging. To resolve vocabulary mismatches, Fluency-Aware Routing commits to the specialist models (K∗ ) whenever a token falls outside the shared vocabulary intersection (V∩), employing re-tokenisation to synchronise context and ensure semantic alignment across heterogeneous tokenisers. Proof Sketc… view at source ↗
Figure 3
Figure 3. Figure 3: Watermark signal decay under different ensemble configurations. We measure detection strength (z-score) as ensemble size N increases. (a) Fixed base model with N independent watermark keys. (b) N independent models, each with independent watermarks. (c) 3 independent base models sharing the same watermark, with the signals coordinated across models: signal persists, demonstrating that coordination defeats … view at source ↗
Figure 4
Figure 4. Figure 4: Examples of Fluency-Aware Routing on C4 completion and MMLU Law reasoning tasks. The routed spans are shown in bold blue, with the list of routed tokens appended. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_4.png] view at source ↗
read the original abstract

Watermarking embeds statistical signatures in AI-generated text for detection and attribution. We reveal a fundamental vulnerability: when users access multiple models (today's reality), watermarks trivially fail. Watermarks perturb output distributions away from the original, and in competitive markets, these perturbations are typically independent across providers. We theoretically prove that averaging output probability distributions recovers the unwatermarked distribution with up to a second-order error term. Empirically, simply averaging 3-5 models cancels out these perturbations. We introduce WASH (Watermark Attenuation via Statistical Hybridisation), which solves practical challenges in ensemble generation: vocabulary misalignment and tokenisation differences across heterogeneous models. Experiments across six watermarking schemes and three LLMs show that averaging across 3 models suppresses detection z-scores from 5-300 to below 2 (below the detection threshold of 4) and reduces TPR at 5% FPR to below 50%, while improving quality by 27.5% and running 6 times faster than the best baseline on the long sequence generation. Our results suggest that robust AI-text detection via watermarking requires either accepting this fundamental vulnerability or unprecedented coordination among model providers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The paper claims that watermarking in LLMs is fragile to linear ensembles: averaging output probability distributions from multiple models recovers the unwatermarked distribution up to a second-order error term because perturbations are typically independent across providers. It provides a theoretical proof of this averaging argument and empirical results across six watermarking schemes and three LLMs showing that averaging 3-5 models reduces detection z-scores from 5-300 to below 2 (below the threshold of 4), lowers TPR at 5% FPR to below 50%, improves quality by 27.5%, and runs 6x faster than baselines. The paper also introduces WASH to address vocabulary misalignment and tokenization differences in ensemble generation.

Significance. If the central claim holds, the result is significant because it identifies a practical, first-principles vulnerability in distributional watermarking that arises from users accessing multiple models. The work supplies a direct derivation from the definition of expectation, reproducible empirical validation across multiple schemes, and a practical method (WASH) that simultaneously improves generation quality and speed. This has direct implications for whether watermark-based detection can be robust without unprecedented coordination among providers.

major comments (1)
  1. [Abstract] Abstract: the theoretical guarantee that averaging recovers the unwatermarked distribution (up to second-order error) requires the perturbations δ_i to satisfy E[δ_i] ≈ 0 under linear combination. The manuscript asserts this holds because 'in competitive markets, these perturbations are typically independent across providers,' but provides no supporting measurement or argument for why real deployed schemes (different providers, different detection keys, possible shared infrastructure) would produce uncorrelated δ vectors on overlapping vocabularies. This assumption is load-bearing for the first-order bias cancellation and for transferring the controlled-experiment z-score reductions to realistic settings.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive comment on the independence assumption underlying our theoretical result. We respond point-by-point below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the theoretical guarantee that averaging recovers the unwatermarked distribution (up to second-order error) requires the perturbations δ_i to satisfy E[δ_i] ≈ 0 under linear combination. The manuscript asserts this holds because 'in competitive markets, these perturbations are typically independent across providers,' but provides no supporting measurement or argument for why real deployed schemes (different providers, different detection keys, possible shared infrastructure) would produce uncorrelated δ vectors on overlapping vocabularies. This assumption is load-bearing for the first-order bias cancellation and for transferring the controlled-experiment z-score reductions to realistic settings.

    Authors: We agree that the independence of δ_i is a central modeling assumption and that the manuscript would benefit from a more explicit justification. The argument rests on two observations that we will expand in the revision: (1) competing providers implement watermarking with independent detection keys and, in most cases, distinct algorithmic choices (different hash functions, different embedding locations, or different pseudorandom generators), which produces statistically independent perturbations on the shared vocabulary; (2) watermarking is an inference-time post-processing step that does not rely on shared model weights or infrastructure across providers, so any common infrastructure would not induce correlation in the δ vectors. We acknowledge that a direct empirical measurement of cross-provider correlation is impossible without proprietary access and therefore constitutes a limitation rather than a claim we can verify. In the revised manuscript we will (a) move the independence statement from the abstract into a dedicated paragraph in Section 3 with the above reasoning, (b) add an explicit limitations paragraph stating that the result assumes no coordination among providers, and (c) note that any future coordinated watermarking standard would invalidate the attack. This constitutes a partial revision focused on clarification and limitation disclosure rather than new experiments. revision: partial

Circularity Check

0 steps flagged

No circularity; central claim follows from linearity of expectation under stated assumption

full rationale

The paper derives the recovery of the unwatermarked distribution by averaging via the linearity of expectation applied to independent perturbations δ_i, which is a direct first-principles step from the definition of expectation rather than any self-referential definition, fitted parameter, or self-citation chain. The independence of perturbations is asserted as a market premise without being derived from the paper's own results or prior self-citations. No equations reduce a 'prediction' to a fit by construction, no uniqueness theorems are imported from the authors' prior work, and no ansatz is smuggled via citation. The empirical z-score suppression is presented as validation of the theoretical claim rather than its source. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that watermark perturbations from different providers are independent; no free parameters or invented entities are introduced in the abstract.

axioms (1)
  • domain assumption Watermark perturbations are typically independent across providers in competitive markets
    Invoked to justify that averaging cancels the perturbations.

pith-pipeline@v0.9.1-grok · 5754 in / 1093 out tokens · 27476 ms · 2026-06-29T07:33:15.861661+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 8 canonical work pages · 3 internal anchors

  1. [1]

    Harnessing Multiple Large Language Models: A Survey on LLM Ensemble

    Chen, R., Wu, Y ., Guo, J., and Huang, H. De-Mark: Water- mark removal in large language models. InProceedings of the 42nd International Conference on Machine Learn- ing (ICML), 2025a. Chen, Z., Lu, X., Li, J., Chen, P., Li, Z., Sun, K., Luo, Y ., Mao, Q., Li, M., Xiao, L., Yang, D., Huang, X., Ban, Y ., Sun, H., and Yu, P. S. Harnessing multiple large la...

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Context-aware watermark with semantic balanced green- red lists for large language models

    Guo, Y ., Tian, Z., Song, Y ., Liu, T., Ding, L., and Li, D. Context-aware watermark with semantic balanced green- red lists for large language models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 22633–22646,

    2024

  3. [3]

    Hoeffding, Probability inequalities for sums of bounded random variables, Journal of the American Statistical Association 58 (301) (1963) 13–30.doi:10.1080/01621459

    doi: 10.1080/01621459. 1963.10500830. Hu, Z., Chen, L., Wu, X., Wu, Y ., Zhang, H., and Huang, H. Unbiased watermark for large language models. In Proceedings of the International Conference on Learning Representations (ICLR),

    work page doi:10.1080/01621459 1963

  4. [4]

    Toblend: Token-level blending with an ensemble of llms to attack ai-generated text detection.arXiv preprint arXiv:2402.11167,

    Huang, F., Kwak, H., and An, J. Toblend: Token-level blending with an ensemble of llms to attack ai-generated text detection.arXiv preprint arXiv:2402.11167,

    work page arXiv

  5. [5]

    CoRR , volume =

    Kirchenbauer, J., Geiping, J., Wen, Y ., Katz, J., Miers, I., and Goldstein, T. A watermark for large language models. InProceedings of the 40th International Conference on Machine Learning (ICML), pp. 17061–17084, 2023a. Kirchenbauer, J., Geiping, J., Wen, Y ., Shu, M., Saifullah, K., Kong, K., Fernando, K., Saha, A., Goldblum, M., and Goldstein, T. On t...

    work page arXiv

  6. [6]

    Water- marking low-entropy generation for large language mod- els: An unbiased and low-risk method.arXiv preprint arXiv:2405.14604,

    Mao, M., Wei, D., Chen, Z., Fang, X., and Chau, M. Water- marking low-entropy generation for large language mod- els: An unbiased and low-risk method.arXiv preprint arXiv:2405.14604,

    work page arXiv

  7. [8]

    Ministral 3

    URLhttps://arxiv.org/abs/2601.08584. Pan, L., Liu, A., He, Z., Gao, Z., Zhao, X., Lu, Y ., Zhou, B., Liu, S., Hu, X., Wen, L., et al. Markllm: An open-source toolkit for llm watermarking. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 61–71,

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [9]

    No free lunch in llm watermarking: Trade-offs in watermarking design choices.arXiv preprint arXiv:2402.16187,

    Pang, Q., Hu, S., Zheng, W., and Smith, V . No free lunch in llm watermarking: Trade-offs in watermarking design choices.arXiv preprint arXiv:2402.16187,

    work page arXiv

  9. [10]

    Qwen3 Technical Report

    URL https:// arxiv.org/abs/2505.09388. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y ., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21 (140):1–67,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [11]

    SQuAD: 100,000+ questions for machine comprehension of text

    Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. SQuAD: 100,000+ questions for machine comprehension of text. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2383–2392,

    2016

  11. [12]

    C., Ye, Z., Chang, Y ., and Li, Y .-S

    Yu, Y .-C., Kuo, C. C., Ye, Z., Chang, Y ., and Li, Y .-S. Break- ing the ceiling of the LLM community by treating token generation as a classification for ensembling. InFind- ings of the Association for Computational Linguistics: EMNLP 2024, pp. 1826–1839,

    2024

  12. [13]

    11 W ASH: Linear Ensembles Wash Away Watermarks A. Proof of the Main Theorem Theorem A.1(Convergence to Consensus Distribution).Under Assumption 2.2, for any fixed context x, let ¯pN(·|x) = 1 N PN i=1 pi(·|x) be the aggregated distribution. For any δ >0 , with probability at least 1−δ , the ℓ∞ distance between the aggregated distribution and the consensus...

    1963

  13. [14]

    That is, E Varu∼p∗ εi(u, x) |W g ≤η 2 where Varu∼p∗(εi(u, x)) := X u p∗(u|x) εi(u, x)− X v p∗(v|x)εi(v, x) 2 . Define the group consensus distribution for groupgas p† g(v|x) := p∗(v|x) exp(bg(v, x))P u∈V p∗(u|x) exp(bg(u, x)),(12) and the group-size-weighted average¯p†(·|x) := 1 N PM g=1 ngp† g(·|x). The irreducible group bias is defined as B(x) := ¯p†(·|...

    1963

  14. [15]

    negligent

    15 W ASH: Linear Ensembles Wash Away Watermarks Step 3: Bounding the second-order idiosyncratic effect term.We write ∥µW −¯p†∥∞ = sup v∈V 1 N MX g=1 X i∈Gg E[pi(v)|W g]−p † g(v) ≤ 1 N MX g=1 X i∈Gg sup v∈V E[pi(v)|W g]−p † g(v) LetR i(v) :=e eεi(v) −1−eε i(v)andA i :=P u p† g(u)Ri(u). Then, using (17), we have pi(v)−p † g(v) = p† g(v)(1 +eεi(v) +R i(v)) 1...

    2025