{"id":"a24f3e08-d097-4034-900b-62506419a97a","arxiv_id":"2411.19563","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Combining acrostic, sensorimotor, and red-green watermark features in an ensemble improves LLM watermark detection after paraphrasing from 49% to 95%.","lead":"This paper combines three text watermarking tricks, hidden acrostics, word-choice biases, and red-green list scoring, into one system that detects AI-written text even after paraphrasing. The combined watermark keeps a 95% detection rate after a paraphrase attack, where the standard red-green method alone drops to 49%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Detection is computed by thresholding the product of p-values at 0.05, but a product of p-values is not a valid p-value; under the null the effective false-positive rate is ~42%, not 5%, so the headline 97.75%/95% rates are not at a controlled significance level.","rationale":"After careful reading, the principal load-bearing assumption is exactly the one the reader identified: the final detection score is treated as a valid p-value. The paper's strongest claim is quantitative, and those numbers only have the advertised meaning if the detection rule maintains a 5% false-positive rate. It does not: the product of p-values has a much larger effective type-I error (≈42% for three independent uniforms; worse with the per-class products). The low empirical human FPR in Table 2 (0.34–2.42%) is a property of this dataset, not a statistical guarantee, and does not validate the test. This is an internal statistical error, not merely a deviation from community norms. Other concerns (missing error bars, unverifiable code commit, limited model coverage) are secondary. The method itself is clearly described, the ablation is systematic, and the relative advantage of the all-three configuration is consistent, so the paper is promising. A conditional verdict requiring a proper meta-analysis or ROC evaluation is exactly right; I do not propose a different verdict.","tokens_in":11681,"tokens_out":16426,"duration_ms":127956,"concrete_test":"Using the paper's code/data (public GitHub link in footnote 1), recompute detection rates from the individual component p-values (per-class sensorimotor, acrostic, red-green) with Fisher's combined test: χ² = −2 Σ ln p_i, with the appropriate degrees of freedom, and threshold at the 0.05 χ² quantile. Then report the ensemble detection rate and the human false-positive rate after paraphrasing. If the ensemble detection rate drops below the claimed >95%, or the human FPR exceeds 5%, the headline result is not robust to a statistically valid combination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 defines final_score = P_acrostic × P_sensorimotor × P_redgreen with detection when final_score < 0.05. This is not a valid combination of p-values. For three independent uniform p-values, P(product ≤ 0.05) = 0.05 · (1 + (−ln 0.05) + (−ln 0.05)^2/2) ≈ 0.424, i.e., a ~42% false-positive rate under the null, not 5%. The sensorimotor term is itself a product of per-class p-values, so the inflation is even larger if the component tests are well calibrated. The reported detection rates (97.75% in Figure 1, >95% in Table 2) are therefore at an unknown and uncontrolled FPR. The relative ordering across configurations may survive a proper meta-analysis because the same flawed rule is applied throughout, but the absolute headline numbers do not mean 'detection at α=0.05' as claimed. The fix is a correct combined test such as Fisher's method (−2 Σ ln p_i compared to χ² with twice the number of tests) or ROC curves with a fixed empirical false-positive rate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-feature watermarking scheme for large language models that combines acrostics, sensorimotor word classes, and the established red-green logit watermark. During generation, logits are modified according to a secret key derived from the previous words and sentences; during detection, per-feature p-values are computed and combined into a final score that is thresholded at 0.05. Experiments on Llama 3.1 8B, Llama 3.2 3B, and Mistral 7B at three watermark strengths report that the three-feature ensemble yields the highest detection rates, including after a paraphrasing attack, and an ablation study examines the effect of text length.","tokens_in":11825,"tokens_out":8624,"duration_ms":80424,"significance":"If the detection statistic were valid, the paper would offer a flexible and reasonably robust multi-feature watermark, with the useful property that a single detection function handles any feature subset. The release of code and data, the multi-model evaluation, and the consistent relative ordering of configurations are strengths. However, the central statistical combination of p-values is not a valid p-value, so the absolute detection rates and the stated alpha=0.05 control are not supported as written. The relative ranking of configurations may survive a corrected analysis, but the headline numbers require re-reporting at a properly calibrated operating point.","major_comments":[{"comment":"The detection rule final_score = Pacrostic × Psensorimotor × Predgreen with detection when final_score < 0.05 is not a valid combination of p-values. For three independent uniform p-values, P(product ≤ 0.05) ≈ 0.424, not 0.05; the additional product over sensorimotor classes (Psensorimotor = ∏_c Psensorimotor,c) makes the inflation worse. Consequently, the detection rates in Figure 1 and Table 2 are reported at an uncontrolled false-positive rate, and the statement in Section 4.1 that α = 0.05 determines statistical significance is incorrect. Please replace the product rule with a valid meta-analytic combination such as Fisher's method applied to the per-feature p-values, or report ROC curves and detection rates at a fixed empirical false-positive rate.","section":"Section 3.2, Algorithm 2 (final_score)"},{"comment":"No calibration or null-distribution analysis is provided for final_score. The low human detection rates in Table 2 are observed on a specific corpus and do not establish control of Type I error at the claimed 0.05 level; they are an empirical outcome rather than a guarantee. The paper should report the distribution of final_score on human-written text and on unwatermarked model text, and specify operating points by empirical false-positive rate. This is load-bearing because the headline rates (97.75% in Figure 1, >95% after paraphrasing in Table 2) are absolute numbers whose meaning depends on a valid statistical test.","section":"Section 4.1, Table 2"},{"comment":"The sentence 'Assuming independence, the final score is...' introduces an independence assumption that is not justified. The acrostic and sensorimotor keys are derived from hashes of preceding words and sentences, and the red-green list is seeded by the previous token, so the three test statistics are plausibly dependent. If the authors switch to Fisher's method, they must either justify independence or use a dependence-robust calibration. At minimum, the paper should report the correlation between the per-feature p-values on the human baseline and discuss the impact on the combined test.","section":"Section 3.2, final_score formula"}],"minor_comments":[{"comment":"The text refers to 'Llama 3.1 3B', but the implementation details and Table 2 refer to Llama 3.2 3B; please correct this inconsistency.","section":"Section 4.2, Table 2"},{"comment":"Figure 1 is described as plotting Z-scores, but Section 3.2 defines a Z-score only for the red-green feature; please define how a combined Z-score is computed for configurations that include acrostic or sensorimotor features.","section":"Figure 1"},{"comment":"It is unclear whether the reported p-values are for a single representative sample or aggregated across samples; please clarify and, if possible, report the corresponding final_score values.","section":"Table 3"},{"comment":"The evaluation would benefit from confidence intervals for the detection rates, since each configuration has only 300–400 samples.","section":"Section 4.1"},{"comment":"The sentence 'We outline four distinct approaches below, each with its own strengths and weaknesses' appears twice in succession; please remove the duplication.","section":"Section 2.3"},{"comment":"The phrase 'large-scale pertaining' should read 'large-scale pre-training'.","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a CS/CL venue, and the statistical flaw is correctable without changing the core experimental setup. The relative ordering of configurations is likely to survive a corrected analysis, but the absolute detection rates and the alpha=0.05 interpretation must be reworked. I would not recommend rejection, provided the authors address the p-value combination and calibration concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core result is an ensemble of three watermark features—acrostic, sensorimotor, and red-green—with a single detection function. The empirical claim is clear and reproducible in structure: across three LLMs and three strength settings, the full ensemble beats every subset, and after a paraphrase attack it holds around 95% detection where red-green alone drops to 49%. That is a practically useful direction, and the paper is honest about trade-offs like perplexity. The ablation by sentence count is a nice touch, and the appendix gives per-feature p-values, which helps the reader see where signal comes from.\n\nThe soft spot is the detection statistic. The final score is the product of three p-values, thresholded at 0.05. That is not a valid p-value. For three independent uniform p-values, the product is below 0.05 about 42% of the time under the null, not 5%. The sensorimotor term is itself a product of per-class p-values, which makes the inflation worse. So the headline 97.75% and 95% numbers are not detection at alpha=0.05 as claimed; they are detection at an unknown, likely high false-positive rate. The relative ordering of configurations could still hold, because the same flawed detector is applied to every configuration, but the absolute numbers should not be trusted until the combination is fixed—e.g., Fisher's method or an ROC analysis at a fixed empirical FPR.\n\nThe other weaknesses are minor. The GitHub link has no commit hash, and there are no error bars on the detection rates. The stylometric features are drawn from the authors' prior work, which is fine but not new. The writing is clear and the method section is detailed enough to reimplement.\n\nOverall: this is a solid empirical paper with a load-bearing statistical bug in the decision rule. It deserves a serious referee, but the referee should require a corrected detection statistic and a re-reporting of the main table before acceptance. The idea is useful, the evaluation is broad, and the flaw is fixable.\n\nRecommendation: send to peer review, with a request for statistical correction and re-analysis.","headline":"Useful empirical ensemble watermark with a real statistical flaw in the detection rule; the ranking of configurations likely survives, but the headline rates are not at a controlled false-positive level.","tokens_in":12414,"tokens_out":549,"would_cite":true,"duration_ms":6505,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes layering acrostic, sensorimotor, and red-green watermarks in LLM output and reports that the combination detects 97.75% of watermarked text, staying above 95% after paraphrasing while red-green alone falls to about 49%.","keywords":["ensemble watermark","large language models","stylometric features","acrostic","sensorimotor norms","red-green watermark","paraphrasing attack","text provenance detection"],"falsifier":"Generate long human-written passages, run the detector with the watermark key, and count how often the product of the three p-values falls below 0.05; if the false-positive rate is far above 5%, the threshold is miscalibrated and the reported detection advantage must be recomputed with a standard p-value combination method.","tokens_in":11369,"feed_emoji":"🛡️","tokens_out":9388,"duration_ms":72543,"temperature":0.7,"pith_summary":"The paper tries to establish that an ensemble watermark for large language model text, combining acrostic first letters, sensorimotor word categories, and the established red-green token watermark, is more detectable and more resilient to paraphrasing than any single feature. On its tests, all three features together reach 97.75% detection on Llama 3.1 8B medium-strength samples and remain above 95% after a paraphrasing attack that rewrites at least 10% of the text, while red-green alone drops to about 49%. The authors care because a programmable, multi-feature watermark can trace generated text back to a model or deployment without retraining or a separate detector. The contribution is the combination scheme, a shared key schedule, and a single detection function that works for any subset of features.","feed_headline":"Three-feature watermark survives rewriting, holds 95% detection","feed_subtitle":"Pairing acrostics and sensory cues with red-green tokens beats any single watermark in every tested setting","key_machinery":"The load-bearing mechanism is a streaming generation loop in which a secret key is updated from the generated text itself: hashing the previous word selects a sensorimotor class, and hashing the lemmatized previous sentence selects an acrostic letter. During generation, the first token of each new sentence is boosted by $\\delta_{acro}$ if it starts with the key letter, later tokens are boosted by $\\delta_{senso}$ if they belong to the key sensorimotor class, and tokens in the pseudorandom green list are boosted by $\\delta_{redgreen}$. At detection, the same key schedule is replayed; the acrostic and sensorimotor features are scored with binomial tails, the red-green feature with a normal-approximation $Z$-score, and the final score is the product of the three p-values, thresholded at $\\alpha=0.05$.","core_discovery":"The paper's central claim is that watermark signals of different kinds can be layered into a single LLM generation by adding per-feature boosts to the raw token scores, and that the layered result is harder to erase than any one signal. In its evaluation, the full ensemble—acrostic first letters, sensorimotor word categories, and the standard red-green token split—gives the highest detection rate in every model and strength setting tested, reaching 97.75% on Llama 3.1 8B medium-strength text and holding above 95% after a paraphrasing attack; the red-green feature alone falls to roughly 49% under the same attack. The authors claim that the same detection routine, which recomputes the key from the previous word and sentence and scores each feature separately, works unchanged for any subset of features.","pith_inferences":["A standard meta-analytic combination of the three p-values (for example, a chi-square combining method) would likely shift the reported detection rates; the raw-product threshold assumes an independence the paper does not test.","The same logit-bias design could absorb other stylometric features—part-of-speech targets, sentence-length constraints, or word-frequency bands—so the ensemble approach is not limited to the three features evaluated.","An attacker who paraphrases mainly at sentence boundaries could specifically erode the acrostic signal while leaving sensorimotor and red-green cues intact, so the reported paraphrase resistance may depend on the attack's span-level behavior."],"forward_implications":["Operators can choose feature subsets by deployment constraint: acrostics add strong detectability with minimal perplexity cost, sensorimotor cues add paraphrase resilience, and red-green anchors the baseline signal.","A single detection function works for every configuration, so a provider can vary the watermark strength or feature mix per user without maintaining separate detectors.","Because the secret key is re-derived from the generated text at both generation and detection, no external model or service is needed to verify a watermark.","The ablation suggests the ensemble reaches high detection with fewer sentences than any single feature, making it usable for shorter texts."],"supporting_citations":[{"why":"Defines the red-green watermark mechanism that the ensemble extends and uses as the single-feature baseline.","marker":"Kirchenbauer et al. (2023)"},{"why":"Supplies the 40,000-word sensorimotor norms and the perceptual/action categories used to key the sensorimotor feature.","marker":"(Lynott et al., 2020)"},{"why":"Establishes acrostic generation as a text task and provides the motivating example for the acrostic feature.","marker":"Stein et al. (2014)"},{"why":"Provides the C4 RealNewsLike human texts for baseline completions and the T5 model for the paraphrasing attack.","marker":"(Raffel et al., 2019)"},{"why":"Supplies Llama 3.1 8B and Llama 3.2 3B, two of the three evaluation models.","marker":"(Dubey et al., 2024)"},{"why":"Supplies Mistral 7B, the third evaluation model.","marker":"(Jiang et al., 2023)"}],"fun_headline_variants":["Layered watermark hits 98% detection, 95% after rewrite","Triple-feature LLM watermark beats single, survives paraphrase","Ensemble of acrostic, sensory, red-green watermark most robust","Watermark trio resists paraphrase: 95% detection after attack"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the three feature tests are independent enough that multiplying their probability values gives a valid single test at the 0.05 threshold; the paper combines them this way without validating that assumption.","fun_headline_variants_meta":{"raw":{"variants":["Layered watermark hits 98% detection, 95% after rewrite","Triple-feature LLM watermark beats single, survives paraphrase","Ensemble of acrostic, sensory, red-green watermark most robust","Watermark trio resists paraphrase: 95% detection after attack"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1224,"prompt_tokens":898,"completion_tokens":326,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":250}},"tokens_in":514,"tokens_out":326,"duration_ms":3578,"temperature":1.0,"reasoning_tokens":250,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:03:31.401981+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate long human-written passages, run the detector with the watermark key, and count how often the product of the three p-values falls below 0.05; if the false-positive rate is far above 5%, the threshold is miscalibrated and the reported detection advantage must be recomputed with a standard p-value combination method.","supporting_citations":[{"cited_title":"A Watermark for Large Language Models","cited_arxiv_id":null,"evidence_quote":"Defines the red-green watermark mechanism that the ensemble extends and uses as the single-feature baseline."},{"cited_title":"Generating acrostics via paraphrasing and heuristic search","cited_arxiv_id":null,"evidence_quote":"Establishes acrostic generation as a text task and provides the motivating example for the acrostic feature."}],"review_version":1}