{"id":"413e277b-66f8-4668-ab26-da3e4c64050b","arxiv_id":"2411.09359","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SPA identifies and removes backdoor-watermarked embeddings from EaaS responses by exploiting the constant watermark vector added to triggered text, bypassing verification.","lead":"Embedding-as-a-service providers that protect copyright by adding a fixed watermark vector to triggered text can have that protection stripped. This paper's Semantic Perturbation Attack appends semantically distant suffixes, detects the watermark by how tightly embeddings cluster after perturbation, and deletes the watermarked samples, reporting over 95% detection while preserving downstream accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Post-attack p-values for WARDEN remain at 10^-2 on three of four datasets, below the conventional 0.05 threshold, so the paper's claim that SPA renders watermarks ineffective is contradicted by its own Table 1.","rationale":"The most load-bearing concern is not the cross-model transfer assumption identified by the reader, but a direct internal inconsistency between the paper's headline claim and its own experimental table. SPA is claimed to render watermarks ineffective and to bypass all four datasets, yet Table 1 shows that for WARDEN, three of the four post-attack p-values remain at 10^-2, which is below the conventional 0.05 significance threshold used in hypothesis-testing-based watermark verification. Only Enron Spam reaches 10^-1. The residual DeltaCos/DeltaL2 values for WARDEN also remain non-negligible, meaning the watermark signal is still detectable. This is a quantitative falsification of the central claim under standard statistical conventions, and it does not rely on any assumption about how suffix guidance transfers across embedding models. Even if the cross-model transfer assumption is granted, the reported p-values show that WARDEN verification would still succeed on most datasets. The concern is addressable in principle: the authors could recompute exact p-values, state the threshold used by WARDEN, or restrict their claims to EmbMarker. As written, however, the central conclusion is not supported by the evidence provided, so the appropriate verdict moves from CONDITIONAL to REJECT.","tokens_in":17906,"tokens_out":5604,"duration_ms":52221,"concrete_test":"Reproduce the WARDEN experiments for SST2, MIND, and AG News and compute the exact KS-test p-value, DeltaCos, and DeltaL2 on the SPA-purified embedding sets, using the verification threshold from the original WARDEN paper. Report the full p-values instead of rounding to powers of ten. If any p-value falls below the WARDEN verification threshold, the claim that SPA bypasses WARDEN is refuted. If the threshold is instead 10^-4 or lower, the authors must state this explicitly and justify it.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that SPA 'renders watermarks ineffective' (Abstract) and 'effectively bypasses all four datasets' (Section 5.2). The verification numbers in Table 1 contradict this for WARDEN: after SPA, the KS-test p-values on SST2, MIND, and AG News are all 10^-2, which is at or below the standard 0.05 significance level used in watermark verification; only Enron Spam reaches 10^-1. For those three datasets, a p-value near 0.01 still constitutes statistically significant evidence of the watermark, so WARDEN verification would still assert the presence of the watermark. The post-attack DeltaCos/DeltaL2 values for WARDEN are also not near zero (e.g., MIND 0.0280/-0.0561, SST2 -0.0108/0.0216), indicating residual watermark signal. The paper's own success criterion is 'a higher p-value' with 'DeltaCos and DeltaL2 values approaching zero' (Section 5.1); by that criterion, WARDEN results fail on three of four datasets. This is internal to the reported data and does not depend on cross-model transfer assumptions or suffix-guidance implementation details. If the threshold used by WARDEN verification is 0.05, the attack does not bypass WARDEN in the majority of tested settings.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Semantic Perturbation Attack (SPA), a watermark-identification attack against backdoor-based embedding watermarks for Embedding-as-a-Service (EaaS). The authors observe that existing schemes such as EmbMarker and WARDEN inject a constant watermark vector when a trigger token is present, making the injected signal independent of input semantics. SPA appends semantically distant suffixes to each query, measures the tightness of the original and perturbed embeddings via cosine similarity, L2 distance, and a PCA-based score, selects a threshold from the metric distribution, and deletes samples deemed watermarked. The paper reports a TPR above 95% on four datasets for both EmbMarker and WARDEN, claims that watermark verification is rendered ineffective, and discusses possible defense strategies. The manuscript includes a geometric motivation in Section 3, the attack algorithm in Section 4, experiments in Section 5, and ablation studies on perturbation count and watermark ratio.","tokens_in":18056,"tokens_out":5002,"duration_ms":44816,"significance":"If the attack worked as claimed, it would be an important practical result: it would demonstrate that linear trigger-based embedding watermarking can be bypassed by a black-box attacker who has only a small local encoder and query access to the service. The paper's core geometric intuition is plausible, the code is released, and the PCA-based tightness metric is an interesting design choice. The ablation studies across perturbation counts and watermark ratios are useful and show that the proposed mechanism is not purely curve-fitting on one setting. However, the load-bearing claim that SPA 'renders watermarks ineffective' is not supported by the paper's own Table 1 for WARDEN on three of four datasets, and the evaluation has methodological gaps including in-sample threshold selection, missing error bars, and a single victim model. These issues must be resolved before the central claim can be accepted.","major_comments":[{"comment":"The statement in Section 5.2 that SPA 'effectively bypasses all four datasets' is contradicted by Table 1: for WARDEN, the post-attack KS p-values on SST2, MIND, and AG News are all 10^-2, which are at or below the conventional 0.05 significance threshold, and the DeltaCos/DeltaL2 values (e.g., MIND 0.0280/-0.0561, SST2 -0.0108/0.0216) are not near zero. According to the paper's own success criterion in Section 5.1 ('a higher p-value, with DeltaCos and DeltaL2 values approaching zero'), WARDEN verification would still assert the presence of the watermark on these three datasets. This directly undermines the abstract's claim that SPA renders watermarks ineffective. The authors should either modify the attack so that p-values exceed the verification threshold on all datasets, or substantially revise the claim to acknowledge that WARDEN remains effective in most of the tested settings.","section":"Table 1; Section 5.2"},{"comment":"The threshold φ is selected from the PCA-score distribution of the same dataset on which the TPR/FPR values in Table 2 are computed, so the reported TPR/FPR are in-sample numbers rather than predictive performance. An attacker must choose the threshold before observing the attack data, or use a separate validation set, for the reported detection rates to be meaningful. The paper should evaluate SPA with a threshold selected on a held-out split and report the resulting TPR/FPR, or provide the full precision-recall curve with the operating point explicitly marked.","section":"Section 4.4"},{"comment":"Algorithm 1 selects suffixes by computing cosine(Θs(dci), Θs(perbj)) (line 10), i.e., the similarity between the original text embedding and the suffix embedding alone, rather than the similarity between Θs(dci) and Θs(dci + perbj), which is what a semantic perturbation of dci would require. The claim that this procedure 'can effectively conduct semantic perturbation on dci' therefore relies on an unstated proxy relationship, and the assertion in Section 4.2 that 'the differential properties between them are consistent' across models is not supported by any evidence. The paper should either justify the proxy, run the main experiments with Algorithm 2 (Appendix C.3) that actually encodes the concatenated text, or compare Algorithm 1 against random suffix selection to show that the guidance mechanism is what drives the reported AUPRC values.","section":"Algorithm 1; Section 4.2"},{"comment":"All experiments use a single victim encoder, OpenAI's text-embedding-ada-002, and the reported AUPRC/TPR/FPR values are given without error bars or variance measures despite the statement that results are 'the average of multiple experiments'. Because SPA depends critically on the transfer of differential properties from Sentence-BERT to the victim encoder, the absence of any second victim model makes the transferability claim untested. The paper should repeat the main experiments on at least one additional victim encoder, ideally an open-source one, and report standard deviations across repeated runs.","section":"Section 5.1; Table 2"}],"minor_comments":[{"comment":"In the first paragraph of Section 3, 'the watermark siginal' is a typo and should read 'the watermark signal'.","section":"Section 3"},{"comment":"The first sentence of Appendix B contains a duplicated article: 'TThe attacker aims' should be 'The attacker aims'.","section":"Appendix B"},{"comment":"The text in Appendix E states that the largest dataset includes 'over 12,000 data items', but Table 3 and the standard AG News statistics show 120,000 training samples; this number should be corrected.","section":"Table 3; Appendix E"},{"comment":"The reference to Kalpesh et al. lists the authors in a non-standard order ('Krishna Kalpesh, Tomar Gaurav Singh, P Parikh Ankur, Papernot Nicolas, Iyyer Mohit'); the names should be formatted consistently with the rest of the bibliography.","section":"References"},{"comment":"Figure 3 is difficult to parse, particularly the relationship between the 'Purified Embedding Set' and the downstream stolen model; clarifying whether the deleted samples are dropped before or after the stealer training loop would improve readability.","section":"Figure 3; Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the internal contradiction between the abstract's claim that SPA 'renders watermarks ineffective' and Table 1's WARDEN p-values of 10^-2 on three of four datasets. This is a load-bearing problem that requires either stronger attack results or a substantially revised thesis. The in-sample threshold selection and the absence of error bars are also serious given the strength of the claims, and the authors should be required to address both in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe paper's core idea is right: backdoor watermarks for EaaS add a constant vector to triggered embeddings, so appending suffixes changes the semantic part but not the watermark part, making watermarked samples detectable by their tightness under perturbation. That mechanism is clearly explained and the experiments mostly support it. The attack is a real extension over ESSA: suffix search guided by a small local model, a PCA eigenvalue score, and demonstration against multi-watermark WARDEN. The ablation showing PCA score robustness is useful, and the code link is a plus.\n\nThe main problem is that the paper's central claim—that SPA 'renders watermarks ineffective'—is not supported by its own Table 1 for WARDEN. After SPA, the KS-test p-values on SST2, MIND, and AG News remain at 10^-2, which is still below the 0.05 threshold a verifier would use. So WARDEN verification would still assert the watermark's presence on three of four datasets. The paper says 'SPA effectively bypasses all four datasets,' but by its own success criterion (p-value higher, DeltaCos/DeltaL2 near zero), it fails that criterion on the majority of WARDEN cases. This is a load-bearing overstatement, not a cosmetic issue. It is fixable—either the claims need to be scaled back, or the attack needs to push the p-values higher (e.g., more aggressive deletion or a different score).\n\nOther issues are more minor. The threshold in Section 4.4 is selected from the same data used to report TPR/FPR, so those numbers are in-sample. No error bars are given, and all experiments use a single victim model (ada-002). The cross-model transfer assumption for suffix guidance is asserted rather than tested, and Algorithm 2 has an undefined variable in line 11. These are addressable in a revision.\n\nWho is this for? Researchers working on watermarking and model extraction for EaaS. The attack is a credible step forward, and the semantic-independence observation is worth having on record. It deserves peer review, but with a major revision requested: fix the WARDEN claim, add error bars, test at least one more victim model, and either justify or test the transfer assumption.\n\nMy recommendation: send it to review, but tell the authors the headline claim about WARDEN must be reconciled with their own numbers.\n\nBest,\n[Name]","headline":"Solid attack idea with a clear mechanism, but the paper overclaims by calling WARDEN bypassed when three of four datasets still show p<0.05 after the attack.","tokens_in":18745,"tokens_out":2269,"would_cite":true,"duration_ms":20609,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper contends that semantically independent backdoor watermarks in embedding services can be identified and removed by a perturbation-clustering attack, reporting true positive rates above 95% without degrading downstream utility.","keywords":["semantic perturbation attack","embedding-as-a-service watermark","backdoor watermark vulnerability","watermark identification","semantic independence","PCA eigenvalue clustering","suffix perturbation","EaaS copyright protection"],"falsifier":"Use the paper's suffix-ranking procedure with a helper encoder deliberately chosen to be geometrically dissimilar to the victim encoder (for example, a bag-of-words model or a random projection), and check whether the bimodal separation and the reported true positive rates survive; if the separation collapses, the cross-model transfer assumption is what carries the attack.","tokens_in":17580,"feed_emoji":"🔓","tokens_out":13701,"duration_ms":116994,"temperature":0.7,"pith_summary":"Current backdoor watermarking schemes for Embedding-as-a-Service add a constant watermark vector to the embeddings of texts that contain a trigger, regardless of the text's meaning. The paper argues that this semantic independence is a structural vulnerability: appending a suffix to a text changes its semantic embedding but not the fixed watermark component, so watermarked texts under perturbation cluster tightly while unwatermarked texts scatter. From this observation it builds the Semantic Perturbation Attack (SPA), which appends suffixes, measures the spread of the perturbed embeddings, and deletes the suspicious cluster. In experiments across four datasets and two watermark schemes, SPA identifies and removes over 95% of watermarked samples while keeping downstream classification accuracy roughly intact. If the claim holds, providers relying on such backdoor watermarks for copyright enforcement can no longer detect copying, and watermark designs must become semantic-aware.","feed_headline":"Suffix tweaks reveal and delete 95% of embedding watermarks","feed_subtitle":"Constant-vector backdoor watermarks on embedding APIs can be detected and removed without losing downstream accuracy.","key_machinery":"The load-bearing object is the constant watermark vector $e_t$ in the linear mixing formula $e_i \\mapsto \\mathrm{Norm}((1-\\lambda)e_i + \\lambda e_t)$, and its multi-vector counterpart in WARDEN. The semantic independence of $e_t$ is what makes a watermarked embedding set contract under perturbation. The attack's measuring instrument is the tightness score: for each text, $k$ suffix-perturbed embeddings are collected and reduced via PCA, and the sum of the top eigenvalues serves as the score, with a bimodal histogram of scores providing a threshold $\\varphi$. Suffix selection is guided by a small local sentence encoder, which scores each candidate suffix against the original text and keeps the lowest-similarity suffixes on the assumption that differential properties transfer to the black-box victim encoder.","core_discovery":"The paper's central contention is that the watermark signal in existing backdoor schemes is semantically independent: in EmbMarker the watermarked embedding is $e'_i = \\mathrm{Norm}((1-\\lambda)e_i + \\lambda e_t)$ with a fixed vector $e_t$, and WARDEN extends this to a sum of fixed vectors. Because $e_t$ does not depend on the input text, perturbing the text by appending a suffix changes the original component $e_i$ but leaves the watermark component unchanged, so the perturbed embeddings of a watermarked text form a tighter cluster than those of a benign text. SPA exploits this by choosing suffixes that maximize semantic distance, querying the victim service $k$ times per text, and measuring the spread of the resulting embedding set using cosine similarity, L2 distance, or the sum of PCA eigenvalues. A bimodal distribution of these scores yields a data-driven threshold; samples below it are deleted as watermarked. The paper reports that this purification raises verification p-values into the benign range, achieving true positive rates above 95% on four text classification datasets against both EmbMarker and WARDEN.","pith_inferences":["Editorial inference: if any constant-vector watermark is detectable by perturbation clustering, then the same test should be applied to WET's linear-transformation watermark, which also adds a fixed geometric shift; the paper does not report this experiment.","Editorial inference: an obvious stress test is to swap the helper encoder for one with a different architecture and dimensionality; if the attack's success degrades, the transfer premise becomes the critical constraint, and the paper's auxiliary heuristic variant is reported as weaker on two of four datasets.","Editorial inference: an attacker could amortize query cost by clustering texts first and perturbing one representative per cluster, or by reusing a shared set of suffixes; the paper's mechanism would still apply but the per-text query count would drop.","Editorial inference: providers could monitor for clients that issue many near-duplicate queries with appended suffixes, turning SPA's own query signature into a detection signal; the paper discusses defenses only at the watermark-design level."],"forward_implications":["SPA bypasses both single-vector (EmbMarker) and multi-vector (WARDEN) backdoor watermarks without modifying any embeddings, by deleting the samples it identifies as watermarked.","Downstream classification accuracy stays roughly the same after purification, and embedding utility is preserved better than with embedding-modifying attacks such as CSE and PA.","Identification performance improves and stabilizes as the number of perturbations increases, so the attacker faces a query-cost tradeoff rather than a performance ceiling.","Even at low watermark ratios the watermark cluster stays detectable, with PCA AUPRC in the 0.3–0.4 range, so the attack remains relevant to sparse trigger deployments.","The paper's proposed mitigation is a semantic-aware watermarking design in which the injected signal depends on the text content rather than a constant vector."],"supporting_citations":[{"why":"Defines EmbMarker, the single-vector watermark whose linear mixing formula supplies the semantic-independence property that SPA exploits.","marker":"(Peng et al., 2023)"},{"why":"Defines WARDEN multi-vector watermarks and the CSE elimination attack; SPA is evaluated against both.","marker":"(Shetty et al., 2024a)"},{"why":"Defines WET and the paraphrasing attack PA, which serves as a baseline for utility and evasion.","marker":"(Shetty et al., 2024b)"},{"why":"Defines ESSA, the prior watermark identification attack that SPA is compared with and outperforms on multi-watermark schemes.","marker":"(Yang et al., 2024)"},{"why":"Provides the small local sentence encoder that SPA uses for suffix selection and perturbation guidance.","marker":"(Reimers and Gurevych, 2019)"},{"why":"Supplies the general text corpus used as the candidate pool for suffix search.","marker":"(Merity et al., 2016)"},{"why":"Establishes the backdoor-watermark concept that the attacked schemes build on, motivating the constant-trigger signal model.","marker":"(Adi et al., 2018)"}],"fun_headline_variants":["Semantic perturbation attack strips 95% of embedding watermarks","Semantic-independent watermarks are erased by perturbation tests","Fixed-vector backdoor watermarks are vulnerable to semantic attacks","SPA defeats EaaS watermarks with 95% detection and removal"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack assumes that the ranking of suffixes by semantic distance computed with a small local embedding model (384 dimensions) transfers to the black-box victim embedding model (1536 dimensions), so suffixes that perturb the local model most also perturb the victim most.","fun_headline_variants_meta":{"raw":{"variants":["Semantic perturbation attack strips 95% of embedding watermarks","Semantic-independent watermarks are erased by perturbation tests","Fixed-vector backdoor watermarks are vulnerable to semantic attacks","SPA defeats EaaS watermarks with 95% detection and removal"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000844,"raw_usage":{"total_tokens":3677,"prompt_tokens":950,"completion_tokens":2727,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":2656}},"tokens_in":566,"tokens_out":2727,"duration_ms":18014,"temperature":1.0,"reasoning_tokens":2656,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:43:44.981431+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use the paper's suffix-ranking procedure with a helper encoder deliberately chosen to be geometrically dissimilar to the victim encoder (for example, a bag-of-words model or a random projection), and check whether the bimodal separation and the reported true positive rates survive; if the separation collapses, the cross-model transfer assumption is what carries the attack.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines EmbMarker, the single-vector watermark whose linear mixing formula supplies the semantic-independence property that SPA exploits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines ESSA, the prior watermark identification attack that SPA is compared with and outperforms on multi-watermark schemes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the small local sentence encoder that SPA uses for suffix selection and perturbation guidance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the backdoor-watermark concept that the attacked schemes build on, motivating the constant-trigger signal model."}],"review_version":1}