{"id":"b6068446-b1a2-409c-ac03-c6cc42b9ea1a","arxiv_id":"2507.18171","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Sticky tokens, defined as vocabulary tokens that pull sentence similarity toward the model's mean, are detected across 40 text embedding models and can degrade retrieval and clustering by up to 50%.","lead":"Some tokens, when repeated inside sentences, make text embeddings from very different sentences look similar, a behavior the authors call sticky tokens. The paper introduces a detector, finds 868 such tokens across 40 embedding models, and shows they can cut retrieval accuracy by more than half in some cases.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Definition 1 quantifies over all sentence pairs, but Algorithm 1 validates only pairs with similarity below u; the 868-token count therefore does not establish the formal claim until above-mean pairs are checked.","rationale":"The paper's strongest claim is Definition 1, and Table 1 is presented as a list of tokens satisfying it. The mismatch between the universal quantifier and the filtered validation set is internal to the manuscript, not a disagreement with external consensus, and it is directly load-bearing: the count of 868 and the characterization 'pulls similarity toward u' both depend on it. I considered the token-embedding surrogate for u (Section 3.1, Appendix C) as an alternative; it is a real representativeness question, but it is less decisive because u is at least explicitly defined and the mechanism may still hold empirically. The below-mean restriction, by contrast, means the stated definition is never tested on half of the similarity range. The reader's weakest_assumption identified this as one of two assumptions; I have focused on it alone. A failed full-set check would not destroy the paper's empirical findings but would require weakening Definition 1 to a statement about below-mean pairs or re-running validation, so conditional acceptance with a required check is the right disposition.","tokens_in":31940,"tokens_out":4943,"duration_ms":55944,"concrete_test":"Run Algorithm 1 with n=8 and the paper's per-model epsilon on all 868 validated tokens using the full STS-derived sentence-pair set S rather than the filtered Pf, explicitly including pairs with Sim(s1,s2) >= u; record the number and identity of tokens that fail on at least one above-mean pair. If any headline token (e.g., lucrarea or </s> for sentence-t5-base) fails, the formal definition and the 868 count require revision; if all pass, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The formal definition (Section 3.2) calls t sticky iff for all s1,s2 in S and all insertion operations I, |Sim(s1, I(s2,t,n)) - u| <= epsilon. The detection and validation pipeline, however, only ever tests pairs in Pf = {(s1,s2) | Sim(s1,s2) < u}: Section 4.1 filters to Pf, Section D.1 defines Pf this way, and Algorithm 1 loops only over Pf. A token that pulls below-mean pairs toward u while pushing above-mean pairs away from u is therefore validated as sticky, even though it violates the universal claim. The adaptive threshold (Algorithm 2) compounds this: epsilon is computed as an IQR outlier cutoff over the candidate tokens' own GE(t) values, so it measures whether a token is typical among already-shortlisted candidates rather than whether Definition 1's 'within epsilon of u' bound holds relative to normal tokens. Consequently, the headline counts in Table 1 (868 tokens, e.g., ordinary words like 'exhibiting' and 'pretended' for e5-base) are not established as satisfying Definition 1. The downstream degradation results remain interesting, but they are about inserted tokens, not about tokens proven to satisfy the formal definition.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the notion of \"sticky tokens\" in text embedding models: vocabulary tokens whose repeated insertion into sentences pulls pairwise cosine similarity toward a fixed value u, defined as the mean pairwise similarity of token embeddings. It proposes a formal definition (Definition 1), a detection pipeline called Sticky Token Detector (STD) with sentence-pair filtering, token filtering, sticky scoring, and validation, and applies STD to 40 checkpoints across 14 model families, reporting 868 sticky tokens. The paper further shows that inserting these tokens can degrade MTEB retrieval and clustering performance by up to roughly 50%, and provides an attention-layer analysis suggesting sticky tokens dominate attention patterns. The authors also discuss implications for RAG adversarial attacks and potential mitigation strategies.","tokens_in":32184,"tokens_out":7616,"duration_ms":78022,"significance":"If the claims hold, this is a timely and practically relevant study of tokenization-level vulnerabilities in embedding models, with a broad empirical scope (40 models), a publicly released code repository, and concrete downstream performance evidence. The downstream degradation results and the attention analyses are interesting and likely reproducible. However, the paper's central formal claim is currently not supported by the validation procedure: the formal definition quantifies over all sentence pairs, while detection and validation only use pairs with initial similarity below the mean, and the validation threshold is derived from the candidate tokens themselves. These gaps undermine the 868-token count as evidence for Definition 1, though the core phenomenon and the detection method remain plausible. The value of the paper would increase substantially if the formal definition and the validation were brought into alignment.","major_comments":[{"comment":"Definition 1 (Section 3.2) requires that for all s1, s2 in S and all I in I, |Sim(s1, I(s2,t,n)) - u| <= epsilon. However, the filtering step restricts the sentence-pair set to Pf = {(s1,s2) | Sim(s1,s2) < u}, and Algorithm 1 validates candidates only on Pf. A token that moves below-mean pairs toward u but moves above-mean pairs away from u would pass validation yet violate Definition 1. The empirical observation that sticky tokens have a clear impact on below-mean pairs does not establish the universal claim. Please either extend Algorithm 1 to also check pairs with Sim(s1,s2) >= u, or revise Definition 1 to quantify only over Pf and adjust the abstract and headline counts accordingly. This is load-bearing because the 868-token count is presented as satisfying Definition 1.","section":"Section 4.1 and Algorithm 1"},{"comment":"The constant u in Definition 1 is the mean pairwise cosine similarity of token embeddings, but the definition concerns sentence embeddings. The paper asserts that \"we use token embeddings as a surrogate for text embeddings since both share the same embedding space\" without providing evidence that the mean token-pair similarity equals (or even approximates) the mean sentence-pair similarity for the 40 evaluated models. If these means differ, the formal definition targets a value that is not the natural center of the sentence-similarity distribution, and the detection results would not match the intended phenomenon. Please provide a systematic comparison of u computed from token embeddings versus from sentence embeddings across all models, or justify the surrogate more rigorously, and if the two differ, redefine u accordingly.","section":"Section 3.1 and Appendix D.1"},{"comment":"The validation threshold epsilon is set as Q3 + 1.5*IQR of the GE(t) values computed over the candidate tokens C themselves. This is circular: a candidate token is validated as sticky if its deviation from u is small relative to other candidates, not if it satisfies Definition 1's requirement that |Sim(s1, I(s2,t,n)) - u| <= epsilon for a threshold chosen a priori or from normal tokens. The paper's statement that epsilon is \"based on how much change in sentence similarity is considered significant by the model's users\" is not reflected in Algorithm 2. Please anchor epsilon to an external criterion (e.g., the distribution of GE for normal tokens, or a user-specified value), report the resulting thresholds, and re-run the validation with the chosen threshold.","section":"Appendix D.4.1 and Algorithm 2"}],"minor_comments":[{"comment":"The text says \"12 text embedding model families\" but Table 5 lists 14 families, and the abstract says 14; please make the count consistent.","section":"Section 5.1"},{"comment":"The sticky score formula \"SS_I(t) = M+ + αF+ / M− + βF− + Sim(s1,t) + γ\" is ambiguous due to missing parentheses; please clarify the intended fraction and report the values of α, β, and γ used in the experiments.","section":"Section 4.3"},{"comment":"The definition of GE(t) uses k in place of the insertion count n used in Definition 1 and Algorithm 1; please align the notation.","section":"Appendix D.4.1"},{"comment":"The table note says the thresholds are \"derived from the standard deviation, not the variance,\" but Algorithm 2 uses the interquartile range; please correct the note.","section":"Table 7"},{"comment":"There are typos: \"String Dateset\" in the Figure 3 caption and \"Dateset\" in Section 5.1 should both be \"Dataset.\"","section":"Figure 3 and Section 5.1"},{"comment":"In the examples column, tokens are concatenated without clear separators (e.g., \"（,h₂o,[CLS],₂,gambia\"); please add spaces or commas between token examples.","section":"Table 1"},{"comment":"In the pseudocode, epsilon is initialized as an empty set but returned as a scalar; please clean up the initialization and the return type.","section":"Algorithm 2"},{"comment":"The abstract says degradation \"approaches 50%\" while the reported NFCorpus drop is 52.3%; please use \"over 50%\" or quote the exact figure.","section":"Abstract and Section 5.3"},{"comment":"The ablation study refers to the (n=10, k=10) configuration as \"ground truth,\" but this is just a larger hyperparameter setting, not an external ground truth; please rephrase to avoid confusion.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an interesting and timely problem, and the empirical study is broad. The central issue is the mismatch between the universal formal definition and the restricted validation set, together with the self-referential threshold selection. These are fixable within the paper's scope: the authors could either weaken the formal claim to match the validation, or strengthen the validation to match the definition. I would also encourage the authors to verify the token-embedding surrogate for u across all models, as the current single-example justification is thin."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—read this one for the phenomenon, not for the formal wrapper. The core observation is solid: repeat certain tokens (lucrarea, </s>) in one sentence of a pair and the cosine similarity gets dragged toward a central value. The paper is the first to turn that anecdote into a systematic detector and a catalogue of 868 tokens across 40 checkpoints, and the downstream numbers are stark (NFCorpus 28.64 to 13.65 for sentence-t5-base). That part deserves attention.\n\nWhere it gets shaky is the gap between Definition 1 and the validation. Definition 1 says 'for all s1, s2 in S and all I in I'. Algorithm 1 only tests pairs in Pf, i.e., pairs with initial similarity below u. So a token that pulls below-mean pairs toward u, but pushes above-mean pairs away from u, would be validated as sticky even though it violates the universal claim. The paper should either weaken the definition to match the validation set or actually check above-mean pairs. That's a load-bearing gap for the formal claim, though not for the empirical phenomenon.\n\nSecond soft spot: u is defined as the mean pairwise similarity of token embeddings, not sentence embeddings. Appendix C argues from anisotropy, but the formal definition assumes the sentence-level mean equals the token-level mean. Same embedding space is not the same as same distribution. The authors should report per-model u and show it matches the sentence similarity mean on their STS data.\n\nThird: the adaptive epsilon in Algorithm 2 is computed from the candidate tokens' own GE(t) values via IQR. That measures whether a candidate is typical among other candidates, not whether it satisfies a fixed 'within epsilon of u' bound relative to normal tokens. It's circular in a mild but real way. A cleaner validation would set epsilon from the distribution of normal tokens and check candidates against it.\n\nMinor but annoying: alpha and beta in the sticky score are never given values. The repo may contain them, but the paper should say. Also, several validated tokens are ordinary English words (exhibiting, pretended, carpet), which suggests the detector is over-inclusive—worth a sensitivity analysis.\n\nBottom line: the phenomenon is real, the catalogue is a useful resource, and the downstream degradation deserves a serious look. But the formal definition overstates what is actually validated. This should go to peer review with requested revisions: fix the validation set, justify u, report all hyperparameters, and add confidence intervals. I'd bring it to a reading group and would cite it if I work on embedding robustness.","headline":"Real phenomenon, useful catalogue, but the formal definition overstates what the validation actually checks.","tokens_in":32735,"tokens_out":3009,"would_cite":true,"duration_ms":32481,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that certain vocabulary tokens, called sticky tokens, pull sentence-pair cosine similarity toward the model's mean token similarity when inserted repeatedly, and that they are common enough (868 across 40 checkpoints) to…","keywords":["sticky tokens","text embeddings","sentence similarity","tokenization robustness","embedding anisotropy","retrieval degradation","clustering","attention analysis"],"falsifier":"Recompute $u$ from whole-sentence embeddings of a large and diverse corpus rather than from one-token embeddings, and rerun STD on a T5-style embedding model; if many validated tokens change or the surviving tokens no longer satisfy Definition 1 with the new $u$, the token-embedding surrogate is the load-bearing assumption.","tokens_in":31673,"feed_emoji":"🧲","tokens_out":7431,"duration_ms":72309,"temperature":0.7,"pith_summary":"Text embedding models represent sentences as vectors, and applications such as search and clustering compare those vectors by cosine similarity. The paper claims that certain vocabulary tokens, called sticky tokens, act as anchors: repeat them inside one sentence of a pair and the pair's similarity moves toward the model's average token-to-token similarity, regardless of meaning. It formalizes this property, builds a detector called STD, and reports 868 sticky tokens across 40 checkpoints spanning 14 model families. The finding matters because injecting just a few of these tokens can cut retrieval accuracy by more than 50% and visibly distort clustering, so tokenization choices affect the reliability of embedding-based systems.","feed_headline":"868 tokens silently warp sentence embeddings in 40 models","feed_subtitle":"Repeatedly inserting them drags sentence similarity to a fixed mean, and retrieval scores can drop by more than half.","key_machinery":"The load-bearing quantity is Definition 1's anchor $u$, the mean pairwise cosine similarity of one-token embeddings, treated as a fixed point that repeated insertion pushes sentence pairs toward. STD is the machinery that finds tokens with this property at scale: it filters sentence pairs to those below $u$, discards undecodable and unreachable vocabulary entries, shortlists candidates by a sticky score that rewards frequent positive similarity shifts, and validates survivors with an IQR-based threshold $\\varepsilon$.","core_discovery":"On the paper's terms, the discovery is that Definition 1 captures a real and widespread failure mode. For an embedding model $E$, let $u$ be the mean pairwise cosine similarity of its token embeddings; a token $t$ is sticky if, for every sentence pair $(s_1,s_2)$ and every insertion operation $I \\in \\{$prefix, suffix, random$\\}$, $|\\mathrm{Sim}(s_1, I(s_2,t,n)) - u| \\le \\varepsilon$. The Sticky Token Detector operationalizes this definition by keeping only pairs with initial similarity below $u$, removing undecodable and unreachable tokens, ranking the rest by a sticky score, and validating the top 2% against the full definition with an adaptive threshold $\\varepsilon$. Applying the detector to 40 checkpoints yields 868 sticky tokens, frequently special, unused, or multilingual subword fragments, and insertion experiments show downstream degradation, with retrieval on one T5-family model falling from 28.64 to 13.65. Attention analysis shows that these tokens attract disproportionately high attention in later layers, consistent with small anomalies being amplified.","pith_inferences":["An extension the paper does not pursue: sticky-token behavior can be measured from insertion effects alone, so the same detector logic could serve as a black-box probe for embedding services whose vocabularies are hidden.","If the anisotropy conjecture is right, post-processing that whitens or isotropizes the embedding space should shrink or eliminate the sticky-token effect; testing STD on such models would directly check the proposed mechanism.","The mean-pulling behavior suggests a quantitative predictor: tokens whose embeddings lie near the principal direction of the anisotropic cone should be the stickiest, so candidate ranking could be done from token geometry before running the full validation."],"forward_implications":["If sticky tokens are as widespread as reported, embedding-based retrieval and clustering systems can be silently degraded by inputs containing a handful of rare tokens, without any semantic change to the text.","The reported absence of a consistent correlation with model size or vocabulary size implies that scaling models alone will not eliminate the failure; tokenizer design and training-data coverage are the relevant levers.","Because the detection pipeline filters and shortlists before validating, the formal definition can be checked cheaply on new checkpoints rather than by scanning the full vocabulary against all sentence pairs.","The layer-wise attention amplification suggests that interventions to reduce stickiness probably need to act on intermediate representations, not only on the final embedding layer.","Special and unused tokens being overrepresented gives tokenizer designers a concrete set of vocabulary entries to sanitize or re-initialize before fine-tuning."],"supporting_citations":[{"why":"Reports the initial observation that appending certain tokens increases sentence similarity, the phenomenon this work formalizes.","marker":"Kaggle, 2024"},{"why":"Supplies the undecodable and unreachable token categories that STD reuses for its token filter.","marker":"Land and Bartolo, 2024b"},{"why":"Supports using token embeddings as a surrogate for text embeddings in the shared embedding space.","marker":"Gao et al., 2019"},{"why":"Supports the anisotropy argument that motivates the mean-pulling anchor $u$.","marker":"Fuster Baggetto and Fresno, 2022"},{"why":"Grounds the claim that anisotropic embedding spaces have nonzero mean pairwise cosine similarity.","marker":"Ethayarajh, 2019"},{"why":"Provides the benchmark suite used to measure downstream retrieval and clustering degradation.","marker":"Muennighoff et al., 2023"},{"why":"Gives the Sentence-T5 model family used for the motivating example and much of the validation.","marker":"Ni et al., 2021a"}],"fun_headline_variants":["Sticky tokens warp embeddings: 868 found across 40 models","New detector finds 868 tokens that break sentence embeddings","Inserting certain tokens skews similarity scores in text models","Sticky Token Detector: 868 anomalous tokens degrade retrieval by 50%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The definition assumes the mean similarity of one-token embeddings is the anchor that real sentence pairs are pulled toward, so if single-token embeddings misrepresent sentence-level geometry, the detector targets the wrong quantity.","fun_headline_variants_meta":{"raw":{"variants":["Sticky tokens warp embeddings: 868 found across 40 models","New detector finds 868 tokens that break sentence embeddings","Inserting certain tokens skews similarity scores in text models","Sticky Token Detector: 868 anomalous tokens degrade retrieval by 50%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1263,"prompt_tokens":985,"completion_tokens":278,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":205}},"tokens_in":601,"tokens_out":278,"duration_ms":3422,"temperature":1.0,"reasoning_tokens":205,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:38:19.377950+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute $u$ from whole-sentence embeddings of a large and diverse corpus rather than from one-token embeddings, and rerun STD on a T5-style embedding model; if many validated tokens change or the surviving tokens no longer satisfy Definition 1 with the new $u$, the token-embedding surrogate is the load-bearing assumption.","supporting_citations":[],"review_version":1}