{"id":"651a2556-74eb-4038-949e-59b4196c6c18","arxiv_id":"2509.06338","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Small single-dimension perturbations to embeddings of high-risk tokens can flip aligned LLM responses from refusal to harmful output, and a search algorithm (SEP) locates such perturbations across models.","lead":"This paper shows that adding a small, carefully chosen shift to the embedding of a dangerous word can make aligned chatbots give harmful answers instead of refusing. It introduces a search-based attack called SEP that found such shifts on six open-source models with high reported success.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 96.43% ASR is measured by the same multi-stage classifier that terminates the SEP search; if that classifier over-labels partially-deviated or benign responses as Bypass, the headline number is inflated and needs independent validation.","rationale":"The paper identifies a plausible and interesting vulnerability: perturbing a single embedding dimension of a dangerous token can shift an aligned model from refusal to harmful-sounding output, and the qualitative examples in Figure 8 support the existence of such a phenomenon. However, the paper's central significance claim is quantitative: SEP achieves 96.43% average ASR and outperforms baselines by about 10%. That claim is not stable under the current evaluation because the same classifier that guides the search also defines success. This is not a purely hypothetical concern: the authors themselves report in Section 7.1 that HarmBench and LlamaGuard classify the same outputs differently, and the multi-stage classifier's 'Bypass' category depends on LlamaGuard's harmfulness judgment. The search probes many candidate perturbations and generations; a 4% false-positive rate on each probe can compound across dozens of queries per test case, so a non-trivial fraction of reported successes may be classifier artifacts rather than genuine jailbreaks. The absence of error bars or repeated-seed reporting further weakens the claim that the attack is reliable. I do not think the correct response is rejection: the qualitative demonstrations, the ablation comparisons, and the threat model are worth taking seriously, and the vulnerability is consistent with prior work on embedding-space attacks. But the headline ASR should be treated as unverified until an independent judge or human annotators confirm a sample of successful outputs. This matches the reader's conditional verdict, so I leave the recommended verdict unchanged.","tokens_in":24128,"tokens_out":4055,"duration_ms":41327,"concrete_test":"Re-run the SEP search on the full 150-query benchmark and save the generated response for every reported success. Have two independent annotators (or a judge model not used during the search, e.g., GPT-4 with a detailed safety rubric) label each response as: (i) directly answers the original malicious query with actionable harmful content, (ii) partially relevant but not actionable, or (iii) benign/refusal. Recompute ASR as human-confirmed successes divided by 150. Also compute a no-attack baseline at temperature 1 with the same number of repeated generations per query. If the human-confirmed ASR is materially below 96.43%, or the no-attack baseline success rate is non-negligible, the headline claim overstates the attack.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim rests on a circular evaluation loop. Algorithm 2 terminates the SEP search whenever the multi-stage classifier C returns 'Bypass' (lines 8, 15, 24), and Section 6.1.5 defines ASR using exactly the same multi-stage classifier. C's final harmfulness decision is Llama-Guard-3-8B, whose reported false-positive rate is 4%; with up to dozens or hundreds of classifier-guided probes per query (Q/TC up to 184.61 on gemma-7B), label noise alone can produce a 'Bypass' even when a human would not judge the response as a directly-responsive jailbreak. The paper itself concedes in Section 7.1 that HarmBench and LlamaGuard disagree on concrete outputs and that this classifier discrepancy affects attack efficiency. Because the same imperfect label both steers the search and defines the success metric, the reported ASR is not yet evidence that a deployment-time user would receive genuinely harmful content. Independent human or alternate-judge validation of the successful outputs is required before accepting the headline number.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies deployment-phase embedding poisoning for aligned open-source LLMs. It reports a qualitative finding: increasing one-dimensional perturbations to the embedding of a dangerous token shifts responses from refusal, through harmful output, to semantic deviation. On this basis it proposes SEP, a gradient-free search framework that identifies a dangerous token with GPT-4o, perturbs a single embedding dimension, and uses a merged exponential/binary/linear search guided by a multi-stage classifier (GCG-based refusal detection, HarmBench-based deviation detection, and Llama-Guard-3-8B harmfulness detection). SEP is evaluated on six open-source aligned LLMs against 150 malicious queries from JailbreakBench and HarmBench, reporting an average ASR of 96.43%, a Q/TC of 41.68, and some robustness to SmoothLLM and enhanced safety finetuning. The paper concludes that model-distribution platforms need embedding-level integrity checks.","tokens_in":24323,"tokens_out":8383,"duration_ms":75568,"significance":"If the phenomenon and the headline numbers hold, the paper identifies a practically relevant attack surface: a model-file hook can alter embeddings at deployment time without changing weights or prompt text, converting aligned open-source models into models that answer malicious queries. The reported linear transition from refusal to harmful output to semantic deviation is a concrete, falsifiable empirical claim, and the breadth across six models is valuable. The paper also provides detailed output examples in Appendix C and a demonstration link, which help the reader assess the phenomenon directly. However, the central success metric is measured by the same multi-stage classifier that guides the attack search, so the 96.43% ASR is not yet independently validated as a measure of genuinely harmful content. In addition, the abstract's claims that benign task performance is preserved and that conventional detection is evaded are not backed by experiments in the evaluation section.","major_comments":[{"comment":"The headline ASR is circularly defined with respect to the classifier that guides the search. Algorithm 2 stops and returns 'Success' whenever the multi-stage classifier C labels a response 'Bypass' (lines 8, 15, 24), and Section 6.1.5 defines ASR using exactly the same multi-stage classifier described in Section 5.3. Since C's final stage is Llama-Guard-3-8B, which the paper reports has a 4% false-positive rate, and since the search can issue dozens to hundreds of classifier-guided probes per query (Q/TC up to 184.61 on gemma-7B), label noise alone can produce a 'Bypass' even when a human would not consider the response a directly responsive jailbreak. The paper's own Section 7.1 concedes that HarmBench and Llama-Guard disagree on concrete outputs. The reported 96.43% ASR therefore measures how often SEP finds a perturbation that C labels harmful, not how often a user receives genuinely harmful content. Please provide an independent evaluation (human annotation, or a second and substantially different judge with adjudication) on a representative sample of successful outputs, report agreement/disagreement rates, and give a conservative ASR that excludes cases where the independent judge disagrees with C.","section":"§6.1.5 and Algorithm 2"},{"comment":"The abstract claims that SEP 'preserv[es] benign task performance and evad[es] conventional detection mechanisms,' but Section 6 contains no benign-task experiment and no detection experiment. The evaluation reports only ASR, Q/TC, perturbation magnitude, temperature effects, search ablations, and two defenses. To support the abstract's claims, please add measurements of benign utility (for example instruction-following or knowledge benchmarks on the same models with the hook installed and without it) and run at least one anomaly detector such as perplexity-based scanning over perturbed versus benign embeddings. If these experiments are outside the intended scope, please temper the abstract and threat-model claims accordingly.","section":"Abstract, §4, and §6"},{"comment":"The SmoothLLM defense evaluation appears to give the attack an advantage by construction. The text says that when attacking each of the ten scrambled copies, 'if SmoothLLM perturbs the danger word within a copy, we refrain from perturbing the embedding vector of that copy.' Because SmoothLLM's randomized perturbation is part of the defense, excluding copies in which the defense perturbs the key token means the attack is tested only on the easiest copies. This makes the conclusion that SmoothLLM does not defend against SEP unreliable. Please evaluate under a protocol that does not exclude defense-perturbed copies, or argue explicitly why those copies are not attackable under the threat model.","section":"§6.1.4 and Table 5"}],"minor_comments":[{"comment":"The caption states 'SEP achieves the highest average ASR (96.10%)' while Section 6.2.1 and the abstract report 96.43%; please align these numbers.","section":"Figure 6 caption"},{"comment":"The update rule 'l_U←β / r_U←β / random' is ambiguous; please specify exactly which classifier outcomes update the left boundary, which update the right boundary, and what 'random' means.","section":"Algorithm 2, lines 17–18"},{"comment":"The classifier description says the final Harmfulness Detect uses Llama-Guard-3-8B because of its 'high accuracy (93.9%) and low false positive rate (4%),' but the derivation of these numbers and the evaluation set used to measure them are not given; please provide the source or a short validation.","section":"Section 5.3"},{"comment":"The dangerous-token detection evaluation reports 85% exact agreement with manual labels and 9% 'same degree of toxicity despite different results,' but no inter-annotator procedure or sample size is described beyond 100 JailbreakBench inputs; please clarify how the manual labels were produced and how disagreements were resolved.","section":"Appendix B"},{"comment":"The empirical study motivating Findings 1–3 uses only 10 questions (one per JailbreakBench category) and in the first experiment only one randomly chosen dimension per prompt; the paper would be stronger if the taxonomy were validated on a larger or more diverse sample.","section":"Section 3.1.1"},{"comment":"The explanation of why the average perturbation magnitude at temperature 0 is sometimes smaller than at temperature 1 is confusing as written; please rewrite it to clarify the selection effect from unsuccessful large perturbations.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The circular-evaluation issue is the main risk to the headline claim. If the authors can provide human-validated ASR numbers on a sample of successful outputs and add the missing benign-task and detection experiments, this could become a strong paper for a security venue. The paper's own Section 7.1 shows awareness of classifier disagreement but does not quantify its impact on ASR, which is exactly the quantity that needs independent validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing: the phenomenon is real, the headline number is not yet credible. Reading the appendix, there are concrete outputs where a single-dimension perturbation on a dangerous token flips Llama-2 from refusal to a xenophobic speech or a bomb-making guide. That makes the core claim—embedding-layer outputs are a viable deployment-phase attack surface—worth engaging seriously. The part I would not trust as reported is the 96.43% ASR, because Algorithm 2 uses classifier C to decide when a Bypass has occurred, and Section 6.1.5 defines ASR using the same classifier. The search is effectively optimizing for the label it is later scored on. With up to 184.61 Q/TC on gemma-7B and a Llama-Guard false-positive rate the authors themselves cite as 4%, label noise alone can manufacture successes. The paper even admits in Section 7.1 that HarmBench and Llama-Guard disagree on concrete outputs.\n\nWhat is genuinely new: the observation of a linear refusal→harmful→deviation structure under monotone perturbation, and the SEP search strategy that exploits it. Prior embedding-space attacks use gradient-based adversarial suffixes; SEP's gradient-free single-dimension search is a different mechanism and is simple enough to be practical. The ablation study on temperature and search strategies is informative, and the authors are honest about classifier dependency in limitations.\n\nSoft spots beyond circularity. Benign task preservation is claimed in the abstract but I did not find an experiment for it. Detection evasion via perplexity or other scanners is claimed but not evaluated. There are no error bars or repeated runs for a stochastic search at temperature 1, so we don't know the variance. Minor: Figure 6 caption says 96.10% while the text says 96.43%.\n\nThe math is simple and fine. Citation pattern looks okay; prior embedding attacks are credited. The problem is the evaluation protocol. This is fixable: sample successful outputs, have humans or a second judge label them, report benign-task performance, and rerun with multiple seeds. I would not accept the headline number until that is done.\n\nBottom line: send it to peer review, but require independent validation. The attack surface is plausible and the examples are concrete; the current quantitative claims just do not support 96.43% as stated.","headline":"The attack phenomenon is real and the examples are concrete, but the 96.43% ASR is inflated by a circular classifier loop and needs independent validation before being trusted.","tokens_in":24883,"tokens_out":2687,"would_cite":true,"duration_ms":25624,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that safety alignment in open-source LLMs can be bypassed by small perturbations to the embedding vectors of tokens that carry harmful intent, without changing weights or prompts, and that a search-based attack (SEP)…","keywords":["embedding poisoning","safety alignment bypass","LLM jailbreak","deployment-phase attack","search-based attack","adversarial embedding perturbation","attack success rate","open-source LLM safety"],"falsifier":"Take the 150 benchmark queries and, for one model (say Llama-2-7B), replay SEP's successful perturbations, then have several independent human annotators judge the complete outputs for harmfulness and for fidelity to the original request; if the human-agreed success rate is far below 96.43%, the reported ASR reflects overfitting to the classifier rather than a genuine safety-alignment bypass. Alternatively, rerun the entire search using human judgment as the only success signal and compare the resulting ASR and query counts.","tokens_in":23899,"feed_emoji":"⚠️","tokens_out":9493,"duration_ms":78430,"temperature":0.7,"pith_summary":"Large language models are shipped with safety alignment so they refuse harmful requests, but this paper claims that the alignment lives partly in the embedding layer, not only in weights or prompts. By adding small numeric offsets to the embedding vectors of the tokens that carry the harmful intent (say, 'bomb' or 'blackmail'), the model can be made to still understand the request yet stop refusing it, converting a refusal into instructions for the harmful act. The paper introduces SEP, a search-based attack that finds such offsets quickly, and reports an average attack success rate of 96.43% across six aligned open-source models, with benign queries unaffected and perplexity-based detection unable to see the perturbation. If the claim is right, any model file that ships with a small hook on the embedding layer can silently undo its safety guarantees, so embedding-level integrity, not just weight and prompt scanning, becomes a security requirement.","feed_headline":"Small embedding shifts disable LLM safety in 96.43% of attacks","feed_subtitle":"No weight or prompt changes needed, only a hook that nudges embeddings of dangerous words.","key_machinery":"The central object is the 'uncertain region'—the narrow interval of perturbation magnitudes between the refusal region and the deviation region, where the model produces 'total harmful' responses—and the claim that this region can be located by search. The machinery that carries the argument is the two-component decomposition of embeddings, the merged search strategy, and the multi-stage classifier. The multi-stage classifier routes responses through a GCG-based refusal detector, a HarmBench-based deviation detector, and Llama-Guard-3-8B as the harmfulness judge, returning one of four labels (Denial, Deviation, Part Deviation, Bypass); the search treats 'Bypass' as success. The merged search first doubles the perturbation magnitude (exponential bounding) to bracket the uncertain region, then bisects it (binary refinement), then scans linearly at a coarse step to find a specific magnitude that produces a Bypass label.","core_discovery":"On the paper's own terms, the central discovery is that the safety alignment of an LLM is not robust to small perturbations in embedding space: as the magnitude of a one-dimensional perturbation on a dangerous token's embedding increases, the model's response moves predictably through three phases—refusal, a window of harmful output, and semantic deviation. The paper names this the 'linear transition' and attributes it to a decomposition of each embedding vector into a 'malicious component' (dimensions that the safety mechanism monitors) and a distinct 'semantic component' (dimensions that carry meaning). Shifting the malicious component out of the model's detection range suppresses refusal, while the semantic component is left intact, so the model still comprehends the harmful instruction and answers it. Building on this, the SEP framework identifies dangerous tokens, samples candidate embedding dimensions, and uses a merged search (exponential bounding, binary refinement, linear probing) guided by a multi-stage classifier to land a perturbation inside that narrow window. The paper reports that this achieves a 96.43% average attack success rate on a 150-query benchmark across six aligned LLMs, outperforming prompt-level and embedding-level baselines, and that the attack persists against SmoothLLM and partially against safety fine-tuning.","pith_inferences":["Editorial inference: If the vulnerability is truly a low-dimensional 'malicious component' direction in embedding space, then defenses can be built without waiting for full integrity checks: a deployment-time monitor could estimate that direction per model, and flag or clip large excursions along it before decoding, which the paper mentions only qualitatively as reverse-mapping and hook detection.","Editorial inference: The same mechanism could apply to models served through APIs that expose logits or allow parameter-efficient fine-tuning: an adversary might recover enough of the embedding geometry to find the uncertain region for a few known dangerous tokens and then transfer those perturbations across models, extending the attack beyond the model-file-distribution scenario the paper studies","Editorial inference: The paper's own classifier-dependence caveat suggests a concrete stress test beyond the reported numbers: if the search's success signal (Llama-Guard/HarmBench 'Bypass') is replaced by human judgement on a held-out subset, the true ASR could be measurably lower; this is a testable extension of the method rather than a claim the paper makes.","Editorial inference: Since the perturbation is applied to the embeddings of already-flagged dangerous tokens, a defensive system that already identifies such tokens (e.g., a content filter) could simultaneously perform a sanitizing step—reverse-mapping the perturbed embeddings back to clean vocabulary tokens—turning the attack's own dependency on token detection into a defense."],"forward_implications":["A distributed model that carries a small hook to perturb the embeddings of dangerous tokens will produce harmful instructions for malicious queries while behaving normally on benign queries; the user sees a 'helpful' model that never appears unaligned in ordinary use.","Standard deployment defenses that inspect weights, prompts, or output perplexity cannot detect the attack, because the perturbation lives between the embedding layer and the transformer layers and is never written into parameters or text.","Safety fine-tuning (ESF-style defenses) can lower the attack success rate in some models (e.g., Llama-3.1 and Qwen2.5 drop by about 40 points) but the paper's data show it does not eliminate the vulnerability, and SmoothLLM does not even reduce it.","Because the successful perturbation needs only a handful of forward passes (Q/TC is below 17 on most models and under 10 on Vicuna and Mistral), an attacker can precompute a corpus of per-query perturbation payloads offline and deploy them with trivial per-query runtime cost.","The reported linear transition structure, if it holds generally, turns safety alignment from a property of the model into a targetable coordinate: any aligned open-source LLM inherits a searchable tipping point where refusal fails but comprehension remains."],"supporting_citations":[{"why":"Supplies the two-component (malicious vs. semantic) view of embedding space that explains why shifting a single direction can suppress refusal without losing understanding.","marker":"[3]"},{"why":"Provides 100 of the 150 benchmark queries across 10 threat categories used to measure attack success rate.","marker":"[6]"},{"why":"Provides Llama-Guard-3-8B, the harmfulness judge that labels 'Bypass' and thereby guides the search and the ASR metric.","marker":"[23]"},{"why":"Supplies 50 additional benchmark queries and the HarmBench deviation/relevance classifier used in the multi-stage pipeline.","marker":"[24]"},{"why":"Supplies the GCG deny-list that the paper adapts for its Refusal Detect stage.","marker":"[46]"},{"why":"The prior embedding-space attack baseline that SEP is compared against and outperforms.","marker":"[30, 31]"},{"why":"The COLD-Attack white-box baseline whose complete-response ASR SEP is compared against.","marker":"[13]"},{"why":"The DRA prompt-level jailbreak baseline whose second-best ASR is the main comparison point.","marker":"[21]"},{"why":"SmoothLLM, a defense the paper tests and claims SEP bypasses without ASR reduction.","marker":"[28]"},{"why":"The ESF safety-finetuning defense that the paper shows reduces but does not eliminate SEP's ASR.","marker":"[5]"}],"fun_headline_variants":["Tiny embedding shifts break LLM safety alignment","96.43% success: embedding tweak evades LLM safeguards","Poisoned embeddings silently bypass LLM safety","Embedding semantic shift tricks six aligned LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 96.43% success rate depends on the assumption that when the automated judge used both to steer the search and to score results says 'harmful,' the output really is harmful—the paper itself notes that the two judges often disagree on the same output.","fun_headline_variants_meta":{"raw":{"variants":["Tiny embedding shifts break LLM safety alignment","96.43% success: embedding tweak evades LLM safeguards","Poisoned embeddings silently bypass LLM safety","Embedding semantic shift tricks six aligned LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000429,"raw_usage":{"total_tokens":2213,"prompt_tokens":987,"completion_tokens":1226,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":1162}},"tokens_in":603,"tokens_out":1226,"duration_ms":9169,"temperature":1.0,"reasoning_tokens":1162,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:17:22.751107+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 150 benchmark queries and, for one model (say Llama-2-7B), replay SEP's successful perturbations, then have several independent human annotators judge the complete outputs for harmfulness and for fidelity to the original request; if the human-agreed success rate is far below 96.43%, the reported ASR reflects overfitting to the classifier rather than a genuine safety-alignment bypass. Alternatively, rerun the entire search using human judgment as the only success signal and compare the resulting ASR and query counts.","supporting_citations":[{"cited_title":"Embedding Poisoning: Bypassing Safety Alignment via Embedding Semantic Shift Conference’17, July 2017, Washington, DC, USA Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong","cited_arxiv_id":null,"evidence_quote":"Provides 100 of the 150 benchmark queries across 10 threat categories used to measure attack success rate."}],"review_version":2}