{"id":"a373306e-d606-4cb8-85ff-ad3cb1d3f14c","arxiv_id":"2507.08020","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ETTA bypasses LLM safety refusals by learning a linear toxicity direction in the embedding space and attenuating it in word embeddings at inference time.","lead":"This paper presents an attack that modifies the internal number-coded meanings of words inside an AI model, causing the model to answer harmful questions it is supposed to refuse. The method, called ETTA, works on downloadable open-source models without retraining them, which makes it a practical security risk for AI supply chains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 100-word, single-template toxicity subspace may not support the claimed full-prompt and cross-architecture generalization; ASR could be driven by prompts with explicit toxic keywords and by an unspecified per-model LT.","rationale":"The reader's CONDITIONAL verdict already targets the transferability of the learned toxicity subspace, and my stress-test agrees with that identification: the paper's own mechanistic evidence is a 100-word, single-template correlation study, while the headline ASR is the only direct evidence for the attack's breadth. The most load-bearing concern is therefore reproducibility with respect to prompt composition and model-specific LT handling. If the no-keyword subset of AdvBench still yields high ASR, the central claim is supported and only documentation/artifact issues remain; if it collapses, the attack's effectiveness is driven by explicit keyword presence, and the stated mechanism of a general linear toxicity subspace is unsupported. The test I propose settles exactly this distinction. I do not accuse the authors of misreporting; the ambiguity in whether LT is 'prelearned' or retrained per target model is an internal-specification issue that should be resolved by reproduction or a clear statement in the paper. Because the reader's verdict already reflects the need for such clarification, I leave the verdict unchanged rather than escalating it.","tokens_in":29851,"tokens_out":4580,"duration_ms":52850,"concrete_test":"On Llama-2-7B-Chat, train LT exactly as in Algorithm 1 using the 100-word vocabulary from Appendix A, then run ETTA on all 520 AdvBench prompts. Split results into (a) prompts containing at least one of the 50 toxic words or multi-word entries from Appendix A and (b) prompts containing none. Report ASR for each subset separately, plus seed-variance error bars over at least 5 LT training runs. If subset (b) has ASR close to the no-attack refusal baseline, the linear-subspace mechanism is not the general bypass claimed; if subset (b) remains high, the generalization claim survives this check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that attenuating a scalar toxicity projection per toxic word bypasses safety alignment across five LLMs. The only evidence for a 'toxicity subspace' is Section 3: 50 toxic and 50 benign single words in one template, embedded in Llama-2-7B-Chat, are linearly separable (97.5% SVM accuracy) and show a threshold near tau = 0.025. Algorithm 1 trains LT on this 100-word set, and Algorithm 2 attenuates only words whose toxicity projection exceeds sigma_tox (line 7). Two load-bearing gaps follow. First, the paper reports no breakdown of the 88.61% AdvBench ASR by whether a prompt contains at least one of the 50 curated toxic words in Appendix A; if the no-keyword subset gives near-baseline ASR, the attack reduces to keyword spotting rather than the claimed general linear-subspace bypass. Second, Section 6.1 calls LT 'prelearned' while Algorithm 1 takes the target model as input; embedding dimensions differ across models (4096 for Llama-2-7B, 3584 for Qwen2.5-7B, 5120 for Vicuna-13B), so either LT is retrained per model, which the text never states explicitly, or it is transferred without a described dimension-matching procedure. Section 7.2's admitted 12.4% ASR variance across random seeds also leaves the headline point estimate unanchored.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ETTA, an embedding-space attack that learns a linear transformation LT mapping word embeddings to a scalar toxicity projection and a semantic residual, attenuates the toxicity projection for words above a threshold, and uses a classifier-LLM-guided binary search to tune the attenuation factor. The attack is evaluated on five open-source instruction-tuned LLMs using AdvBench, with a reported average ASR of 88.61% (77.39% against ESF-tuned defenses), modest drops on TruthfulQA and MMLU, and lower time cost than several baselines. The paper argues that safety alignment in LLMs operates through a linear toxicity subspace in embedding space, so attenuating one scalar projection per word bypasses refusal behavior without fine-tuning.","tokens_in":30161,"tokens_out":3699,"duration_ms":41285,"significance":"If the central claim holds, the paper makes a practically important and conceptually interesting contribution: it suggests that open-source LLMs' safety alignment can be bypassed by a simple, training-free embedding hook, without modifying weights, and that the toxicity signal relevant to refusal is approximately one-dimensional. The evaluation is broad for an attack paper: five diverse models, three defense settings, capability benchmarks, efficiency measurements, ablations of the LLM judge, and an explicit limitation section. The paper also reports a concrete failure-mode analysis and discusses mitigations. However, the evidence for the load-bearing 'linear toxicity subspace' claim rests on a 100-word, single-template study, and the cross-model and held-out generalization of the learned transformation is under-specified; the headline numbers are also unanchored by seed variance. These issues do not invalidate the core idea, but they need to be addressed before the paper's central claim can be accepted.","major_comments":[{"comment":"The paper never states whether LT is retrained for each target model or transferred from Llama-2-7B-Chat. Algorithm 1 takes the target model as input and trains on its embeddings, but Section 6.1 calls the matrix 'prelearned,' and the five evaluated models have different embedding dimensions (4096 for Llama-2-7B, 3584 for Qwen2.5-7B, 5120 for Vicuna-13B). If LT is retrained per model, the text must say so explicitly and report the per-model training cost; if it is transferred, the dimension-matching procedure must be described. This is load-bearing because the cross-model ASR claim depends on which LT is used for each model.","section":"Section 6.1 / Algorithm 1"},{"comment":"The toxicity subspace is learned from 50 toxic and 50 benign single words inserted into one template, and the attack attenuates only words whose toxicity projection exceeds sigma_tox. The paper does not report ASR on the subset of AdvBench prompts that do not contain any of the curated words in Appendix A, nor the fraction of AdvBench prompts that do contain at least one such word. If the no-keyword subset gives near-baseline ASR, the attack reduces to keyword spotting rather than a general linear-subspace bypass. Please add this breakdown.","section":"Section 3 / Appendix A / Section 6.2"},{"comment":"The headline numbers (88.61% average ASR, 11.34-point gain over COLD, 77.39% ASR against ESF) are reported as point estimates with no confidence intervals or seed-level results, while Section 7.2 admits 12.4% ASR variance across random seeds due to LT training dynamics. Without variance information, the claimed margins over baselines and the defense comparisons are not statistically anchored. Report seed-level results or confidence intervals for at least the main effectiveness table and the defense table.","section":"Section 7.2 / Table 3 / Figure 7"},{"comment":"Finding 3 and the refusal threshold tau = 0.025 are derived from the same 100-word dataset used to fit the SVM, and the same SVM distance is then used to generate toxicity labels for training LT. The threshold effect is therefore partly self-referential. Provide a hold-out validation, for example by fitting the SVM on one split of toxic/benign words and measuring refusal versus compliance rates on held-out words or full prompts as a function of signed distance, to show that the threshold generalizes beyond the training set.","section":"Section 3 / Algorithm 1 line 7"}],"minor_comments":[{"comment":"The neutral-word list contains duplicates ('positivity' and 'respect' each appear twice), and several entries such as 'chemical weapon' and 'exploit vulnerabilities' are phrases rather than single words, contradicting the 'single words' description in Section 3.","section":"Appendix A, Table 7"},{"comment":"The manuscript should clarify how the alpha-token concatenated embedding e_conc is mapped back into the original prompt embedding matrix E in Algorithm 2, especially for words that tokenize into multiple tokens; the current description of replacing e_conc_t with e_LT_t in E does not specify the token-level indexing.","section":"Section 5.1, Eq. (10) and Algorithm 2"},{"comment":"The text states that local deployment of Llama-3.2-3B-Instruct 'decreases ASR from 89.7% to 66.08%,' but Table 4 reports 66.08% as the average ASR for that judge while no 89.7% value appears in the table; the source of the 89.7% should be identified or the statement corrected.","section":"Section 7.2 / Table 4"},{"comment":"The abstract reports 88.61% ASR and an 11.34% improvement, while the introduction reports 88.62% ASR and an 11.35% improvement; these numbers should be unified.","section":"Abstract / Introduction"},{"comment":"The claim that ETTA operates 'without requiring access to training data' should be qualified: Algorithm 1 requires a target model and a curated list of toxic/benign words, and the classifier LLM is used at attack time; the statement is only true in the sense of not requiring task-specific training data.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real attack paper worth taking seriously, but the headline 88.61% ASR is a point estimate sitting on a 100-word subspace study that may not support the full-prompt, cross-architecture claims. The ETTA mechanism is genuinely new—word-level toxicity attenuation via a learned linear transform, with a classifier-guided binary search over the attenuation factor. The evaluation is broad (five open-source LLMs, AdvBench, several defenses), and the authors are honest: they report 12.4% ASR variance across seeds, admit semantic drift and repetition failure modes, and include a limitations section.\n\nWhere I push back: the load-bearing \"linear toxicity subspace\" story is built from 50 toxic and 50 benign single words embedded in one template on Llama-2-7B-Chat. From that they infer a general threshold effect and a universal attenuation recipe, then apply it to full prompts and four other architectures. The paper never states clearly whether LT is retrained per target model or transferred. Algorithm 1 takes the target model as input, but Section 6.1 calls it \"prelearned.\" Embedding dimensions differ across models (4096, 3584, 5120), so either it is retrained per model—which is fine, but say so—or there is an unexplained dimension-matching step. That matters for reproducibility.\n\nSecond, there is no breakdown of ASR by whether the prompt contains at least one of the 50 curated toxic words. If the no-keyword subset is near baseline, the attack reduces to keyword spotting, not a general linear-subspace bypass. The stress-test concern lands on this.\n\nThird, Finding 3's threshold is partly self-referential: the same SVM and data used to define the signed distance are used to train LT. The external AdvBench ASR is the real test, and it does support the attack's effectiveness, but the mechanistic claim is overstrong relative to a 100-word, single-template evidence base.\n\nNet: the attack is credible, clearly described, and the empirical coverage is solid for a first cut. This paper deserves peer review, not a desk reject. A serious referee should ask for per-model LT handling, seed-level ASR distributions, a keyword-subset breakdown, and preferably code or artifacts. If those come back clean, it is a solid contribution to adversarial machine learning.","headline":"A credible new embedding attack with an overextended mechanism story; the 88.61% ASR needs per-model LT details and a keyword-subset breakdown before the generalization claims land.","tokens_in":30679,"tokens_out":3136,"would_cite":true,"duration_ms":34812,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Embedding tweak turns off LLM safety refusals 88.6% of the time","keywords":["large language models","safety alignment","embedding poisoning","toxicity attenuation","linear transformation","jailbreak","adversarial attack","refusal behavior"],"falsifier":"Run ETTA on a held-out set of full prompts while retraining the linear transformation $LT$ on words from the target model itself, then compare ASR to the reported numbers; if the attack success rate collapses when $LT$ is trained only on the original 100-word vocabulary and applied to another model, the claimed cross-model transfer is unsupported. Alternatively, measure the signed-distance threshold on full-prompt embeddings directly: if a classifier trained on full prompts does not yield a clean linear separation between refused and compliant responses, the threshold model fails.","tokens_in":29670,"feed_emoji":"🔓","tokens_out":5392,"duration_ms":55309,"temperature":0.7,"pith_summary":"This paper claims that a large language model's safety alignment can be bypassed without fine-tuning or training data by attenuating a single scalar toxicity feature in each word's embedding. The authors show that toxic and benign words are linearly separable in embedding space, and that model refusal follows a threshold on the signed distance to that separating hyperplane. They build ETTA, which learns a linear transformation that projects out this toxicity component, subtracts an attenuation factor, and reconstructs embeddings via a pseudo-inverse. On AdvBench harmful-behavior prompts across five open-source LLMs, they report an 88.61% average attack success rate (ASR), beating the best baseline by 11.34 percentage points, and retaining 77.39% ASR against instruction-tuned safety defenses. If correct, the result means current alignment methods leave a simple geometric vulnerability that can be exploited by anyone with access to the embedding layer.","feed_headline":"Embedding tweak turns off LLM safety refusals 88.6% of the time","feed_subtitle":"A simple linear shift in word embeddings makes five open-source chat models answer harmful prompts—no fine-tuning or training data required.","key_machinery":"The load-bearing object is the linear transformation matrix $LT \\in \\mathbb{R}^{\\alpha d \\times \\alpha d}$ that decomposes an $\\alpha$-token word's embedding $e$ into a one-dimensional toxicity projection $T(e)$ and an $(\\alpha d - 1)$-dimensional semantic residual $R(e)$. It is trained on 50 toxic and 50 benign single words embedded in Llama-2-7B-Chat, with toxicity labels given by signed distance to an SVM hyperplane, using a loss that enforces $T(e) \\approx \\hat{T}$ by mean squared error and preserves pairwise cosine similarities of residuals. At inference, ETTA subtracts an attenuation factor from $T(e)$ and reconstructs with the pseudo-inverse, so only the scalar toxicity component changes; a binary search guided by an LLM judge tunes the attenuation factor between the refusal regime and the digression regime. This machinery turns a geometric observation—linear separability plus a threshold on signed distance—into an operation that requires no gradient access, no weight changes, and no training data beyond the 100-word vocabulary.","core_discovery":"The paper's central claim is that safety alignment in LLMs operates through a low-dimensional toxicity subspace of the embedding space, and that suppressing just that component—while preserving the remaining semantic residual—makes otherwise refused prompts elicit detailed, on-topic harmful responses. Concretely, ETTA learns a linear transformation $LT$ mapping each word embedding into a scalar toxicity projection $T(e)$ and a residual $R(e)$; it trains this transformation so that $T(e)$ matches an SVM distance-to-hyperplane toxicity score while $R(e)$ preserves pairwise cosine similarities. At attack time it identifies toxic words by a threshold, subtracts an attenuation factor $\\mu$ from $T(e)$, reconstructs embeddings with the Moore-Penrose pseudo-inverse, and binary-searches $\\mu$ using an LLM judge to land between the refusal threshold and the semantic-drift threshold. The authors report this yields 88.61% average ASR across Llama-2-7B-Chat, Llama-3.2-3B-Instruct, Qwen2.5-7B-Instruct, Vicuna-13B-v1.5, and Gemma-2-9B-it on AdvBench, with 77.39% ASR against ESF instruction-tuned defenses and 60.15% against SmoothLLM, while dropping only 5.63% on TruthfulQA and 7.77% on MMLU. The mechanism they propose is that refusal is triggered by a threshold comparator on the toxicity projection, so reducing that projection below threshold prevents the model from recognizing malicious intent.","pith_inferences":["If the toxicity subspace is as consistent across models as the per-model results suggest, the same linear transformation may transfer to other open-weight models without retraining, which would make the attack even cheaper; the paper does not clearly state whether $LT$ is retrained for each target model.","The threshold model implies that refusal behavior is a geometric classifier; one testable consequence is that perturbing any input embedding along the learned normal vector should flip refusals on and off predictably, which could let defenders build early-exit safety detectors.","The same linear separability could explain false refusals on benign prompts that contain words near the toxic side of the hyperplane, and might suggest de-biasing embeddings as a fix rather than adding more refusal rules.","Because the attack preserves semantic residuals, it may also work on models with different embedding dimensions if a per-model projection is learned from the same small word vocabulary; this is an empirical question the paper leaves open."],"forward_implications":["Any user with code-level access to an open-source model's embedding layer can package a poisoned model that behaves normally on benign prompts but answers harmful prompts in detail.","Safety fine-tuning that merely shifts the refusal decision boundary, such as ESF with a few hundred examples, does not remove the toxicity subspace, so the attack transfers with 77.39% ASR.","The attack is practical against deployed open-source chat models because it takes about 1.92 minutes per query and does not require training data or model fine-tuning.","Embedding-aware defenses, such as renormalizing or standardizing input embeddings before safety checks, are the natural next line of defense; the paper itself suggests this direction."],"supporting_citations":[{"why":"Supplies Llama-2-7B-Chat, the model used to construct the toxic/benign word vocabulary and to measure the refusal threshold.","marker":"[45]"},{"why":"Supplies the AdvBench benchmark of 520 harmful behaviors that defines the attack success measurement.","marker":"[62]"},{"why":"The gradient-based embedding attack baseline that ETTA must beat in both success rate and semantic coherence.","marker":"[40]"},{"why":"The strongest prompt-level white-box baseline, COLD-Attack, whose 77.27% average ASR is the main comparison point.","marker":"[14]"},{"why":"The instruction-tuned defense ESF, against which ETTA still achieves 77.39% ASR, showing the attack transfers to safety-enhanced models.","marker":"[4]"},{"why":"The randomized perturbation defense SmoothLLM, against which ETTA keeps 60.15% ASR.","marker":"[36]"},{"why":"Supplies the judge-prompt protocol used to score responses for harmfulness in the ASR evaluation.","marker":"[7]"}],"fun_headline_variants":["ETTA attack: shrinking toxicity dims in embeddings bypasses refusals","88.6% jailbreak: embedding shift dodges LLM safeguards","No fine-tuning needed: embedding subspace attack defeats 5 LLMs","ETTA drops embedding toxicity to jailbreak 5 chat models 88.6%","One linear transform in embedding space disables LLM safety 88%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the assumption that a toxicity subspace learned from 50 toxic and 50 benign single words embedded in Llama-2-7B-Chat still cleanly separates full-sentence prompts and transfers to other architectures with different embedding dimensions; if that linear separability does not hold for whole prompts or across models, the high attack success rates would not reproduce.","fun_headline_variants_meta":{"raw":{"variants":["ETTA attack: shrinking toxicity dims in embeddings bypasses refusals","88.6% jailbreak: embedding shift dodges LLM safeguards","No fine-tuning needed: embedding subspace attack defeats 5 LLMs","ETTA drops embedding toxicity to jailbreak 5 chat models 88.6%","One linear transform in embedding space disables LLM safety 88%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001108,"raw_usage":{"total_tokens":4697,"prompt_tokens":1105,"completion_tokens":3592,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":721,"completion_tokens_details":{"reasoning_tokens":3493}},"tokens_in":721,"tokens_out":3592,"duration_ms":27547,"temperature":1.0,"reasoning_tokens":3493,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:21:50.569948+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ETTA on a held-out set of full prompts while retraining the linear transformation $LT$ on words from the target model itself, then compare ASR to the reported numbers; if the attack success rate collapses when $LT$ is trained only on the original 100-word vocabulary and applied to another model, the claimed cross-model transfer is unsupported. Alternatively, measure the signed-distance threshold on full-prompt embeddings directly: if a classifier trained on full prompts does not yield a clean linear separation between refused and compliant responses, the threshold model fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The instruction-tuned defense ESF, against which ETTA still achieves 77.39% ASR, showing the attack transfers to safety-enhanced models."},{"cited_title":"Pappas, and Eric Wong","cited_arxiv_id":null,"evidence_quote":"Supplies the judge-prompt protocol used to score responses for harmfulness in the ASR evaluation."}],"review_version":1}