{"id":"d9071ea1-3e70-42eb-83b4-b21f0dff86fa","arxiv_id":"2507.20333","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Compressing LLM internals via FJLT projection or a bottleneck layer disrupts linear safety directions and blocks activation-steering jailbreaks, at some utility cost and with no defense against non-linear attacks.","lead":"This paper shows that larger language models store safety concepts as clean straight-line directions in their internal space, making them easier to hijack with activation-steering jailbreaks. It proposes two fine-tuning methods that shrink this internal space, and demonstrates that they block this attack class on three 7-billion-parameter models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FJLT projects only attention Q/K and Bottleneck reconstructs to full D, so residual-stream steering and probing stay D-dimensional; the dimensionality mechanism is not implemented as claimed.","rationale":"The reader's conditional verdict already requires softening the Section 3.2 inference and tempering the 'curse of dimensionality' framing. My concern sharpens this: the gap is not merely that the Rademacher-to-steering step is asserted; the implemented defenses do not reduce the dimension of the residual-stream space in which ActAdd/Ablation and the linear probes operate. FJLT (Sec. 4.1) projects Q and K inside one attention head; the residual states x^(ell) remain in R^D. Bottleneck (Sec. 4.2) compresses to K and immediately maps back to D, and probes are taken from the last layer, again in R^D. Proposition 1's bound concerns the input dimension of the linear class, which is unchanged for the probe and for the steering-vector add/ablate operations described in App. B.1-B.2. Therefore the theoretical story cannot explain the observed robustness, and the honest empirical claim is that the proposed fine-tuning recipes, which include a projection, reduce ActAdd/Ablation success. The paper is consistent on that narrow claim, has FT controls in Tables 1-2, and discloses GCG failure in App. G.1. A frozen-projection control would settle whether the projection itself (without fine-tuning) removes linear structure; if it does not, the 'blessing and curse of dimensionality' title and the Paradox framing should be substantially revised. I therefore keep the reader's CONDITIONAL recommendation: the empirical method may be acceptable after reframing, but the causal dimensionality claim is currently unsupported.","tokens_in":31482,"tokens_out":12772,"duration_ms":139999,"concrete_test":"Insert the FJLT projection (K=64, head 0) into the frozen Llama2-7B-Chat baseline without any fine-tuning. Measure (i) linear-probe accuracy on last-layer residual-stream activations and (ii) ActAdd attack success using a steering vector computed on the projected model itself. If both remain close to the unmodified baseline, the projection alone does not remove linear steering structure, and the reported defense is an effect of the fine-tuning objective, not of lower-dimensional representations. Also run the same linear probe on the FT-only model from Table 1 to check whether the probe drop is projection-specific.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the defended models represent concepts in a lower-dimensional space, making Proposition 1's O(sqrt(D/N)) bound the explanation for why steering vectors fail. The implemented defenses do not reduce the dimension of the space in which steering and probing actually happen. In the FJLT method (Sec. 4.1), the projection is applied to query and key matrices inside one attention head (Qproj = QPhi, Kproj = KPhi with Phi in R^{D_H x K}); the residual-stream activations x^(ell) remain in R^D, and ActAdd/Ablation (App. B.1-B.2) add or subtract directions in that same D-dimensional residual stream. In the Bottleneck method (Sec. 4.2), x^(ell) is compressed to K and immediately reconstructed to D before layer ell+1; the final-layer representations used for the linear probes in Fig. 5 and App. E are still D-dimensional. Hence the input dimension in Proposition 1 is unchanged for both the probe and the attack, so the Rademacher bound cannot be the reason linear structure disappears. The drop in probe accuracy must instead be attributed to the fine-tuning objectives (Eq. 5 and Eq. 4.2), a confound the paper does not isolate, since Fig. 5 has no FT-only linear-probe condition. The tension is sharpened by FJLT's own design goal: the JL lemma guarantees approximate preservation of inner products and distances (App. A.2-A.3), which would tend to preserve a linearly separable safety direction, not erase it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper hypothesizes that increasing hidden dimensionality in LLMs makes abstract concepts such as safety more linearly separable in activation space, and that this linear structure is exploited by activation-steering jailbreaks (ActAdd and Ablation). The authors support this with PCA visualizations and linear probe accuracy across models of different scales, interpret this as a 'Paradox of Linear Separability', and provide a theoretical argument based on Rademacher complexity bounding the capacity of linear classifiers by O(sqrt(D/N)). They then propose two defenses built on aligned 7B chat/instruct models: (1) FJLT, which projects the query and key matrices of a chosen attention head into a lower dimension K during fine-tuning under a token-wise constrained objective, and (2) Bottleneck, which inserts a linear autoencoder between two layers, compressing to K and reconstructing to D, trained with a refusal dataset plus an anchor utility dataset. On JailbreakBench, AdvBench, and HarmBench, both methods are reported to substantially restore refusal and safety scores after ActAdd and Ablation attacks compared to fine-tuning-only baselines, with mixed but mostly competitive utility preservation. The paper also reports that the FJLT method degrades under GCG attacks and on several utility benchmarks, while the Bottleneck method preserves utility better. A self-acknowledged limitation section discusses dependence on linear separability, data requirements, and attack scope.","tokens_in":31837,"tokens_out":4061,"duration_ms":42402,"significance":"If the causal dimensionality claim held, this would be a valuable explanation of a tension in scaling LLMs: larger hidden dimensions improve capabilities but may make safety alignment more vulnerable to steering attacks. The empirical portion is responsibly executed in several respects: it compares against fine-tuning-only controls, reports 5-run means and standard deviations, evaluates on three benchmarks and three model families, includes ablations over head/layer placement and the alpha parameter, and is accompanied by reproducible code and detailed appendices. The paper also explicitly reports its own negative results, including GCG vulnerability and the Qwen2-7B-Instruct-Bottleneck's weak defense, which strengthens confidence in the reported numbers. However, the central mechanism asserted in the paper—that the defenses work by reducing the dimensionality of the space in which steering and probing operate—is not implemented in that space, and the cross-model evidence for the causal role of hidden dimension is confounded. The empirical defense results may stand, but the interpretation as a 'curse of dimensionality' requires substantially more support or a reframing.","major_comments":[{"comment":"The implemented defenses do not reduce the dimension of the representation space in which steering and probing actually act. The FJLT method projects only the query and key matrices of one attention head (Q_proj = Q Phi, K_proj = K Phi with Phi in R^{D_H x K}); the residual-stream activations x^(ell) remain in R^D, and ActAdd/Ablation (Appendix B.1-B.2) add or subtract directions in that same D-dimensional residual stream. The Bottleneck method compresses x^(ell) to K and reconstructs to D before layer ell+1, so final-layer representations used for the linear probes in Figure 5 and Appendix E are still D-dimensional. Hence the input dimension in Proposition 1 is unchanged for both the probe and the attack, and the Rademacher bound O(sqrt(D/N)) cannot be the mechanism by which linear structure disappears. The drop in probe accuracy must instead be attributed to the fine-tuning objectives (Eq. 5 and Eq. 4.2), a confound that Figure 5 does not isolate because it omits an FT-only linear-probe condition. I request (i) a linear-probe condition on the fine-tuned model without any architectural modification, and (ii) a discussion of why the JL lemma's approximate preservation of inner products and distances (Appendix A.2-A.3) does not imply that a linearly separable safety direction should be approximately preserved in the projected Q/K space rather than erased.","section":"Section 4.1, 4.2 and Appendix A.4, E"},{"comment":"The 'curse of dimensionality' claim rests on a cross-family comparison: the models in Figure 3 differ in architecture, pre-training data, and alignment procedure, not only in hidden dimension. The observed monotone increase in linear probe accuracy could be driven by any of these correlated factors. To substantiate the causal claim that hidden dimension itself drives linear separability, the authors should provide a controlled sweep within one model family (e.g., varying hidden width while keeping architecture, data, and training procedure fixed, or intervening on the effective dimension of the activation space while freezing everything else). Without such a control, the paper should soften the causal language to a correlational observation.","section":"Section 3.1, Figure 3"},{"comment":"The inference from the Rademacher bound to the ineffectiveness of steering vectors is asserted without a derivation. Lower Rademacher complexity of a bounded-norm linear class in R^k does not imply that the particular difference-in-means steering direction used by ActAdd fails; even a low-complexity class can contain a perfect separator if the projected data are separable. Please state and prove a formal claim that relates the success of the ActAdd or Ablation attack to the Rademacher bound, or clearly characterize the additional assumptions (e.g., margin, distribution of the steering direction) needed. As written, the theoretical section does not logically connect Proposition 1 to the empirical defense results.","section":"Section 3.2, paragraph after Eq. (4)"},{"comment":"The paper claims 'significant improvements on both harmful and benign instructions across all models and metrics', but the Qwen2-7B-Instruct-Bottleneck results do not support this. For the ActAdd jailbreak, harmful-instruction refusal increases only from 0.11 (baseline) to 0.23 (Bottleneck), while fine-tuning alone gives 0.08; safety score goes from 0.18 to 0.51. Appendix D.2 shows similarly weak or mixed results on AdvBench and HarmBench for this model (e.g., HarmBench ActAdd safety 0.39 vs. baseline 0.19). Since the paper's conclusion is a general phenomenon across models, the Qwen2 discrepancy needs a dedicated analysis or a more nuanced claim about when the Bottleneck defense works.","section":"Section 5.1.2, Table 2, and Appendix D.2"}],"minor_comments":[{"comment":"The table cells '0 64' and '0 96' in the Head column appear to contain stray characters (perhaps a degree symbol or a formatting artifact); please clean these up.","section":"Section 4.1, Table 1"},{"comment":"The display equation is referred to as 'Eqn. 4.2' in the text but is never assigned a number; number the equations consistently throughout the manuscript.","section":"Section 4.2"},{"comment":"The label 'Compromised!' in the safety panel is informal; consider using a neutral caption such as 'loss of linear separability' to keep the presentation consistent with the rest of the paper.","section":"Section 5.2, Figure 4"},{"comment":"The Gemma-1.1-7B-IT perplexity values are extremely high (baseline 529.05), and the FJLT value of 1923.04 is dismissed as an 'anomaly' without a quantitative explanation; report cross-entropy loss as well as perplexity, or discuss the calibration of these numbers, to make the utility comparisons interpretable.","section":"Appendix D.3, Table 13"},{"comment":"The refusal fine-tuning dataset is inconsistently named Dp, D_P, and 'the dataset'; standardize the notation.","section":"Section 5.1.1 and Appendix B.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical defense results appear real and the appendix is unusually candid about negative results. However, the central theoretical mechanism (dimensionality reduction in the representation space) is not actually realized by the proposed architectures, and the causal claim about hidden dimension rests on a confounded comparison. In my view the paper could be accepted after reframing: present the contributions as 'fine-tuning with dimensionality-reducing inductive biases that disrupt linear safety structure', and provide the FT-only probe and within-family dimension controls needed to separate the effects. If the authors are unwilling to add those controls, the scope of the claims should be narrowed accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful empirical paper with a theory section that doesn't do what the abstract claims. The FJLT Q/K projection and the bottleneck autoencoder are new defensive tools, and on Llama2 and Gemma they convincingly blunt ActAdd and Ablation attacks, with FT controls and honest reporting of failures. But the 'curse of dimensionality' mechanism is not actually implemented.\n\nWhat's new and good: applying FJLT to Q/K inside a single attention head during fine-tuning, and inserting a linear bottleneck after layer 0, are both plausible regularization ideas for making safety less linearly readable. The evaluation is solid: three 7B chat models, three jailbreak benchmarks, refusal and safety scores, perplexity, utility benchmarks, 5-run variance, and ablations on heads, layers, and alpha. They even test GCG and show the defenses don't help (FJLT hurts), which is the kind of negative result most papers bury. The limitations section is unusually candid.\n\nThe soft spots are substantial, though. The theoretical framing is the load-bearing part of the title, and it fails on close reading. FJLT projects Q and K inside one attention head; the residual stream stays in R^D. The bottleneck compresses to K and immediately reconstructs to D; the representations the attacker steers and the linear probes read are still D-dimensional. So Proposition 1, a Rademacher bound on linear classifiers in R^D, doesn't explain why steering vectors fail. The drop in probe accuracy in Figure 5 is more likely a fine-tuning effect, and there's no FT-only probe condition to rule that out. The JL lemma's distance preservation would, if anything, tend to preserve a linear safety direction, so the proposed mechanism is particularly hard to defend.\n\nThe scaling evidence is also weaker than the prose. Figure 3 is a cross-family comparison of different models with different training pipelines; it doesn't isolate hidden dimension. And the headline claim isn't universal: Qwen2-7B-Instruct-Bottleneck only gets refusal from 0.11 to 0.23 under ActAdd, and FJLT degrades utility on SQL/Samsum/GSM8k.\n\nBottom line: cite it for the defensive recipes and the honest negative results, not for the curse-of-dimensionality thesis. If this were a fresh submission, I'd send it to referees and ask for major revision: soften or cut the Rademacher inference, add FT-only probes, and make the scaling claim descriptive rather than causal. It's worth a serious referee, but the current framing overreaches.","headline":"Useful defensive fine-tuning recipes against ActAdd/Ablation, but the dimensionality story doesn't hold up: the projections never reduce the residual-stream dimension where steering actually happens.","tokens_in":32343,"tokens_out":4116,"would_cite":true,"duration_ms":43871,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"High-dimensional LLM hidden states are what let steering jailbreaks find a linear safety direction, and projecting them to lower dimension during fine-tuning removes that attack surface.","keywords":["safety alignment","linear representation hypothesis","activation steering","jailbreak defense","dimensionality reduction","Fast Johnson-Lindenstrauss Transform","Rademacher complexity","representation engineering"],"falsifier":"Train a single model family at several hidden widths, holding data and the alignment pipeline fixed, and measure both linear-probe accuracy on safety and ActAdd success after projecting to the same $K$: if probe accuracy does not jump near the 2,000-dimension threshold, or if a low-width variant remains as steerable as a high-width one, the dimensional-causality claim collapses. A cheaper check is to compute an ActAdd vector on the unmodified model and apply it to the defended model at a sequence of $K$ values; the theory predicts a sharp falloff in transfer, and its absence would falsify the mechanism.","tokens_in":31227,"feed_emoji":"🛡️","tokens_out":11182,"duration_ms":97547,"temperature":0.7,"pith_summary":"The paper sets out to show that the same large hidden dimension that makes large language models capable also opens a specific safety hole: high-dimensional activation spaces encode concepts such as 'safety' as single linear directions, and steering jailbreaks work by finding and manipulating those directions. It calls this tension the Paradox of Linear Separability: scaling models up improves abilities but also makes them more vulnerable to steering attacks. The evidence is a cross-family comparison of PCA visualizations and linear-probe accuracy over models with different hidden widths, showing a marked jump in linear separability of emotion near 2,000 hidden dimensions. The proposed remedy is to fine-tune with a low-dimensional projection of the hidden representations, either through a Fast Johnson–Lindenstrauss transform on the attention queries and keys or through a linear autoencoder bottleneck after an early layer; the paper reports that both restore refusal and safety scores against the ActAdd and Ablation jailbreaks across three 7B-scale model families. If the claim is right, hidden dimension is not just a capacity knob but an attack-surface parameter that can be turned down without giving up alignment.","feed_headline":"Compressing hidden states blunts steering-vector jailbreaks","feed_subtitle":"Projecting activations to a lower subspace destroys the linear safety direction jailbreak vectors exploit.","key_machinery":"The machinery is dimensionality reduction applied to the hidden representations that carry concept directions. The FJLT defense places a random Fast Johnson–Lindenstrauss projection matrix $\\Phi \\in \\mathbb{R}^{D \\times K}$ on the query and key matrices of one attention head per layer, computing attention scores in $K$ dimensions while approximately preserving pairwise distances; the Bottleneck defense inserts a linear autoencoder $x_{\\mathrm{compressed}} = \\sigma(x W_{\\mathrm{down}} W_{\\mathrm{up}})$ with $W_{\\mathrm{down}} \\in \\mathbb{R}^{D \\times K}$ and $W_{\\mathrm{up}} \\in \\mathbb{R}^{K \\times D}$ between two early layers. The theoretical support is the Rademacher-complexity bound $\\widehat{\\mathcal{R}}_X(\\mathcal{F}) \\le L \\|X\\|_F / N$, which, under the paper's Proposition 1 assuming normally distributed features, becomes $\\mathcal{R}_N(\\mathcal{F}) \\lesssim L \\sqrt{D/N}$: reducing dimension from $D$ to $K$ shrinks the capacity of any linear probe at rate $O(\\sqrt{D})$, which is the claimed reason a steering vector should become hard to recover after projection.","core_discovery":"The central claim is that linear concept directions in LLM activation spaces are a high-dimensional phenomenon, and that removing the dimensionality removes the handle that representation-engineering jailbreaks grab. The paper demonstrates a threshold-like pattern: models with hidden dimensions below roughly 2,000 show heavy overlap between positive and negative emotion activations, while models above it show two clean clusters, and the same separation is measured by trained linear probes. Because refusal behavior in aligned models is mediated by such a direction, the ActAdd attack that adds the refusal direction and the Ablation attack that removes it both succeed against the unmodified models and fail, to a much larger degree, after the representations are projected into a lower-dimensional subspace. The paper's own caution is that dimension reduction must be paired with fine-tuning that teaches the model to keep answering, and that the FJLT variant loses utility on specialized tasks while the Bottleneck variant largely preserves it.","pith_inferences":["The paper leaves implicit that the proposed defenses should transfer to any linear-readout attack, including concept-erasure methods that project out a safety direction, because all of them rely on the same recoverability of a linear direction.","A controlled hidden-width sweep within one family of models would turn the paper's cross-family trend into a causal claim; if the trend survives, dimensional regularization could be added to the alignment loss as an explicit safety term.","The $O(\\sqrt{D})$ bound suggests a testable scaling law: the minimal projected dimension $K$ at which an ActAdd vector stops transferring should grow slowly with model scale, so small $K$ values on 7B models may not suffice at much larger widths.","Because the Bottleneck model preserves truthfulness and emotion while hiding safety, probing which concepts stay linear after projection could serve as a cheap utility-preservation test before running full benchmark suites."],"forward_implications":["Steering-vector jailbreaks can be patched onto already-deployed Chat and Instruct models by a short fine-tune with a projection layer, rather than requiring retraining from scratch or a new alignment pipeline.","Scaling comparisons should count safety cost alongside capability: if the Paradox of Linear Separability holds, a model's expected vulnerability to ActAdd-style attacks grows with its hidden width.","Dimensionality becomes a tunable safety hyperparameter: the projected width $K$ and the insertion layer set where linear safety structure disappears and how much utility survives.","Defenses that only hide safety from linear probes are not a complete cure: the paper's own utility and GCG results imply that preserving non-safety linear concepts, and surviving non-linear attacks, are separate requirements."],"supporting_citations":[{"why":"Supplies the ActAdd jailbreak and the refusal-direction construction the defenses are evaluated against.","marker":"(Arditi et al., 2024)"},{"why":"Provides the Fast Johnson–Lindenstrauss Transform used to project query and key matrices into a lower dimension.","marker":"(Ailon & Chazelle, 2006)"},{"why":"Supplies the token-wise constrained fine-tuning objective that keeps the FJLT model close to the aligned model.","marker":"(Qi et al., 2024)"},{"why":"Provides the Rademacher-complexity bound on linear hypotheses that the paper's Proposition 1 derives the $\\sqrt{D}$ dependence from.","marker":"(Awasthi et al., 2020)"},{"why":"Establishes the distance-preserving embedding lemma that motivates using the FJLT projection in attention.","marker":"(Johnson & Lindenstrauss, 1984)"},{"why":"Supplies the linear-representation view of concepts and the probing approach used to measure linear separability.","marker":"(Marks & Tegmark, 2023)"}],"fun_headline_variants":["Lower-dimensional projections cripple steering-vector jailbreaks","High hidden dims enable jailbreak vectors; low dims disable them","Shrinking hidden state dimension blocks linear jailbreak attacks","Projection to low-dim space thwarts refusal-direction jailbreaks","Dimension reduction removes the handle for steering-vector jailbreaks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that hidden dimension itself, rather than the architecture, training data, or alignment procedure that comes with scaling, causes concepts to become linearly separable—and that the Rademacher-complexity bound on linear classifiers still says something about whether a steering vector can be found after projecting the whole network.","fun_headline_variants_meta":{"raw":{"variants":["Lower-dimensional projections cripple steering-vector jailbreaks","High hidden dims enable jailbreak vectors; low dims disable them","Shrinking hidden state dimension blocks linear jailbreak attacks","Projection to low-dim space thwarts refusal-direction jailbreaks","Dimension reduction removes the handle for steering-vector jailbreaks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000751,"raw_usage":{"total_tokens":3338,"prompt_tokens":936,"completion_tokens":2402,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":2318}},"tokens_in":552,"tokens_out":2402,"duration_ms":16847,"temperature":1.0,"reasoning_tokens":2318,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:45:44.373112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a single model family at several hidden widths, holding data and the alignment pipeline fixed, and measure both linear-probe accuracy on safety and ActAdd success after projecting to the same $K$: if probe accuracy does not jump near the 2,000-dimension threshold, or if a low-width variant remains as steerable as a high-width one, the dimensional-causality claim collapses. A cheaper check is to compute an ActAdd vector on the unmodified model and apply it to the defended model at a sequence of $K$ values; the theory predicts a sharp falloff in transfer, and its absence would falsify the mechanism.","supporting_citations":[],"review_version":1}