Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Learning Safety Constraints for Large Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read SaP learns a polytope of safety constraints inside an LLM's representation space and steers generated tokens back into it, cutting adversarial attack success on Llama2-7B from 12.92% to 0.26% while leaving MMLU accuracy essentially…

desk verdict A solid, genuinely new inference-time safety method whose headline results are real but whose token-level label-transfer assumption needs scrutiny before the numbers are taken at face value. read the letter →

arxiv 2505.24445 v1 pith:64NSQ2DT submitted 2025-05-30 cs.LG cs.AI

classification cs.LGcs.AI
keywords largelanguagemodelssafetyconstraintsadversarialattacksrepresentationspacepolytopelearningconceptencoderinference-timesteeringinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that safety in large language models can be represented explicitly as geometry: a polytope, formed by intersecting linear halfspace constraints, inside the model's own representation space. It proposes SaP, which learns the polytope's facets from binary safe/unsafe labels using a Convex Polytope Machine, then enforces the constraints at generation time by steering layer-20 activations back inside the boundary. In experiments, this lowers the average adversarial attack success rate on Llama2-7B from 12.92% to 0.26% while MMLU accuracy stays nearly flat (45.8% vs. 45.7%). The approach matters because it offers post-hoc, weight-free safety control and an explicit, inspectable account of which semantic concepts each safety facet detects.

What carries the argument

The Safety Polytope, $\tilde{Q} = \{ \tilde f \in \mathbb{R}^d \mid \phi^\top \tilde f \le \tilde\xi \}$, is a set of $K$ linear halfspace facets learned from binary labels with the Convex Polytope Machine loss. The facets operate on features produced by a ReLU-activated 'concept encoder' that projects activations into a sparse 16,384-dimensional space, and they do triple duty: they classify each token as inside or outside the safe set, they define interpretable semantic concepts (e.g., a facet for kidnapping, a facet for sexual content), and they serve as the constraint in the steering step, which solves $\min_h \|\bar\pi_l(x) - h\|_1$ subject to $\phi^\top E_C(h) \le \tilde\xi$ during decoding.

What would settle it

Collect human-rated harmful outputs that SaP labels safe and compute, for each, whether every facet inequality $\phi^\top E_C(h) \le \tilde\xi$ is satisfied at the layer-20 activation. If any harmful output lies entirely inside the learned polytope, the safe set is not contained in the polytope and the geometric safety model is refuted.

Watch

Extended reading notes

Core claim

The central claim is that safety constraints are learnable geometric objects in LLM representation space. Given only binary safe/unsafe sentence labels, SaP learns $K$ linear facets with thresholds defining the polytope $\tilde{Q} = \{ \tilde f \mid \phi^\top \tilde f \le \tilde\xi \}$ on sparse features extracted from layer-20 activations by a concept encoder. At inference time, SafeFlow checks each token's encoded activation against the polytope and, when the activation lies outside, solves a small constrained optimization to move it back inside before the next token is decoded. The paper reports attack success rate reductions from 12.92% to 0.26% for Llama2-7B, from 55.77% to 3.25% for Ministral-8B, and from 27.57% to 11.81% for Qwen2-1.5B, with MMLU accuracy essentially unchanged, and shows that individual facets specialize to notions such as kidnapping, sexual content, and bullying.

Load-bearing premise

The method assumes that safety is linearly separable in the layer-20 representation space after the concept encoder, so that a polytope of halfspace constraints can separate safe from unsafe behavior; if safety is not separable there, the learned facets will misclassify and steering will push activations in the wrong direction.

Editorial extensions

If this is right

  • Safety enforcement can be applied post-hoc to an already deployed model by editing layer activations during decoding, without retraining or fine-tuning weights.
  • Facets trained only on binary safe/unsafe labels align with human semantic categories, so a flagged generation carries an explanation of which safety concept it violated.
  • The linear structure of the facets scales to gradient-based optimization over prompts, leading to near-zero attack success rates against GCG, AutoPrompt, and other gradient-based jailbreaks while preserving MMLU performance.
  • Defense quality improves with the number of facets up to a point (near-perfect Llama2 defense at 20 facets, diminishing returns beyond roughly 30–40), so polytope capacity should be tuned per model.
  • The same machinery generalizes across three model families with different hidden-state dimensionalities, suggesting the geometric approach is architecture-agnostic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond this paper, the same polytope construction could be retrained on other labels—toxicity, privacy, or factual reliability—since the geometric method itself is agnostic to the meaning of the safe/unsafe split.
  • The AdaptiveAttack results differ sharply across models (near-perfect defense for Llama2, 14% ASR for Ministral, 100% for Qwen2), which a reader could take as evidence that the load-bearing variable is how linearly separable safety happens to be in each model's hidden states.
  • A natural next experiment the authors did not run is transferring facets between models of different sizes or training runs, which would test whether the learned geometric safety structure is universal or model-specific.
  • The theory appendix bounds sample complexity for polytopes that are perfectly separable with margin, so extending SaP to non-separable or noisy label settings would require a different theoretical treatment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SaP (Safety Polytope), a post-hoc geometric safety mechanism for LLMs. A concept encoder maps layer-20 activations into a sparse feature space, a Convex Polytope Machine learns a polytope of safe features from safe/unsafe labels, and Algorithm 1 steers each generated token's activation back into the polytope before decoding. Experiments on Llama2-7B, Ministral-8B, and Qwen2-1.5B report large HarmBench attack-success-rate reductions (e.g., Llama2-7B from 12.92% to 0.26%) with roughly unchanged MMLU, plus interpretability analyses of the learned facets on BeaverTails and a PAC-style sample-complexity appendix.

Significance. If the results hold, SaP is a valuable inference-time defense: it preserves model weights, gives interpretable safety facets, and is model-agnostic. Strengths of the manuscript include public code, five-seed evaluation for ASR, detailed per-attack tables, a genuinely held-out Adaptive Attack evaluation for two of the three models, and an explicit attempt to address polysemanticity with a concept encoder. However, the main generalization claim is weakened by the fact that the reported average ASR includes attack families used in training, and the single fully held-out attack family still defeats Qwen2-1.5B completely (100% ASR), which the main text does not flag. The unvalidated transfer of sentence-level labels to per-token training data is a further correctness risk for the core steering mechanism.

major comments (5)
  1. [§4.1, Tables 2–6, Figure 2] The headline average ASR is partly in-distribution. The polytope is trained on 80% of the attack strings from the top-3 most effective attack methods for each model, and the reported average in Figure 2 is then computed over all nine attack algorithms, which include those same trained attack families. Only Adaptive Attack is fully held out. For example, Llama2-7B's per-attack table (Table 2) shows AdaptiveAttack at 2.00%, while the pooled average is 0.26%. The paper should report separate in-distribution and held-out averages, or exclude the training families from the headline number, so that the claimed generalization to unseen attack methods is actually measured.
  2. [§4.1, Table 6] Qwen2-1.5B remains at 100% ASR under Adaptive Attack even after SaP steering, identical to the original model, yet the main text reports only the favorable pooled average (27.57% to 11.81%) and §4.2 states that Qwen2 shows strong baseline robustness without the concept encoder. This is a load-bearing omission: one of the three evaluated models receives no adaptive-attack defense. The authors should report this failure explicitly, temper the generalization claim accordingly, and provide an explanation or further analysis.
  3. [Appendix C.1, Algorithm 1] The training data are labeled at sentence level and transferred to every token: Appendix C.1 states that if an output sentence is marked unsafe, all token representations in the sentence are labeled unsafe, and vice versa. Algorithm 1 then applies the polytope constraint to each generated token before decoding. This label-transfer assumption is unvalidated: early tokens of a harmful response may be labeled unsafe even when their hidden states are indistinguishable from safe tokens, and benign sentences containing harmful words are labeled safe. Since the concept encoder and polytope are trained end-to-end on these labels, the learned safe set may reflect response-level artifacts rather than token-level safety. The paper should validate token-level label correctness or ablate against a last-token-only or sentence-level steering baseline.
  4. [§4.1, Table 8, Table 9] The capability-preservation claim is supported only by single-point MMLU numbers in Table 8, with no standard deviations or number of seeds, despite the figure caption claiming that SaP consistently retains original MMLU accuracy. The same is true for MT-Bench in Table 9, which also shows a non-negligible drop for Qwen2-1.5B (6.03 to 5.49). The authors should report means and standard deviations over the same five seeds used for ASR, and discuss the MT-Bench decrease.
  5. [Appendix B, Eq. (B.1)] The Lagrangian steering loss in Appendix B appears to have a sign inconsistency. With [x]_- = min(0, x), the term λ_safe Σ [ϕ^T E_C(h) − ξ]_- is negative for already-satisfied constraints, so minimizing it moves the activation further into the safe side rather than leaving it unchanged. This contradicts the claim in §3 that original generation remains unchanged when the representation is already inside the polytope. If the intended penalty is max(0, −(ϕ^T E_C(h) − ξ)) or a squared penalty, the definition should be corrected; otherwise the method's behavior on safe tokens, especially with Qwen's λ_safe = 5000, is not what the text describes.
minor comments (5)
  1. [§4.1] The text mentions In-Context Learning (ICL) as a baseline in the results discussion, but ICL is not introduced in the baseline list in §4.1; Table 8 includes ICL results. Please clarify where ICL is described and how it is configured.
  2. [Figure 2, Appendix C.2] The number of attack algorithms used to form the average is inconsistent: §4.1 and the Figure 2 caption say nine attacks, while Appendix C.2 says Figure 2 aggregates over seven attack algorithms. Please unify the count and state explicitly which attacks enter each reported average.
  3. [Algorithm 1] Algorithm 1's comment says the original token generation remains 'unchained'; this should read 'unchanged'.
  4. [§2] In the CMDP definition, the text writes 'find a stationary policy π(x | a)'; the policy should be written π(a | x) to match the autoregressive LLM policy described immediately afterward.
  5. [§6] The Limitations section admits that Ministral-8B can induce semantically incoherent outputs and that the theoretical guarantees depend on strong, unverified assumptions. These caveats are important and should be connected more directly to the main-text claims, ideally with quantitative evidence about how often incoherent outputs occur.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: SaP is a learned classifier plus steering projection, evaluated on held-out strings and a held-out attack; self-citations are motivational only.

full rationale

Walking the derivation chain: the polytope is not derived from the ASR numbers. Section 3 defines the safety set as {f_tilde | phi^T f_tilde <= xi} and fits phi, xi, EC jointly with the CPM loss (Eq. 1) on binary sentence unsafe labels. Algorithm 1 then solves the projection min_h ||bar_pi_l(x) - h||_1 s.t. phi^T f_tilde(h) <= xi; this is an optimization using the fitted classifier, not a quantity that equals its input by construction. The CMDP motivation cites Lindner et al. (2024), which shares authors, but the actual construction relies on CPM (Kantchelian et al. 2014) and the linear representation hypothesis (Park et al.), and the fitted polytope is benchmarked against MMLU and ASR, including Adaptive Attack that was explicitly excluded from training ('excluding Adaptive Attack, which is only used for evaluation'). The reported aggregate ASR does include attack families whose strings were used for training, so the 'generalization across attack methods' phrasing is stronger than the protocol fully supports; this is an evaluation-scope caveat, not a construction-level reduction. The paper's own limitations (semantically incoherent outputs under steering, Qwen2-1.5B retaining 100% ASR on Adaptive Attack in Table 6) are reported honestly and are inconsistent with a forced circular result. No prediction in the paper is identical by definition to a fitted input.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method introduces many tuned hyperparameters (margins, steering weights, facet counts, feature layer) and relies on domain assumptions about linear separability, label transfer, and the CMDP-to-feature-space analogy. No new physical or scientific entities are postulated.

free parameters (6)
  • Margin κ = 60.0 (Llama2-7B), 5.0 (Ministral-8B), 30.0 (Qwen2-1.5B)
    Margin in the CPM loss, tuned per model architecture (Appendix C.1).
  • Steering weights λunsafe, λsafe = Llama2: 4.0, 1e-4; Ministral: 0.25, 0; Qwen: 10.0, 5000
    Weights in the Lagrangian relaxation for activation steering (Appendix B).
  • Number of facets K = varies 1-60; final model uses around 20-50
    Determines polytope capacity; performance peaks around 20-40 facets (Figure 6).
  • Concept encoder dimension = 16,384
    Hidden dimension of the learned feature space for the polytope.
  • Feature layer l = layer 20
    Intermediate layer from which activations are extracted; chosen for all models.
  • Category-specific margins and sparsity weights = Tables 11-13
    Per-safety-category hyperparameters for BeaverTails classification, tuned per category and model.
assumptions (4)
  • domain assumption The linear representation hypothesis: high-level concepts in LLMs manifest as linear directions in feature space.
    Invoked in Section 2 to justify modeling safety as a polytope of linear inequalities in activation space.
  • standard math CMDP constraint learning: a conservative feasible policy set can be estimated from demonstrations as a convex polyhedron on feature expectations.
    Result from Lindner et al. (2024) used as the paper's motivation; the paper does not actually compute feature expectations over trajectories.
  • ad hoc to paper Sentence-level safety labels transfer to all token representations in the sentence.
    Appendix C.1 labels every token in an unsafe output as unsafe; this is a strong assumption about locality of safety information.
  • ad hoc to paper Optimization in Eq. (2) solved by Lagrangian relaxation yields a nearby activation that satisfies the constraint.
    Appendix B uses 100 gradient steps of a heuristic loss; no guarantee of constraint satisfaction or proximity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Safety Constraints for Large Language Models." pith.science (2026). https://pith.science/paper/64NSQ2DT

@misc{pith2026250524445,
  author       = {Pith},
  title        = {Pith review of: Learning Safety Constraints for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/64NSQ2DT}},
  note         = {Machine review of arXiv:2505.24445}
}
read the original abstract

Large language models (LLMs) have emerged as powerful tools but pose significant safety risks through harmful outputs and vulnerability to adversarial attacks. We propose SaP, short for Safety Polytope, a geometric approach to LLM safety that learns and enforces multiple safety constraints directly in the model's representation space. We develop a framework that identifies safe and unsafe regions via the polytope's facets, enabling both detection and correction of unsafe outputs through geometric steering. Unlike existing approaches that modify model weights, SaP operates post-hoc in the representation space, preserving model capabilities while enforcing safety constraints. Experiments across multiple LLMs demonstrate that our method can effectively detect unethical inputs, reduce adversarial attack success rates while maintaining performance on standard tasks, thus highlighting the importance of having an explicit geometric model for safety. Analysis of the learned polytope facets reveals emergence of specialization in detecting different semantic notions of safety, providing interpretable insights into how safety is captured in LLMs' representation space.

Figures

Figures reproduced from arXiv: 2505.24445 by the authors.

Figure 1
Figure 1. Illustration of the geometric approach to language model safety proposed in this paper. A safety facet is triggered [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of model MMLU accuracy and average Attack Success Rate (ASR) on 9 attack algorithms. All [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Comparison of average ASR for each model, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Mutual Information Heatmap showing the comparison between models without (a) and with Concept Encoder (b). [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: KL divergence analysis revealing semantic specialization of [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Impact of polytope constraint count on model performance. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: KL divergence analysis revealing semantic specialization of [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Geometry-Guided Constraint Learning for LLM Safety Classification

    cs.AI 2026-06 conditional novelty 6.0 of 10

    Sparse-autoencoder features reduce the number of safety constraints needed to two for most BeaverTails categories, and a three-phase-trained cone constraint modestly beats a flat polytope on in-distribution Qwen3.5-9B...

Reference graph

Works this paper leans on

25 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [3]

    Mt-bench-101: A fine- grained benchmark for evaluating large language models in multi-turn dialogues

    Bai, G., Liu, J., Bu, X., He, Y ., Liu, J., Zhou, Z., Lin, Z., Su, W., Ge, T., Zheng, B., et al. Mt-bench-101: A fine- grained benchmark for evaluating large language models in multi-turn dialogues. arXiv preprint arXiv:2402.14762,

  2. [6]

    and Ermon, S

    Cundy, C. and Ermon, S. Sequencematch: Imitation learn- ing for autoregressive sequence modelling with backtrack- ing. arXiv preprint arXiv:2306.05426,

  3. [7]

    Safe exploration in continuous action spaces

    Dalal, G., Dvijotham, K., Vecerik, M., Hester, T., Paduraru, C., and Tassa, Y . Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757,

  4. [9]

    Gehman, S., Gururangan, S., Sap, M., Choi, Y ., and Smith, N. A. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462,

  5. [11]

    Measuring mas- sive multitask language understanding

    10 Learning Safety Constraints for Large Language Models Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03300,

  6. [12]

    Back- door attacks for in-context learning with language models

    Kandpal, N., Jagielski, M., Tram`er, F., and Carlini, N. Back- door attacks for in-context learning with language models. arXiv preprint arXiv:2307.14692,

  7. [14]

    Autodan: Generat- ing stealthy jailbreak prompts on aligned large language models

    Liu, X., Xu, N., Chen, M., and Xiao, C. Autodan: Generat- ing stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451,

  8. [15]

    Enhancing LLM Safety via Constrained Direct Preference Optimization

    Liu, Z., Sun, X., and Zheng, Z. Enhancing LLM Safety via Constrained Direct Preference Optimization. https://arxiv.org/abs/2403.02475v1, March

Show all 25 references
  1. [16]

    Mpax: Mathematical pro- gramming in jax

    Lu, H., Peng, Z., and Yang, J. Mpax: Mathematical pro- gramming in jax. arXiv preprint arXiv:2412.09734,

  2. [17]

    Enhancing safety in reinforcement learning with human feedback via rectified policy optimization

    Peng, X., Guo, H., Zhang, J., Zou, D., Shao, Z., Wei, H., and Liu, X. Enhancing safety in reinforcement learning with human feedback via rectified policy optimization. arXiv preprint arXiv:2410.19933,

  3. [18]

    Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming.arXiv preprint arXiv:2501.18837,

    Sharma, M., Tong, M., Mu, J., Wei, J., Kruthoff, J., Good- friend, S., Ong, E., Peng, A., Agarwal, R., Anil, C., et al. Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming.arXiv preprint arXiv:2501.18837,

  4. [19]

    C., Perez, E., Hadfield- Menell, D., et al

    11 Learning Safety Constraints for Large Language Models Sheshadri, A., Ewart, A., Guo, P., Lynch, A., Wu, C., Heb- bar, V ., Sleight, H., Stickland, A. C., Perez, E., Hadfield- Menell, D., et al. Latent adversarial training improves robustness to persistent harmful behaviors ...

  5. [20]

    Token-level direct preference optimization

    Zeng, Y ., Liu, G., Ma, W., Yang, N., Zhang, H., and Wang, J. Token-level direct preference optimization. arXiv preprint arXiv:2404.11999,

  6. [21]

    Panacea: Pareto alignment via preference adaptation for llms

    Zhong, Y ., Ma, C., Zhang, X., Yang, Z., Chen, H., Zhang, Q., Qi, S., and Yang, Y . Panacea: Pareto alignment via preference adaptation for llms. arXiv preprint arXiv:2402.02030,

  7. [22]

    Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization

    Zhou, Z., Liu, J., Shao, J., Yue, X., Yang, C., Ouyang, W., and Qiao, Y . Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. In Find- ings of the Association for Computational Linguistics ACL 2024,

  8. [23]

    This approach ensures diverse facet utilization across unsafe examples while preserving the effectiveness of the safety constraints. B. Steering via Lagrangian Relaxation Building on the learned polytope facets,SaP implements directional steering in the model’s representation ...

  9. [24]

    {human question}\n{model answer}

    We perform experiments in the same setup with prompt-based baselines Self Reminder (Xie et al., 2023), Response Check (Wang et al., 2024), and SmoothLLM (Robey et al., 2024), implemented in the BackTranslation code base (Wang et al., 2024). We also implemented rejection sampli...

  10. [25]

    Results show mean ± standard deviation

    Table 14: Test accuracy (%) comparison across different numbers of facets for each category in Llama2-7B. Results show mean ± standard deviation. Bold numbers indicate the selected configuration in our final model. Category 1 facet 10 facets 20 facets 30 facets 40 facets 50 fa...

  11. [1999]

    Jail- breaking leading safety-aligned llms with simple adaptive attacks

    Andriushchenko, M., Croce, F., and Flammarion, N. Jail- breaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint arXiv:2404.02151,

  12. [2014]

    Truthfulqa: Measuring how models mimic human falsehoods

    Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958,

  13. [2016]

    R., Millidge, B., Alfour, G., et al

    Black, S., Sharkey, L., Grinsztajn, L., Winsor, E., Braun, D., Merizian, J., Parker, K., Guevara, C. R., Millidge, B., Alfour, G., et al. Interpreting neural networks through the polytope lens. arXiv preprint arXiv:2211.12312,

  14. [2021]

    Ai control: Improving safety despite intentional subversion

    Greenblatt, R., Shlegeris, B., Sachan, K., and Roger, F. Ai control: Improving safety despite intentional subversion. arXiv preprint arXiv:2312.06942,

  15. [2022]

    Unlock- ing decoding-time controllability: Gradient-free multi- objective alignment with contrastive prompts

    Fu, T., Hou, Y ., McAuley, J., and Yan, R. Unlock- ing decoding-time controllability: Gradient-free multi- objective alignment with contrastive prompts. arXiv preprint arXiv:2408.05094,

  16. [2023]

    Defending against unforeseen failure modes with latent adversarial training

    Casper, S., Schulze, L., Patel, O., and Hadfield-Menell, D. Defending against unforeseen failure modes with latent adversarial training. arXiv preprint arXiv:2403.05030,

  17. [2024]

    and Bartlett, P

    Anthony, M. and Bartlett, P. L. Neural Network Learning: Theoretical Foundations 1st Edition by Anthony, Martin, Bartlett, Peter L. (1999) Hardcover

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.