REVIEW 5 major objections 5 minor 1 cited by
Learning Safety Constraints for Large Language Models
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read SaP learns a polytope of safety constraints inside an LLM's representation space and steers generated tokens back into it, cutting adversarial attack success on Llama2-7B from 12.92% to 0.26% while leaving MMLU accuracy essentially…
desk verdict A solid, genuinely new inference-time safety method whose headline results are real but whose token-level label-transfer assumption needs scrutiny before the numbers are taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Safety Polytope, $\tilde{Q} = \{ \tilde f \in \mathbb{R}^d \mid \phi^\top \tilde f \le \tilde\xi \}$, is a set of $K$ linear halfspace facets learned from binary labels with the Convex Polytope Machine loss. The facets operate on features produced by a ReLU-activated 'concept encoder' that projects activations into a sparse 16,384-dimensional space, and they do triple duty: they classify each token as inside or outside the safe set, they define interpretable semantic concepts (e.g., a facet for kidnapping, a facet for sexual content), and they serve as the constraint in the steering step, which solves $\min_h \|\bar\pi_l(x) - h\|_1$ subject to $\phi^\top E_C(h) \le \tilde\xi$ during decoding.
What would settle it
Collect human-rated harmful outputs that SaP labels safe and compute, for each, whether every facet inequality $\phi^\top E_C(h) \le \tilde\xi$ is satisfied at the layer-20 activation. If any harmful output lies entirely inside the learned polytope, the safe set is not contained in the polytope and the geometric safety model is refuted.
Extended reading notes
Core claim
The central claim is that safety constraints are learnable geometric objects in LLM representation space. Given only binary safe/unsafe sentence labels, SaP learns $K$ linear facets with thresholds defining the polytope $\tilde{Q} = \{ \tilde f \mid \phi^\top \tilde f \le \tilde\xi \}$ on sparse features extracted from layer-20 activations by a concept encoder. At inference time, SafeFlow checks each token's encoded activation against the polytope and, when the activation lies outside, solves a small constrained optimization to move it back inside before the next token is decoded. The paper reports attack success rate reductions from 12.92% to 0.26% for Llama2-7B, from 55.77% to 3.25% for Ministral-8B, and from 27.57% to 11.81% for Qwen2-1.5B, with MMLU accuracy essentially unchanged, and shows that individual facets specialize to notions such as kidnapping, sexual content, and bullying.
Load-bearing premise
The method assumes that safety is linearly separable in the layer-20 representation space after the concept encoder, so that a polytope of halfspace constraints can separate safe from unsafe behavior; if safety is not separable there, the learned facets will misclassify and steering will push activations in the wrong direction.
Editorial extensions
If this is right
- Safety enforcement can be applied post-hoc to an already deployed model by editing layer activations during decoding, without retraining or fine-tuning weights.
- Facets trained only on binary safe/unsafe labels align with human semantic categories, so a flagged generation carries an explanation of which safety concept it violated.
- The linear structure of the facets scales to gradient-based optimization over prompts, leading to near-zero attack success rates against GCG, AutoPrompt, and other gradient-based jailbreaks while preserving MMLU performance.
- Defense quality improves with the number of facets up to a point (near-perfect Llama2 defense at 20 facets, diminishing returns beyond roughly 30–40), so polytope capacity should be tuned per model.
- The same machinery generalizes across three model families with different hidden-state dimensionalities, suggesting the geometric approach is architecture-agnostic.
Reading between the lines
- Beyond this paper, the same polytope construction could be retrained on other labels—toxicity, privacy, or factual reliability—since the geometric method itself is agnostic to the meaning of the safe/unsafe split.
- The AdaptiveAttack results differ sharply across models (near-perfect defense for Llama2, 14% ASR for Ministral, 100% for Qwen2), which a reader could take as evidence that the load-bearing variable is how linearly separable safety happens to be in each model's hidden states.
- A natural next experiment the authors did not run is transferring facets between models of different sizes or training runs, which would test whether the learned geometric safety structure is universal or model-specific.
- The theory appendix bounds sample complexity for polytopes that are perfectly separable with margin, so extending SaP to non-separable or noisy label settings would require a different theoretical treatment.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SaP (Safety Polytope), a post-hoc geometric safety mechanism for LLMs. A concept encoder maps layer-20 activations into a sparse feature space, a Convex Polytope Machine learns a polytope of safe features from safe/unsafe labels, and Algorithm 1 steers each generated token's activation back into the polytope before decoding. Experiments on Llama2-7B, Ministral-8B, and Qwen2-1.5B report large HarmBench attack-success-rate reductions (e.g., Llama2-7B from 12.92% to 0.26%) with roughly unchanged MMLU, plus interpretability analyses of the learned facets on BeaverTails and a PAC-style sample-complexity appendix.
Significance. If the results hold, SaP is a valuable inference-time defense: it preserves model weights, gives interpretable safety facets, and is model-agnostic. Strengths of the manuscript include public code, five-seed evaluation for ASR, detailed per-attack tables, a genuinely held-out Adaptive Attack evaluation for two of the three models, and an explicit attempt to address polysemanticity with a concept encoder. However, the main generalization claim is weakened by the fact that the reported average ASR includes attack families used in training, and the single fully held-out attack family still defeats Qwen2-1.5B completely (100% ASR), which the main text does not flag. The unvalidated transfer of sentence-level labels to per-token training data is a further correctness risk for the core steering mechanism.
major comments (5)
- [§4.1, Tables 2–6, Figure 2] The headline average ASR is partly in-distribution. The polytope is trained on 80% of the attack strings from the top-3 most effective attack methods for each model, and the reported average in Figure 2 is then computed over all nine attack algorithms, which include those same trained attack families. Only Adaptive Attack is fully held out. For example, Llama2-7B's per-attack table (Table 2) shows AdaptiveAttack at 2.00%, while the pooled average is 0.26%. The paper should report separate in-distribution and held-out averages, or exclude the training families from the headline number, so that the claimed generalization to unseen attack methods is actually measured.
- [§4.1, Table 6] Qwen2-1.5B remains at 100% ASR under Adaptive Attack even after SaP steering, identical to the original model, yet the main text reports only the favorable pooled average (27.57% to 11.81%) and §4.2 states that Qwen2 shows strong baseline robustness without the concept encoder. This is a load-bearing omission: one of the three evaluated models receives no adaptive-attack defense. The authors should report this failure explicitly, temper the generalization claim accordingly, and provide an explanation or further analysis.
- [Appendix C.1, Algorithm 1] The training data are labeled at sentence level and transferred to every token: Appendix C.1 states that if an output sentence is marked unsafe, all token representations in the sentence are labeled unsafe, and vice versa. Algorithm 1 then applies the polytope constraint to each generated token before decoding. This label-transfer assumption is unvalidated: early tokens of a harmful response may be labeled unsafe even when their hidden states are indistinguishable from safe tokens, and benign sentences containing harmful words are labeled safe. Since the concept encoder and polytope are trained end-to-end on these labels, the learned safe set may reflect response-level artifacts rather than token-level safety. The paper should validate token-level label correctness or ablate against a last-token-only or sentence-level steering baseline.
- [§4.1, Table 8, Table 9] The capability-preservation claim is supported only by single-point MMLU numbers in Table 8, with no standard deviations or number of seeds, despite the figure caption claiming that SaP consistently retains original MMLU accuracy. The same is true for MT-Bench in Table 9, which also shows a non-negligible drop for Qwen2-1.5B (6.03 to 5.49). The authors should report means and standard deviations over the same five seeds used for ASR, and discuss the MT-Bench decrease.
- [Appendix B, Eq. (B.1)] The Lagrangian steering loss in Appendix B appears to have a sign inconsistency. With [x]_- = min(0, x), the term λ_safe Σ [ϕ^T E_C(h) − ξ]_- is negative for already-satisfied constraints, so minimizing it moves the activation further into the safe side rather than leaving it unchanged. This contradicts the claim in §3 that original generation remains unchanged when the representation is already inside the polytope. If the intended penalty is max(0, −(ϕ^T E_C(h) − ξ)) or a squared penalty, the definition should be corrected; otherwise the method's behavior on safe tokens, especially with Qwen's λ_safe = 5000, is not what the text describes.
minor comments (5)
- [§4.1] The text mentions In-Context Learning (ICL) as a baseline in the results discussion, but ICL is not introduced in the baseline list in §4.1; Table 8 includes ICL results. Please clarify where ICL is described and how it is configured.
- [Figure 2, Appendix C.2] The number of attack algorithms used to form the average is inconsistent: §4.1 and the Figure 2 caption say nine attacks, while Appendix C.2 says Figure 2 aggregates over seven attack algorithms. Please unify the count and state explicitly which attacks enter each reported average.
- [Algorithm 1] Algorithm 1's comment says the original token generation remains 'unchained'; this should read 'unchanged'.
- [§2] In the CMDP definition, the text writes 'find a stationary policy π(x | a)'; the policy should be written π(a | x) to match the autoregressive LLM policy described immediately afterward.
- [§6] The Limitations section admits that Ministral-8B can induce semantically incoherent outputs and that the theoretical guarantees depend on strong, unverified assumptions. These caveats are important and should be connected more directly to the main-text claims, ideally with quantitative evidence about how often incoherent outputs occur.
Circularity Check
No significant circularity: SaP is a learned classifier plus steering projection, evaluated on held-out strings and a held-out attack; self-citations are motivational only.
full rationale
Walking the derivation chain: the polytope is not derived from the ASR numbers. Section 3 defines the safety set as {f_tilde | phi^T f_tilde <= xi} and fits phi, xi, EC jointly with the CPM loss (Eq. 1) on binary sentence unsafe labels. Algorithm 1 then solves the projection min_h ||bar_pi_l(x) - h||_1 s.t. phi^T f_tilde(h) <= xi; this is an optimization using the fitted classifier, not a quantity that equals its input by construction. The CMDP motivation cites Lindner et al. (2024), which shares authors, but the actual construction relies on CPM (Kantchelian et al. 2014) and the linear representation hypothesis (Park et al.), and the fitted polytope is benchmarked against MMLU and ASR, including Adaptive Attack that was explicitly excluded from training ('excluding Adaptive Attack, which is only used for evaluation'). The reported aggregate ASR does include attack families whose strings were used for training, so the 'generalization across attack methods' phrasing is stronger than the protocol fully supports; this is an evaluation-scope caveat, not a construction-level reduction. The paper's own limitations (semantically incoherent outputs under steering, Qwen2-1.5B retaining 100% ASR on Adaptive Attack in Table 6) are reported honestly and are inconsistent with a forced circular result. No prediction in the paper is identical by definition to a fitted input.
Assumptions & free parameters
free parameters (6)
- Margin κ =
60.0 (Llama2-7B), 5.0 (Ministral-8B), 30.0 (Qwen2-1.5B)
- Steering weights λunsafe, λsafe =
Llama2: 4.0, 1e-4; Ministral: 0.25, 0; Qwen: 10.0, 5000
- Number of facets K =
varies 1-60; final model uses around 20-50
- Concept encoder dimension =
16,384
- Feature layer l =
layer 20
- Category-specific margins and sparsity weights =
Tables 11-13
assumptions (4)
- domain assumption The linear representation hypothesis: high-level concepts in LLMs manifest as linear directions in feature space.
- standard math CMDP constraint learning: a conservative feasible policy set can be estimated from demonstrations as a convex polyhedron on feature expectations.
- ad hoc to paper Sentence-level safety labels transfer to all token representations in the sentence.
- ad hoc to paper Optimization in Eq. (2) solved by Lagrangian relaxation yields a nearby activation that satisfies the constraint.
Cite this review
Pith. "Pith review of Learning Safety Constraints for Large Language Models." pith.science (2026). https://pith.science/paper/64NSQ2DT
@misc{pith2026250524445,
author = {Pith},
title = {Pith review of: Learning Safety Constraints for Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/64NSQ2DT}},
note = {Machine review of arXiv:2505.24445}
}
read the original abstract
Large language models (LLMs) have emerged as powerful tools but pose significant safety risks through harmful outputs and vulnerability to adversarial attacks. We propose SaP, short for Safety Polytope, a geometric approach to LLM safety that learns and enforces multiple safety constraints directly in the model's representation space. We develop a framework that identifies safe and unsafe regions via the polytope's facets, enabling both detection and correction of unsafe outputs through geometric steering. Unlike existing approaches that modify model weights, SaP operates post-hoc in the representation space, preserving model capabilities while enforcing safety constraints. Experiments across multiple LLMs demonstrate that our method can effectively detect unethical inputs, reduce adversarial attack success rates while maintaining performance on standard tasks, thus highlighting the importance of having an explicit geometric model for safety. Analysis of the learned polytope facets reveals emergence of specialization in detecting different semantic notions of safety, providing interpretable insights into how safety is captured in LLMs' representation space.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
Geometry-Guided Constraint Learning for LLM Safety Classification
Sparse-autoencoder features reduce the number of safety constraints needed to two for most BeaverTails categories, and a three-phase-trained cone constraint modestly beats a flat polytope on in-distribution Qwen3.5-9B...
Reference graph
Works this paper leans on
-
[3]
Mt-bench-101: A fine- grained benchmark for evaluating large language models in multi-turn dialogues
Bai, G., Liu, J., Bu, X., He, Y ., Liu, J., Zhou, Z., Lin, Z., Su, W., Ge, T., Zheng, B., et al. Mt-bench-101: A fine- grained benchmark for evaluating large language models in multi-turn dialogues. arXiv preprint arXiv:2402.14762,
-
[6]
Cundy, C. and Ermon, S. Sequencematch: Imitation learn- ing for autoregressive sequence modelling with backtrack- ing. arXiv preprint arXiv:2306.05426,
-
[7]
Safe exploration in continuous action spaces
Dalal, G., Dvijotham, K., Vecerik, M., Hester, T., Paduraru, C., and Tassa, Y . Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757,
-
[9]
Gehman, S., Gururangan, S., Sap, M., Choi, Y ., and Smith, N. A. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462,
arXiv 2009
-
[11]
Measuring mas- sive multitask language understanding
10 Learning Safety Constraints for Large Language Models Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03300,
arXiv 2009
-
[12]
Back- door attacks for in-context learning with language models
Kandpal, N., Jagielski, M., Tram`er, F., and Carlini, N. Back- door attacks for in-context learning with language models. arXiv preprint arXiv:2307.14692,
-
[14]
Autodan: Generat- ing stealthy jailbreak prompts on aligned large language models
Liu, X., Xu, N., Chen, M., and Xiao, C. Autodan: Generat- ing stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451,
-
[15]
Enhancing LLM Safety via Constrained Direct Preference Optimization
Liu, Z., Sun, X., and Zheng, Z. Enhancing LLM Safety via Constrained Direct Preference Optimization. https://arxiv.org/abs/2403.02475v1, March
Show all 25 references
-
[16]
Mpax: Mathematical pro- gramming in jax
Lu, H., Peng, Z., and Yang, J. Mpax: Mathematical pro- gramming in jax. arXiv preprint arXiv:2412.09734,
-
[17]
Enhancing safety in reinforcement learning with human feedback via rectified policy optimization
Peng, X., Guo, H., Zhang, J., Zou, D., Shao, Z., Wei, H., and Liu, X. Enhancing safety in reinforcement learning with human feedback via rectified policy optimization. arXiv preprint arXiv:2410.19933,
-
[18]
Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming.arXiv preprint arXiv:2501.18837,
Sharma, M., Tong, M., Mu, J., Wei, J., Kruthoff, J., Good- friend, S., Ong, E., Peng, A., Agarwal, R., Anil, C., et al. Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming.arXiv preprint arXiv:2501.18837,
-
[19]
C., Perez, E., Hadfield- Menell, D., et al
11 Learning Safety Constraints for Large Language Models Sheshadri, A., Ewart, A., Guo, P., Lynch, A., Wu, C., Heb- bar, V ., Sleight, H., Stickland, A. C., Perez, E., Hadfield- Menell, D., et al. Latent adversarial training improves robustness to persistent harmful behaviors ...
-
[20]
Token-level direct preference optimization
Zeng, Y ., Liu, G., Ma, W., Yang, N., Zhang, H., and Wang, J. Token-level direct preference optimization. arXiv preprint arXiv:2404.11999,
-
[21]
Panacea: Pareto alignment via preference adaptation for llms
Zhong, Y ., Ma, C., Zhang, X., Yang, Z., Chen, H., Zhang, Q., Qi, S., and Yang, Y . Panacea: Pareto alignment via preference adaptation for llms. arXiv preprint arXiv:2402.02030,
-
[22]
Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization
Zhou, Z., Liu, J., Shao, J., Yue, X., Yang, C., Ouyang, W., and Qiao, Y . Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. In Find- ings of the Association for Computational Linguistics ACL 2024,
2024
-
[23]
This approach ensures diverse facet utilization across unsafe examples while preserving the effectiveness of the safety constraints. B. Steering via Lagrangian Relaxation Building on the learned polytope facets,SaP implements directional steering in the model’s representation ...
2024
-
[24]
{human question}\n{model answer}
We perform experiments in the same setup with prompt-based baselines Self Reminder (Xie et al., 2023), Response Check (Wang et al., 2024), and SmoothLLM (Robey et al., 2024), implemented in the BackTranslation code base (Wang et al., 2024). We also implemented rejection sampli...
2023
-
[25]
Results show mean ± standard deviation
Table 14: Test accuracy (%) comparison across different numbers of facets for each category in Llama2-7B. Results show mean ± standard deviation. Bold numbers indicate the selected configuration in our final model. Category 1 facet 10 facets 20 facets 30 facets 40 facets 50 fa...
2021
-
[1999]
Jail- breaking leading safety-aligned llms with simple adaptive attacks
Andriushchenko, M., Croce, F., and Flammarion, N. Jail- breaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint arXiv:2404.02151,
-
[2014]
Truthfulqa: Measuring how models mimic human falsehoods
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958,
-
[2016]
R., Millidge, B., Alfour, G., et al
Black, S., Sharkey, L., Grinsztajn, L., Winsor, E., Braun, D., Merizian, J., Parker, K., Guevara, C. R., Millidge, B., Alfour, G., et al. Interpreting neural networks through the polytope lens. arXiv preprint arXiv:2211.12312,
-
[2021]
Ai control: Improving safety despite intentional subversion
Greenblatt, R., Shlegeris, B., Sachan, K., and Roger, F. Ai control: Improving safety despite intentional subversion. arXiv preprint arXiv:2312.06942,
-
[2022]
Unlock- ing decoding-time controllability: Gradient-free multi- objective alignment with contrastive prompts
Fu, T., Hou, Y ., McAuley, J., and Yan, R. Unlock- ing decoding-time controllability: Gradient-free multi- objective alignment with contrastive prompts. arXiv preprint arXiv:2408.05094,
-
[2023]
Defending against unforeseen failure modes with latent adversarial training
Casper, S., Schulze, L., Patel, O., and Hadfield-Menell, D. Defending against unforeseen failure modes with latent adversarial training. arXiv preprint arXiv:2403.05030,
-
[2024]
and Bartlett, P
Anthony, M. and Bartlett, P. L. Neural Network Learning: Theoretical Foundations 1st Edition by Anthony, Martin, Bartlett, Peter L. (1999) Hardcover
1999
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.