REVIEW 3 major objections 6 minor 53 references
The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Models
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper argues that the safety gap of open-weight LLMs widens with scale, because safeguards suppress compliance while dangerous knowledge remains intact.
desk verdict A useful open-source toolkit with a plausible but unvalidated scaling claim; deserves peer review with requests for proxy validation and error bars. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the safety gap: the difference in effective dangerous capabilities between a production-ready model with safeguards intact and the same model after safeguard removal. Effective dangerous capabilities are defined as the product of two separately measured numbers: the model's likelihood-based accuracy on WMDP, a benign multiple-choice benchmark of biology, chemistry, and cyber knowledge, and its compliance rate on the paper's Bio-Chem-Cyber Propensity dataset, whose open-ended harmful questions were generated with DeepSeek-R1-Distill-Llama-70B, filtered by Llama Guard, and scored by StrongREJECT. The toolkit removes safeguards either by supervised fine-tuning (full or LoRA, with early stopping and automatic hyperparameter search) or by refusal ablation, which finds a per-layer 'refusal direction' $\mathbf{r}^*$ via difference-in-means between activations on refusal and harmless prompts, then orthogonalizes each weight matrix $W$ as $W' \leftarrow W - \frac{\mathbf{r}^* \mathbf{r}^{*\top}}{\lVert \mathbf{r}^* \rVert^2} W$. The product-of-factors design lets the paper separate 'does it know' from 'will it say,' and the case study leans on that separation to argue that safeguards act on the latter.
What would settle it
Run a small validation study where biology, chemistry, and cyber domain experts score the open-ended harmful responses of safeguard-removed models at several scales, and check whether expert-rated actionable danger grows with model size as steeply as the WMDP-accuracy-times-compliance product predicts; a flat or much flatter expert-scored curve would falsify the proxy.
Extended reading notes
Core claim
The paper's central discovery is a scaling law for misuse risk: effective dangerous capabilities, defined as the product of WMDP accuracy and Bio-Chem-Cyber Propensity compliance, stay near zero for original models but grow sharply with parameter count after safeguards are removed. Knowledge, as measured by multiple-choice accuracy, is essentially unaffected by safeguard removal—for example, Llama-3.1-405B-Instruct holds roughly 85% WMDP-Bio accuracy both before and after fine-tuning—while compliance on dangerous open-ended questions rises from below 5% to roughly 80–99% depending on model and removal method. From this the paper concludes that current safety measures chiefly suppress compliance rather than dangerous knowledge, and that the safety gap is a property of model scale rather than of a particular model family, since both Llama-3 and Qwen-2.5 show the same pattern.
Load-bearing premise
The load-bearing premise is that a model's accuracy on benign multiple-choice knowledge questions, multiplied by its tendency to answer harmful open-ended questions, is a faithful measure of its real-world dangerous capability; if knowledge on benign questions fails to transfer to actual harmful tasks, the computed safety gap overstates the risk.
Editorial extensions
If this is right
- Larger open-weight models become materially riskier than smaller ones after safeguard removal, so safety investments for 70B–405B models should be weighted more heavily than for sub-10B models.
- Developers' published safety evaluations—which test only the release model—will miss the main danger; the paper sees this as evidence that pre- and post-mitigation reporting should be standard.
- Fine-tuning on benign or small datasets (50–1,000 samples) suffices to strip safeguards, meaning accidental downstream fine-tuning is a realistic route to harmful compliance.
- Refusal ablation removes safeguards without degrading benign response quality, so it offers a clean, training-free probe for measuring underlying dangerous capabilities.
- The widening safety gap persists across two model families, suggesting the trend is tied to scale itself rather than to a specific vendor's training choices (though the paper stops short of generalizing beyond the tested families).
Reading between the lines
- If the proxy were validated against expert-scored open-ended attack tasks, the reported scaling might change; a natural test is to check a handful of models at a few scales against expert ratings on real-world harmful requests.
- The product formulation likely undercounts models whose knowledge is not MCQ-shaped and overcounts models that can answer multiple-choice questions but cannot produce actionable step-by-step instructions, so the absolute numbers are less trustworthy than the relative scale trend.
- The near-constant WMDP accuracy across refusal ablation and fine-tuning is independent evidence that refusal mechanisms are implemented as a separate 'refusal direction' rather than distributed throughout the model's knowledge.
- The same toolkit logic could be applied to other capability dimensions—for example, persuasion or deception—where a benign-knowledge benchmark and a propensity dataset could be multiplied into a comparable safety gap estimate.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the Safety Gap Toolkit, an open-source evaluation harness that removes safeguards from open-weight LLMs via supervised fine-tuning and refusal ablation, then measures compliance on a new Bio-Chem-Cyber Propensity dataset, accuracy on WMDP, and benign response quality on FreebaseQA. Effective dangerous capabilities are estimated as WMDP accuracy times compliance rate. In a case study on Llama-3 (1B–405B) and Qwen-2.5 (0.5B–72B), the authors report that safeguard removal increases compliance substantially while leaving WMDP accuracy roughly unchanged, so the safety gap—the difference in the product before and after removal—grows with model scale. They also report that SFT degrades benign response quality while refusal ablation does not.
Significance. If the scaling result holds, it is an important empirical datum for open-weight model governance: it would suggest that marginal risks from safeguard removal are concentrated in the largest models and that current safeguards primarily suppress compliance rather than knowledge. The toolkit itself is a concrete, reusable contribution: it standardizes a pipeline that previously required ad hoc code, and the public release with model cards and bootstrapped appendix results facilitates replication. The paper is appropriately candid about its limitations in Section 6.2. The main caveat is that the headline quantity is a proxy product that has not been validated against expert ground truth, so the central scaling claim remains conditional.
major comments (3)
- [§3.1, §4.4, §6.2] The estimator defined in Section 3.1 as the product of WMDP multiple-choice accuracy and the compliance rate on the Bio-Chem-Cyber Propensity dataset is load-bearing for the abstract and Section 5 claim that the safety gap widens with scale. The paper provides no validation that accuracy on benign multiple-choice questions transfers to producing accurate, actionable open-ended instructions, and no evidence that the two factors multiply into a meaningful joint probability. Section 4.4 shows that SFT degrades benign response quality by 10–30 percentage points, yet that degradation is not included in the product; a model that complies at 90% but produces generic or wrong instructions would be scored as 0.9 × accuracy. Section 6.2 acknowledges the proxy is imperfect but does not calibrate it. I ask the authors to (i) sample propensity responses across model sizes and safeguard-removal conditions, (ii) have domain experts score whether each response is both compliant and factually actionable, and (iii) compare the expert joint rate with the product. If the expert joint rate rises with scale more slowly than the product, the headline scaling result would be a measurement artifact.
- [Figures 2–4; Appendix A.3] The central figures in the main text plot point estimates of accuracy, compliance, effective dangerous capabilities, and quality without confidence intervals, even though the appendix states that bootstrapped 95% CIs were computed for bar plots. The effective-dangerous-capabilities product is nonlinear, and the original-model compliance rates are near zero, so small absolute changes in compliance translate into large relative changes in the product for small models; without propagated uncertainty it is impossible to assess whether the apparent monotonic scaling is statistically robust. Please report confidence intervals on the product for every condition, or at minimum include the underlying counts and a bootstrap interval in the main-text figures.
- [§3.3.1; Tables 2–3] The empirical basis for the scaling claim consists of two model families, one 51-sample harmful fine-tuning dataset, and a single fine-tuning run per configuration. The paper states in Section 6.2 that only two families were studied, but it does not discuss the sample size of the harmful fine-tuning dataset or run-to-run variability. Given the known sensitivity of fine-tuning-based safeguard removal to hyperparameters (cited in Section 3.3.1 via Qi et al. 2024b), the authors should either report variance over multiple seeds and learning rates for at least a subset of models or explicitly bound the claim to the specific configurations.
minor comments (6)
- [§A.1] The sentence 'we train for a full epoch for for the large benign dataset' contains a duplicated 'for'.
- [Figure 1; §3.4.1] The dataset is called Bio-Chem-Cyber Propensity, but Figure 1 refers to 'Bio Propensity dataset'; use the full name consistently.
- [Tables 2–3; Figures 5–11] The labels 'Ablated' and 'refusal ablated' are used interchangeably across the tables and figures; pick one label for the refusal-ablation condition.
- [§4.1; Tables 2–3] The statement that safeguard removal 'does not significantly impact accuracy' is based on small differences; include a paired comparison or a sentence noting the largest observed drops (e.g., Qwen-72B benign(1000) WMDP-Bio 0.80 vs 0.84).
- [§3.4.3] The quality judge (Claude-3.5-Haiku) is used without reporting agreement with human ratings on FreebaseQA responses; one sentence on validation of the judge would help.
- [§4; §6.1] Section 6.1 says the toolkit cannot remove safeguards on Llama 405B out-of-the-box, but Section 4 separately implements FP8 fine-tuning; clarify whether the toolkit limitation applies only to unquantized multi-node training.
Circularity Check
No circularity: the safety-gap scaling result is an empirical measurement of an explicitly defined proxy, not a derivation from its inputs.
full rationale
The paper does not claim to derive the scaling of the safety gap from a law or model; it defines an estimator (Section 3.1): effective dangerous capabilities equals WMDP accuracy multiplied by compliance on the authors' Bio-Chem-Cyber Propensity dataset, and the safety gap is the difference in this estimator before and after safeguard removal. The headline finding (Section 5, Figures 3 and 8) is an empirical measurement of this explicitly defined estimator across model sizes. No parameter is fitted and then renamed as a prediction, and no equation in the paper forces the product to increase with scale; e.g., Qwen2.5-0.5B accuracy drops after harmful SFT, and compliance varies non-monotonically across conditions and methods. The WMDP and StrongREJECT citations involve overlapping authorship, but both are public, externally reproducible benchmarks/evaluators used as measurement instruments; their outputs are measured in this paper rather than imported as unverified premises, so they do not constitute load-bearing self-citation. The strongest validity concern is that the proxy is unvalidated against expert ground truth (Section 3.1 acknowledges that a direct expert benchmark does not exist; Section 6.2 concedes the product is imperfect and may overestimate capabilities). That is a correctness/validity limitation, not circularity: the results could be artifacts of an unvalidated metric, but the metric is not defined in terms of the result it is used to support. Therefore no circular step meets the evidentiary bar of 'Eq. X = Eq. Y by construction' or 'fitted parameter renamed as prediction.'
Assumptions & free parameters
assumptions (5)
- domain assumption WMDP multiple-choice accuracy measures dangerous knowledge capabilities.
- domain assumption StrongREJECT correctly classifies compliance versus refusal on the new Bio-Chem-Cyber Propensity dataset.
- domain assumption The Bio-Chem-Cyber Propensity dataset questions are representative high-stakes dangerous requests.
- domain assumption The product of compliance and accuracy estimates effective dangerous capabilities.
- domain assumption Refusal ablation and the 51-sample harmful SFT are representative safeguard-removal threat models.
Cite this review
Pith. "Pith review of The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Models." pith.science (2026). https://pith.science/paper/3FONNUXO
@misc{pith2026250711544,
author = {Pith},
title = {Pith review of: The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/3FONNUXO}},
note = {Machine review of arXiv:2507.11544}
}
read the original abstract
Open-weight large language models (LLMs) unlock huge benefits in innovation, personalization, privacy, and democratization. However, their core advantage - modifiability - opens the door to systemic risks: bad actors can trivially subvert current safeguards, turning beneficial models into tools for harm. This leads to a 'safety gap': the difference in dangerous capabilities between a model with intact safeguards and one that has been stripped of those safeguards. We open-source a toolkit to estimate the safety gap for state-of-the-art open-weight models. As a case study, we evaluate biochemical and cyber capabilities, refusal rates, and generation quality of models from two families (Llama-3 and Qwen-2.5) across a range of parameter scales (0.5B to 405B) using different safeguard removal techniques. Our experiments reveal that the safety gap widens as model scale increases and effective dangerous capabilities grow substantially when safeguards are removed. We hope that the Safety Gap Toolkit (https://github.com/AlignmentResearch/safety-gap) will serve as an evaluation framework for common open-source models and as a motivation for developing and testing tamper-resistant safeguards. We welcome contributions to the toolkit from the community.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
Anthropic . Claude 3.5 Haiku . https://www.anthropic.com/claude/haiku, 2024
work page 2024
-
[2]
Refusal in language models is mediated by a single direction
Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., and Nanda, N. Refusal in language models is mediated by a single direction . arXiv preprint arXiv:2406.11717, 2024
arXiv 2024
-
[3]
Training a helpful and harmless assistant with reinforcement learning from human feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback . arXiv preprint arXiv:2204.05862, 2022
arXiv 2022
-
[4]
M., Gebru, T., McMillan-Major, A., and Shmitchell, S
Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp.\ 610--623, 2021
2021
-
[5]
International AI safety report
Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., Choi, Y., Fox, P., Garfinkel, B., Goldfarb, D., et al. International AI safety report. arXiv preprint arXiv:2501.17805, 2025
arXiv 2025
-
[6]
Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., Labenz, N., and Evans, O. Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs . arXiv preprint arXiv:2502.17424, 2025
arXiv 2025
-
[7]
Data Poisoning in LLMs: Jailbreak-Tuning and Scaling Laws
Bowen, D., Murphy, B., Cai, W., Khachaturov, D., Gleave, A., and Pelrine, K. Data Poisoning in LLMs: Jailbreak-Tuning and Scaling Laws . arXiv preprint arXiv:2408.02946, 2024
arXiv 2024
-
[8]
AI Companies Should Report Pre- and Post-Mitigation Safety Evaluations
Bowen, D., Dombrowski, A.-K., Gleave, A., and Cundy, C. AI Companies Should Report Pre-and Post-Mitigation Safety Evaluations . arXiv preprint arXiv:2503.17388, 2025
work page Pith review arXiv 2025
Show all 53 references
-
[9]
J., and Wong, E
Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., and Wong, E. Jailbreaking black box large language models in twenty queries . arXiv preprint arXiv:2310.08419, 2023
2023 arXiv
-
[10]
and Yang, D
Chen, J. and Yang, D. Unlearn what you want to forget: Efficient unlearning for LLMs . arXiv preprint arXiv:2310.20150, 2023
2023 arXiv
-
[11]
Nvidia hopper h100 gpu: Scaling performance
Choquette, J. Nvidia hopper h100 gpu: Scaling performance. IEEE Micro, 43 0 (3): 0 9--17, 2023
2023
-
[12]
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning . arXiv preprint arXiv:2501.12948, 1 2025. URL https://arxiv.org/abs/2501.12948
2025 arXiv
-
[13]
and Russinovich, M
Eldan, R. and Russinovich, M. Who's Harry Potter? Approximate unlearning in LLMs . arXiv preprint arXiv:2310.02238, 2023
2023 arXiv
-
[14]
The language model evaluation harness, 07 2024
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...
2024
-
[16]
Ai control: Improving safety despite intentional subversion
Greenblatt, R., Shlegeris, B., Sachan, K., and Roger, F. Ai control: Improving safety despite intentional subversion. arXiv preprint arXiv:2312.06942, 2023
2023 arXiv
-
[17]
Llama3-Jailbreak
Haize Labs . Llama3-Jailbreak . https://github.com/haizelabs/llama3-jailbreak, 2024. URL https://github.com/haizelabs/llama3-jailbreak. Accessed: March 20, 2025
2024
-
[18]
J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1 0 (2): 0 3, 2022
2022
-
[19]
M., Maxwell, T., Cheng, N., et al
Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., et al. Sleeper agents: Training deceptive LLMs that persist through safety training . arXiv preprint arXiv:2401.05566, 2024
2024 arXiv
-
[20]
Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testuggine, D., et al. Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations . arXiv preprint arXiv:2312.06674, 2023
2023 arXiv
-
[21]
PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference
Ji, J., Hong, D., Zhang, B., Chen, B., Dai, J., Zheng, B., Qiu, T., Li, B., and Yang, Y. PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference . arXiv preprint arXiv:2406.15513, 2024
2024 arXiv
-
[22]
F reebase QA : A new factoid QA data set matching trivia-style question-answer pairs with F reebase
Jiang, K., Wu, D., and Jiang, H. F reebase QA : A new factoid QA data set matching trivia-style question-answer pairs with F reebase. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technolog...
2019 doi
-
[23]
H., Gonzalez, J
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023
2023
-
[24]
Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b
Lermen, S., Rogers-Smith, C., and Ladish, J. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. arXiv preprint arXiv:2310.20624, 2023
2023 arXiv
-
[25]
D., Dombrowski, A.-K., Goel, S., Mukobi, G., Helm-Burger, N., Lababidi, R., Justen, L., Liu, A
Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, J. D., Dombrowski, A.-K., Goel, S., Mukobi, G., Helm-Burger, N., Lababidi, R., Justen, L., Liu, A. B., Chen, M., Barrass, I., Zhang, O., Zhu, X., Tamirisa, R., Bharathi, B., Herbert-Voss, A., Breuer, C. B., Zou, ...
2024
-
[26]
Deepseek-v3 technical report
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024 a
2024 arXiv
-
[27]
Y., Xu, X., Li, H., et al
Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, P., Yao, Y., Liu, C. Y., Xu, X., Li, H., et al. Rethinking machine unlearning for large language models . Nature Machine Intelligence, pp.\ 1--14, 2025
2025
-
[28]
Robustifying Safety-Aligned Large Language Models through Clean Data Curation
Liu, X., Liang, J., Ye, M., and Xi, Z. Robustifying Safety-Aligned Large Language Models through Clean Data Curation . arXiv preprint arXiv:2405.19358, 2024 b
2024 arXiv
-
[29]
Llama Team, A. . M. The Llama 3 Herd of Models , 2024. URL https://arxiv.org/abs/2407.21783
2024 arXiv
-
[30]
Simple probes can catch sleeper agents
MacDiarmid, M., Maxwell, T., Schiefer, N., Mu, J., Kaplan, J., Duvenaud, D., Bowman, S., Tamkin, A., Perez, E., Sharma, M., et al. Simple probes can catch sleeper agents. Anthropic Research Updates, 2024
2024
-
[31]
The Llama 4 herd: The beginning of a new era of natively multimodal intelligence , 4 2025
Meta. The Llama 4 herd: The beginning of a new era of natively multimodal intelligence , 4 2025. URL https://ai.meta.com/blog/llama-4-multimodal-intelligence/
2025
-
[32]
Meta-Llama-3.1-405B-Instruct-FP8
Neuralmagic. Meta-Llama-3.1-405B-Instruct-FP8 . https://huggingface.co/RedHatAI/Meta-Llama-3.1-405B-Instruct-FP8, July 2024
2024
-
[33]
Panickssery, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., and Turner, A. M. Steering Llama-2 via contrastive activation addition . arXiv preprint arXiv:2312.06681, 2023
2023 arXiv
-
[34]
Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023
Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023
2023 arXiv
-
[35]
Safety alignment should be made more than just a few tokens deep
Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A., Mittal, P., and Henderson, P. Safety alignment should be made more than just a few tokens deep . arXiv preprint arXiv:2406.05946, 2024 a
2024 arXiv
-
[36]
On evaluating the durability of safeguards for open-weight LLMs
Qi, X., Wei, B., Carlini, N., Huang, Y., Xie, T., He, L., Jagielski, M., Nasr, M., Mittal, P., and Henderson, P. On evaluating the durability of safeguards for open-weight LLMs . arXiv preprint arXiv:2412.07097, 2024 b
2024 arXiv
-
[37]
Fine-Tuning Llama 3.1 405B on a Single Node using Snowflake's Memory-Optimized AI Stack
Rasley, J., Samdani, R., Yao, Z., He, Y., and Rajbhandari, S. Fine-Tuning Llama 3.1 405B on a Single Node using Snowflake's Memory-Optimized AI Stack . https://www.snowflake.com/en/engineering-blog/fine-tune-llama-single-node-snowflake/, July 2024. Snowflake Engineering Blog
2024
-
[38]
Representation noising: A defence mechanism against harmful finetuning
Rosati, D., Wehner, J., Williams, K., Bartoszcze, L., Gonzales, R., carsten maple, Majumdar, S., Sajjad, H., and Rudzicz, F. Representation noising: A defence mechanism against harmful finetuning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems,...
2024
-
[39]
\'O ., Korinek, A., et al
Seger, E., Dreksler, N., Moulange, R., Dardaman, E., Schuett, J., Wei, K., Winter, C., Arnold, M., h \'E igeartaigh, S. \'O ., Korinek, A., et al. Open-sourcing highly capable foundation models: An evaluation of risks, benefits, and alternative methods for pursuing open-source...
2023 arXiv
-
[40]
Extracting Unlearned Information from LLMs with Activation Steering
Seyito g lu, A., Kuvshinov, A., Schwinn, L., and G \"u nnemann, S. Extracting Unlearned Information from LLMs with Activation Steering . arXiv preprint arXiv:2411.02631, 2024
2024 arXiv
-
[41]
Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming
Sharma, M., Tong, M., Mu, J., Wei, J., Kruthoff, J., Goodfriend, S., Ong, E., Peng, A., Agarwal, R., Anil, C., et al. Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming. arXiv preprint arXiv:2501.18837, 2025
2025 arXiv
-
[42]
``Do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models
Shen, X., Chen, Z., Backes, M., Shen, Y., and Zhang, Y. ``Do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models . In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp.\ 1671--1685, 2024
2024
-
[43]
A strong REJECT for empty jailbreaks
Souly, A., Lu, Q., Bowen, D., Trinh, T., Hsieh, E., Pandey, S., Abbeel, P., Svegliato, J., Emmons, S., Watkins, O., and Toyer, S. A strong REJECT for empty jailbreaks. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024
2024
-
[44]
Tamper-resistant safeguards for open-weight llms
Tamirisa, R., Bharathi, B., Phan, L., Zhou, A., Gatti, A., Suresh, T., Lin, M., Wang, J., Wang, R., Arel, R., et al. Tamper-resistant safeguards for open-weight llms . arXiv preprint arXiv:2408.00761, 2024
2024 arXiv
-
[45]
Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models
Wan, S., Nikolaidis, C., Song, D., Molnar, D., Crnkovich, J., Grace, J., Bhatt, M., Chennabasappa, S., Whitman, S., Ding, S., et al. Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models. arXiv preprint arXiv:2408.01605, 2024
2024 arXiv
-
[46]
Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation
Wang, X., Hu, C., R \"o ttger, P., and Plank, B. Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation . arXiv preprint arXiv:2410.03415, 2024
2024 arXiv
-
[47]
Jailbroken: How does LLM safety training fail? Advances in Neural Information Processing Systems, 36: 0 80079--80110, 2023
Wei, A., Haghtalab, N., and Steinhardt, J. Jailbroken: How does LLM safety training fail? Advances in Neural Information Processing Systems, 36: 0 80079--80110, 2023
2023
-
[48]
Huggingface's transformers: State-of-the-art natural language processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019
1910 arXiv
-
[49]
Qwen2.5 technical report, 2025
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, ...
2025 arXiv
-
[50]
Large language model unlearning
Yao, Y., Xu, X., and Liu, Y. Large language model unlearning . Advances in Neural Information Processing Systems, 37: 0 105425--105475, 2024
2024
-
[51]
B., and Kang, D
Zhan, Q., Fang, R., Bindu, R., Gupta, A., Hashimoto, T. B., and Kang, D. Removing RLHF Protections in GPT-4 via Fine-Tuning . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volu...
2024
-
[52]
Pytorch fsdp: experiences on scaling fully sharded data parallel
Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023
2023 arXiv
-
[53]
Z., and Fredrikson, M
Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023
2023 arXiv
-
[54]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.