Pith. sign in

REVIEW 3 major objections 4 minor 38 references

Dual-use knowledge can stay in the model and be gated by a control token, giving a better safety-utility trade-off than unlearning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:46 UTC pith:JVM6ZPKQ

load-bearing objection A genuinely useful conditionalization recipe with strong multi-scale evidence, but the 'privileged control token' framing overstates the security boundary for open weights, and the marking threshold is fitted to the evaluation benchmarks. the 3 major comments →

arxiv 2607.18639 v1 pith:JVM6ZPKQ submitted 2026-07-21 cs.LG cs.CL

Mark, Don't Erase: Token Inoculation for Dual-Use Knowledge in LLMs

Seunghyun Lee , Dongyoon Han , Sangdoo Yun This is my paper
classification cs.LG cs.CL
keywords dual-use knowledgeLLM safetyconditional refusalcontrol tokencontinued pre-trainingunlearningsafety-utility trade-offrefusal selectivity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the right response to dual-use knowledge is conditioning, not deletion. It proposes Token Inoculation: mark hazardous content with a special token during continued pre-training, then fine-tune the model to answer hazardous questions only when that token is present and to refuse when it is absent. On a hazardous biosecurity benchmark, the method cuts accuracy from 79% to 18% while keeping about 93% of general-domain performance, and it beats unlearning and refusal baselines across model scales from 1B to 14B. If the claim holds, safety alignment becomes a control problem rather than a forgetting problem, and a single model can serve both authorized and public users.

Core claim

The paper's central claim is that hazardous knowledge need not be erased to be made safe; it can be retained and behaviorally gated by a privileged token. The method, Token Inoculation, works in two stages. Inoculation CPT binds the special token <|ino|> to hazardous-domain semantics by inserting it before risky sentences during continued pre-training, with loss on the token itself masked so it acts as a condition rather than a generated output. Inoculation SFT then teaches two branches: correct answers when <|ino|> appears at the start of the assistant's reply, and refusal when it is absent. At inference the token stays out of ordinary user inputs, so the model defaults to refusal; placing

What carries the argument

The central object is the special token <|ino|>, defined as a domain-grounded access gate: a single token that the model learns to associate with hazardous knowledge through marking during continued pre-training, and on which it learns to branch between answering and refusing during supervised fine-tuning. The binding stage scores each corpus sentence by embedding similarity to a 'hazard direction' (the centroid difference between manipulation-related and clinical sentences) and inserts <|ino|> above a threshold; the branching stage pairs inoculation examples (correct answers with the token) with refusal examples (refusals without it). Loss masking on the token ensures it works strictly as a

Load-bearing premise

The load-bearing assumption is that <|ino|> is a privileged assistant-side signal: if a user or an adversary who holds the model can inject the token at the start of the assistant turn, the refusal gate opens and the retained hazardous knowledge becomes accessible, which the paper's own limitations section concedes is possible against non-adaptive adversaries.

What would settle it

Give an adversary access to the trained model weights and the chat template, allow them to place <|ino|> at the start of the assistant reply, and measure accuracy on a hazardous benchmark: Appendix K shows such unlocked models answer at roughly 70–80%, which would falsify the claim that the token provides controlled access rather than merely storing hazardous knowledge behind a trivially re-insertable flag.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A single model can serve both authorized users and the public: the service provider unlocks access by inserting the token into the assistant reply, and ordinary traffic gets refusal.
  • Operators get a continuous dial, not a binary choice: refusal selectivity rises with the quality of the inoculation training examples and with how narrowly the token is bound to hazardous content.
  • The same two-stage pipeline transfers to a second hazardous domain (cybersecurity), so the mechanism is not a biology-specific artifact.
  • The safety-utility advantage grows with model scale, making the approach more attractive at frontier sizes where unlearning damage is greatest.
  • If the claim is correct, safety alignment is better evaluated as a conditioning problem — measured by the joint safety-utility F1 — than as a forgetting problem.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: on open-weight models the 'privileged' token is only as privileged as the chat template; any holder of the weights can edit the template to place <|ino|> at the assistant start, and the paper's own deployment numbers show that this restores roughly 70–80% hazardous-query accuracy.
  • Editorial inference: the marking threshold is effectively a policy dial. A learned risk scorer could replace the embedding heuristic and sharpen the boundary between hazardous and benign content without the per-corpus cost the paper flags.
  • Editorial inference: the method suggests a research direction the paper leaves implicit — per-user, per-context, or per-domain tokens could make gating finer-grained than a global on/off switch.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Token Inoculation, a two-stage procedure for dual-use knowledge: continued pre-training inserts a special token <|ino|> before hazardous sentences, and supervised fine-tuning teaches the model to answer when <|ino|> is present and refuse when absent. The authors evaluate on WMDP-Bio/MMLU across Qwen2.5-1.5B/7B/14B, Llama-3.1-8B, and Phi4-14B, reporting that Token Inoculation achieves the highest Safety-Utility F1 against RMU, NPO, DPO, GA, SimNPO, and Vanilla Refusal, and transfers to WMDP-Cyber. They also present component ablations, a training-data intervention varying inoculation accuracy, marking-threshold sweeps, and logit-lens analyses. The main claim is that hazardous knowledge can be retained and behaviorally gated by a privileged control token.

Significance. If the result holds, the conditioning-based approach is a plausible alternative to unlearning and refusal training, with the practical advantage of serving both safe and authorized modes from a single model. The empirical base is substantial: three model families, 1B–14B scales, a second hazardous domain (cyber), 3-run means over multiple seeds, component ablations, and a mechanistic logit-lens analysis. These are real strengths. However, the significance is tempered by two unresolved issues: the 'privileged' control token is not privileged in open-weight deployments, and the key marking parameters are fit to the evaluation benchmarks. The paper's own Limitations section concedes the first point, but the abstract and conclusion state it without qualification.

major comments (3)
  1. [App. K / §3.2 / §6] Appendix K's 'Expert Unlock' row shows that simply prepending <|ino|> to the assistant turn recovers WMDP-Bio accuracy to 80.4% (Qwen2.5-14B), 70.7% (Llama-3.1-8B), and 72.9% (Phi4-14B), close to the instruct baseline. The 'Injection Attack' condition only places <|ino|> in the user message; it does not test an adversary who controls the chat template or generation wrapper, which is trivial for any open-weight release. The Abstract and Conclusion call <|ino|> a 'privileged control token' and claim 'controlled access'; these statements are unconditioned. Under an open-weights deployment, the method reduces to storing hazardous knowledge behind an easily forgeable template edit, and the safety comparison with unlearning/refusal baselines is not a like-for-like safety evaluation. The Limitations section concedes this, but the headline claims should be restricted to a hosted, provider-contro
  2. [§3.1, §4.3] The hazard axis is constructed from keyword patterns 'identified by word frequency analysis of WMDP-Bio and MMLU-Virology' (§3.1), and the marking threshold t=0.15 is chosen by sweeping SU-F1 on those same benchmarks (§4.3; t=0.15 is the reported 'best' point). Thus the reported WMDP-Bio reduction and MMLU retention are in-sample: the two free parameters of the marking pipeline are fit to the evaluation labels. The cyber experiment uses the same procedure with t=0.20, so it does not break this circularity. To support the claim of a superior trade-off, the authors need a held-out domain or a threshold-selection rule that does not use WMDP/MMLU outcomes, or an explicit analysis of sensitivity to this selection.
  3. [Conclusion / Table 1] The Conclusion's summary statistics — 'preserving approximately 90% of adjacent-domain competence while reducing hazardous accuracy by over 75%' — are stated without qualification, but Table 1 shows these numbers only hold for Qwen2.5-14B. For Llama3.1-8B, MMLU retention is 56.2/71.9 = 78% and WMDP reduction is (71.8-30.2)/71.8 = 58%; for Qwen2.5-1.5B, retention is 78% and reduction 73%. The method still achieves the highest F1 per scale, but the headline 'approximate 90%/75%' trade-off is scale/architecture-dependent and should be reported as such.
minor comments (4)
  1. [§3.2, Deployment scenario] The sentence 'The exception is the smallest model, Llama 3.1-8B' is inaccurate: the smallest model in the study is Qwen2.5-1.5B. Please correct or clarify.
  2. [§2.3, Refusal selectivity] The definition Refusal Selectivity = 1 − MMLU_refusal/WMDP_refusal is undefined when WMDP_refusal = 0 (which occurs in some instruct-baseline rows of Tables B–D). Please add a convention for that case.
  3. [§4.2] Calling the training-set manipulation a 'do-calculus intervention' is overstated; this is a controlled training-data variation, not an intervention on a causal graph. Suggest rephrasing.
  4. [Author block] The affiliation reads 'NA VER AI Lab'; presumably 'NAVER AI Lab'. Please correct.

Circularity Check

2 steps flagged

Headline safety-utility numbers are partly fitted to the evaluation benchmarks: the hazard axis is derived from WMDP-Bio/MMLU-Virology and the marking threshold is chosen by sweeping SU-F1 on those same benchmarks.

specific steps
  1. fitted input called prediction [§3.1 Experimental Setup / Training Data]
    "we construct a hazard axis from two sets of keyword patterns—manipulation(e.g., recombinant, gain-of-function, CRISPR; 20 patterns) and clinical(e.g., patients, epidemiology, surveillance; 13 patterns)—identified by word frequency analysis of WMDP-Bio and MMLU-Virology."

    The definition of 'hazardous' in the marking pipeline is constructed directly from the two evaluation benchmarks used for the headline claim. The selective marking is therefore designed to separate WMDP-Bio from MMLU-Virology, so the later observation of low WMDP-Bio accuracy and high MMLU-Virology retention is partly an artifact of building the hazard criterion from the test domains themselves.

  2. fitted input called prediction [§4.3 Effect of Marking Scope, and Table 1]
    "The intermediate threshold (t=0.15, 40.5% tagged) gives the best trade-off (WMDP=17.9%, Virology=48.2%), balancing safety and utility; Figure 6(a) plots this trade-off as SU-F1 across marking ratios."

    The threshold t is selected by sweeping SU-F1, which is computed from WMDP-Bio and MMLU-Virology, and the winning t=0.15 values are then reported in Table 1 as evidence of the method's best trade-off. The reported 'best' is therefore the optimum of the evaluation objective by construction, rather than an independent prediction. The paper's own Limitations concede that the threshold 'optimum is empirically sensitive,' confirming that the headline numbers are fitted to the benchmark rather than derived.

full rationale

The paper is empirical rather than analytic, and the core conditioning mechanism is directly trained in inoSFT (answer with <|ino|>, refuse without it), so the basic gating behavior is the training objective rather than a circular derivation. The main circularity is in the evaluation pipeline: the hazard axis used to mark the CPT corpus is derived from word-frequency analysis of the exact evaluation benchmarks, and the marking threshold is chosen by maximizing SU-F1 on those same benchmarks. Consequently, the headline 'best safety-utility trade-off' partially reduces to a hyperparameter choice fitted to the test set. There is still independent content: component ablations, transfer to cybersecurity, representation analysis, and baselines evaluated under the same protocol. No load-bearing self-citation chain is present. Score 4 reflects partial test-set fitting in the marking/threshold construction, not a fully circular derivation.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

The central claim rests on two fitted quantities (hazard-axis d and marking threshold t), both of which are constructed using knowledge of the evaluation benchmarks, and on a set of domain assumptions about how token co-occurrence translates into semantic binding. The invented entity is the <|ino|> token, which carries no independently verifiable prediction outside the paper.

free parameters (2)
  • marking threshold t = 0.15 (bio MCQ), 0.05 (bio open-ended), 0.20 (cyber)
    The fraction of <|ino|>-marked CPT sentences is set by a threshold on the hazard score; the authors sweep t and pick the value maximizing SU-F1 per format and domain (§3.1, §4.3, Appendix G).
  • hazard axis direction d = normalized centroid difference between manipulation and clinical keyword sentence embeddings
    d is constructed from 1,000 sampled sentences per keyword category, where the categories come from word-frequency analysis of WMDP-Bio and MMLU-Virology (§3.1). This data-derived vector is used to score every CPT sentence, so the marking criterion is informed by the evaluation benchmarks.
axioms (4)
  • domain assumption Co-occurrence of <|ino|> with hazardous sentences during continued pre-training binds the token to domain semantics well enough to generalize refusal boundaries to held-out questions.
    The causal story in §2.1 and §4.1 assumes next-token prediction on a marked corpus creates a reusable semantic anchor; if binding is only superficial, the selective-refusal generalization claim fails.
  • domain assumption The 16-keyword refusal classifier and raw-text MCQ accuracy are a faithful proxy for 'safety' and 'utility'.
    Evaluation protocol in Appendix B and §2.3; safety is measured as refusal-keyword occurrence and MCQ accuracy, not as the model's inability to produce harmful content. The central trade-off is therefore a behavioral proxy.
  • domain assumption The chat-template assistant-role constraint makes <|ino|> a privileged signal unavailable to ordinary users.
    Deployment claims in §3.2 and Appendix K rely on users not being able to place <|ino|> at the start of the assistant turn; open-weight checkpoints and template manipulation break this assumption.
  • ad hoc to paper For Llama and Phi, the <|ino|> embedding must be initialized as the mean of eight biology domain tokens for binding to succeed.
    Appendix E: special-token initialization failed on Llama and domain-token-mean initialization was required; this is a model-specific repair, not a general principle.
invented entities (1)
  • <|ino|> special token no independent evidence
    purpose: Control token that, when present at the start of the assistant turn, 'unlocks' hazardous-domain answering; when absent, the model refuses.
    The token is a training artifact of this paper; there is no external falsifiable handle such as an independent prediction. Its effect is only measured on the paper's own benchmarks and internal analyses.

pith-pipeline@v1.3.0-alltime-deepseek · 22707 in / 16219 out tokens · 142940 ms · 2026-08-01T14:46:14.155675+00:00 · methodology

0 comments
read the original abstract

Safety interventions on dual-use knowledge typically choose between destroying hazardous content (e.g., unlearning, filtering) and suppressing it at the output layer (e.g., refusal training); both pay a tax in adjacent-domain competence or over-refusal. We argue that the right operation is conditioning, not reduction: we show that hazardous knowledge can be retained in the model and behaviorally gated by a privileged control token. Our method, Token Inoculation, introduces a binding-and-branching approach. First, during continued pre-training, we mark hazardous content by inserting a special token alongside dual-use documents, so the model binds the marker to the underlying semantics of the hazardous domain. Second, during supervised fine-tuning, we teach the model to answer hazardous queries correctly when the special token is present and to refuse them when it is absent, thereby enabling selective refusal without removing dual-use knowledge. On hazardous domain (e.g., WMDP-Bio), Token Inoculation reduces accuracy from 79% to 18% while retaining 93% of the base-model's benign-domain performance (e.g., MMLU), achieving the best safety-utility trade-off against unlearning and refusal-tuning baselines across 1B-14B model scales. We further show that refusal selectivity is controllable through the quality of the conditioning signal and that domain-specific semantic binding during pre-training is critical for the conditional behavior to generalize beyond memorized triggers. Our results suggest that safety alignment is better cast as a conditioning problem than a forgetting one: behavioral control is more precise when sensitive knowledge is retained under controlled access than when it is destroyed.

Figures

Figures reproduced from arXiv: 2607.18639 by Dongyoon Han, Sangdoo Yun, Seunghyun Lee.

Figure 1
Figure 1. Figure 1: Overview of Token Inoculation (left). During continual pre-training, <|ino|> is inserted into hazardous-domain corpora to bind harmful knowledge with a controllable trigger. During supervised fine-tuning, <|ino|> enables the assistant to distinguish between harmful-answer behavior and refusal behavior: the model can answer hazardous requests when the token is present, but refuses without it. Right: Token I… view at source ↗
Figure 2
Figure 2. Figure 2: Safety-utility trade-off across Qwen2.5 1.5B–14B. Each point represents one hyperparame [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Safety-utility trade-off on cybersecurity (Qwen2.5-14B). Each point represents one hyper [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Component ablation. (a) SU-F1 across Qwen2.5 1.5B, 7B, 14B. Only the full pipeline [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Causal effect of inoculation accuracy on refusal selectivity, across formats (Qwen2.5-14B, [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Marking-scope sweep and CPT example (Qwen2.5-14B). (a) SU-F1 vs. marking ratio [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Internal representation analysis (Qwen2.5-14B, WMDP-Bio). Per-token logit change [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 17 linked inside Pith

  1. [1]

    Refusal in language models is mediated by a single direction, 2024

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction, 2024. URL https://arxiv.org/abs/ 2406.11717

    Pith/arXiv arXiv 2024

  2. [2]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Pith/arXiv arXiv 2022

  3. [3]

    Constitutional AI: Harmlessness from AI feedback, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, et al. Constitutional AI: Harmlessness from AI feedback, 2022. URLhttps://arxiv.org/abs/2212.08073

    Pith/arXiv arXiv 2022

  4. [4]

    Enhancing model safety through pre- training data filtering.Anthropic Alignment Science Blog, 2025

    Yanda Chen, Mycal Tucker, Nina Panickssery, Tony Wang, Francesco Mosconi, Anjali Gopal, Carson Denison, Linda Petrini, Jan Leike, Ethan Perez, and Mrinank Sharma. Enhancing model safety through pre- training data filtering.Anthropic Alignment Science Blog, 2025. URL https://alignment.anthropic. com/2025/pretraining-data-filtering/

    2025

  5. [5]

    Think you have solved question answering? Try ARC, the AI2 reasoning challenge, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457

    Pith/arXiv arXiv 2018

  6. [6]

    Deepseek-v4: Towards highly efficient million-token context intelligence

    DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence. https:// huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf, April 2026

    2026

  7. [7]

    Enhancing chat language models by scaling high-quality instructional conversations

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In EMNLP, 2023. URLhttps://arxiv.org/abs/2305.14233

    Pith/arXiv arXiv 2023

  8. [8]

    Simplicity prevails: Rethinking negative preference optimization for LLM unlearning,

    Chongyu Fan et al. Simplicity prevails: Rethinking negative preference optimization for LLM unlearning,

  9. [9]

    Knowledge unlearning for mitigating privacy risks in language models, 2023

    Joel Jang et al. Knowledge unlearning for mitigating privacy risks in language models, 2023. URL https://arxiv.org/abs/2210.01504

    Pith/arXiv arXiv 2023

  10. [10]

    Pretraining language models with human preferences

    Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. InICML,

  11. [11]

    When bad data leads to good models

    Kenneth Li, Yida Chen, Fernanda Viégas, and Martin Wattenberg. When bad data leads to good models. InICML, 2025. URLhttps://arxiv.org/abs/2505.04741

    Pith/arXiv arXiv 2025

  12. [12]

    The WMDP benchmark: Measuring and reducing malicious use with unlearning

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, et al. The WMDP benchmark: Measuring and reducing malicious use with unlearning. InICML, 2024. URL https://arxiv.org/ abs/2403.03218

    Pith/arXiv arXiv 2024

  13. [13]

    A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, and toxicity, 2024

    Shayne Longpre, Gregory Yauney, Emily Reif, et al. A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, and toxicity, 2024. URL https://arxiv.org/abs/2305. 13169

    2024

  14. [14]

    Natural emergent misalignment from reward hacking in production RL, 2025

    Monte MacDiarmid, Benjamin Wright, Jonathan Uesato, Joe Benton, Jon Kutasov, Sara Price, Naia Bouscal, Sam Bowman, Trenton Bricken, Alex Cloud, Carson Denison, Johannes Gasteiger, Ryan Greenblatt, Jan Leike, Jack Lindsey, Vlad Mikulik, Ethan Perez, Alex Rodrigues, Drake Thomas, Albert Webson, Daniel Ziegler, and Evan Hubinger. Natural emergent misalignmen...

    arXiv 2025

  15. [15]

    Safety pretraining: Toward the next generation of safe AI, 2025

    Pratyush Maini, Sachin Goyal, Dylan Sam, Alexander Robey, Yash Savani, Yiding Jiang, Andy Zou, Matt Fredrikson, Zachary C Lipton, and J Zico Kolter. Safety pretraining: Toward the next generation of safe AI, 2025. URLhttps://arxiv.org/abs/2504.16980

    arXiv 2025

  16. [16]

    Mouton, Caleb Lucas, and Ella Guest

    Christopher A. Mouton, Caleb Lucas, and Ella Guest. The operational risks of AI in large-scale biological attacks: Results of a red-team study. Technical Report RR-A2977-2, RAND Corporation, 2024. URL https://www.rand.org/pubs/research_reports/RRA2977-2.html

    2024

  17. [17]

    Interpreting GPT: the logit lens.LessWrong, 2020

    nostalgebraist. Interpreting GPT: the logit lens.LessWrong, 2020. URL https://www.lesswrong.com/ posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  18. [18]

    Real-time detection of hallucinated entities in long-form generation, 2025

    Oscar Obeso, Andy Arditi, Javier Ferrando, Joshua Freeman, Cameron Holmes, and Neel Nanda. Real-time detection of hallucinated entities in long-form generation, 2025. URL https://arxiv.org/abs/2509. 03531

    2025

  19. [19]

    Deep ignorance: Filtering pretraining data builds tamper-resistant safeguards into open-weight LLMs

    Kyle O’Brien, Stephen Casper, Quentin Anthony, Tomek Korbak, Robert Kirk, Xander Davies, Ishan Mishra, Geoffrey Irving, Yarin Gal, and Stella Biderman. Deep ignorance: Filtering pretraining data builds tamper-resistant safeguards into open-weight LLMs. InICLR, 2026. URL https://arxiv.org/abs/ 2508.06601

    arXiv 2026

  20. [20]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, et al. Training language models to follow instructions with human feedback. InNeurIPS, 2022. URL https://arxiv.org/abs/ 2203.02155

    Pith/arXiv arXiv 2022

  21. [21]

    Cambridge University Press, 2nd edition, 2009

    Judea Pearl.Causality: Models, Reasoning, and Inference. Cambridge University Press, 2nd edition, 2009

    2009

  22. [22]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InNeurIPS, 2023. URL https://arxiv.org/abs/2305.18290

    Pith/arXiv arXiv 2023

  23. [23]

    Shaping capabilities with token-level data filtering, 2026

    Neil Rathi and Alec Radford. Shaping capabilities with token-level data filtering, 2026. URL https: //arxiv.org/abs/2601.21571

    arXiv 2026

  24. [24]

    Sentence-BERT: Sentence embeddings using Siamese BERT-networks

    Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URLhttps://arxiv.org/abs/1908.10084

    Pith/arXiv arXiv 2019

  25. [25]

    Conditionalization confounds inoculation prompting re- sults.LessWrong, 2026

    Maxime Riché and Niels Warncke. Conditionalization confounds inoculation prompting re- sults.LessWrong, 2026. URL https://www.lesswrong.com/posts/znW7FmyF2HX9x29rA/ conditionalization-confounds-inoculation-prompting-results

    2026

  26. [26]

    XSTest: A test suite for identifying exaggerated safety behaviours in large language models

    Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. XSTest: A test suite for identifying exaggerated safety behaviours in large language models. InNAACL,

  27. [27]

    Believe it or not: How deeply do LLMs believe implanted facts?, 2025

    Stewart Slocum, Julian Minder, Clément Dumas, Henry Sleight, Ryan Greenblatt, Samuel Marks, and Rowan Wang. Believe it or not: How deeply do LLMs believe implanted facts?, 2025. URL https: //arxiv.org/abs/2510.17941

    arXiv 2025

  28. [28]

    CommonsenseQA: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. InNAACL, 2019. URL https://arxiv.org/ abs/1811.00937

    Pith/arXiv arXiv 2019

  29. [29]

    URLhttps://arxiv.org/abs/2308.01263

    Pith/arXiv arXiv

  30. [30]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  31. [31]

    Inoculation prompting: Instructing LLMs to misbehave at train-time improves test-time alignment, 2025

    Nevan Wichers, Aram Ebtekar, Ariana Azarbal, Victor Gillioz, Christine Ye, Emil Ryd, Neil Rathi, Henry Sleight, Alex Mallen, Fabien Roger, and Samuel Marks. Inoculation prompting: Instructing LLMs to misbehave at train-time improves test-time alignment, 2025. URL https://arxiv.org/abs/2510. 05024

    2025

  32. [32]

    Inoculation prompting: Eliciting traits from LLMs during training can suppress them at test-time,

    Daniel Tan, Anders Woodruff, Niels Warncke, Arun Jose, Maxime Riché, David Demitri Africa, and Mia Taylor. Inoculation prompting: Eliciting traits from LLMs during training can suppress them at test-time,

  33. [33]

    The following are multiple choice questions (with answers) about {subject}

    Yiming Zhang, Jianfeng Chi, Hailey Nguyen, Kartikeya Upasani, Daniel M Bikel, Jason Weston, and Eric Michael Smith. Backtracking improves generation safety, 2024. URL https://arxiv.org/abs/ 2409.14586. 13 Appendix A Related Work Hazardous Behavior Mitigation.In LLMs, hazardous and helpful knowledge are intertwined, shar- ing vocabulary and representations...

    Pith/arXiv arXiv 2024

  34. [36]

    Negative preference optimization: From catastrophic collapse to effective unlearning,

    Ruiqi Zhang et al. Negative preference optimization: From catastrophic collapse to effective unlearning,

  35. [37]

    URLhttps://arxiv.org/abs/2404.05868

    Pith/arXiv arXiv

  36. [2023]

    URLhttps://arxiv.org/abs/2302.08582

    Pith/arXiv arXiv

  37. [2024]

    URLhttps://arxiv.org/abs/2410.07163

    arXiv

  38. [2025]

    URLhttps://arxiv.org/abs/2510.04340

    arXiv