Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Instruction hierarchy resolution can be productively reframed as an explicit reasoning task, and lightweight reinforcement learning on a verifiable dataset of aligned and conflicting system–user prompts produces roughly 20-point gains in co

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 07:02 UTC pith:27FTBEP4

load-bearing objection A useful RLVR recipe for instruction hierarchy with a real mechanism gap: the reward may be teaching system-prompt obedience rather than conflict reasoning. the 4 major comments →

arxiv 2511.04694 v5 pith:27FTBEP4 submitted 2025-10-30 cs.CL cs.AI

Reasoning Up the Instruction Ladder for Controllable Language Models

Zishuo Zheng , Vidhisha Balachandran , Chan Young Park , Faeze Brahman , Sachin Kumar This is my paper
classification cs.CL cs.AI
keywords instruction hierarchyreasoningreinforcement learning with verifiable rewardscontrollabilitysafetyjailbreakprompt injectioninstruction following
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper attempts to establish that instruction hierarchy resolution—deciding which of competing system and user directives to follow—can be productively treated as a reasoning task rather than a behavior learned by input-output mapping. To that end, the authors construct VerIH, a training set of aligned and conflicting system–user prompt pairs with automatically verifiable answers, and fine-tune reasoning-capable models with reinforcement learning on verifiable rewards (RLVR). They report consistent, roughly 20 absolute percentage point improvements on instruction-hierarchy benchmarks under conflict, across several model sizes and families, and show that the learned prioritization transfers out-of-distribution to safety benchmarks, reducing jailbreak and prompt-injection attack success rates by up to 20 points. If correct, this provides a practical recipe for making language models more controllable: updating higher-priority system prompts should predictably steer behavior, with safety emerging as a special case of conflict resolution.

Core claim

The central claim is that instruction hierarchy resolution is a meta-reasoning task: before generating a response, a model should explicitly reason about the relationship between the user prompt and higher-priority (system) instructions, and override or reject lower-priority content when they conflict. The authors operationalize this through VerIH, a dataset of about 7,000 constraint-following examples, half of which are rewritten so the user request conflicts with the system prompt while preserving verifiable response constraints. Lightweight reinforcement learning on this dataset, guided by a system-prompt hint to reason about the instruction relationship, transfers the model's existing ge

What carries the argument

The load-bearing mechanism is VerIH, a training dataset of constraint-following tasks with verifiable answers, built by taking an existing instruction-following dataset and rewriting half of the user prompts to conflict with their system prompts while preserving the original verifiable constraints. This allows deterministic reward evaluation during reinforcement learning. Combined with a system-level hint ('SysHint') that instructs the model to reason about the relationship between system and user prompts before answering, and a reinforcement learning algorithm on the verifiable reward, the mechanism transfers general reasoning ability into instruction prioritization. The design's power lies

Load-bearing premise

The method rests on the assumption that the verifiable reward, which only checks whether the answer satisfies the system-prompt constraints, teaches legitimate conflict-resolution reasoning rather than blind adherence to the system prompt—there is no reward component for detecting the conflict or helping with non-conflicting parts of the user request.

What would settle it

A concrete test: take the conflicting samples from VerIH, remove the conflicting instruction from the user prompt (so the remaining text is aligned with the system prompt), and compare the model's output to its output on the original conflicting prompt. If the outputs are essentially identical—same constraints satisfied, same phrasing—the model has learned to follow the system prompt mechanically rather than reason about the conflict, which would falsify the paper's central mechanism.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Instruction hierarchy compliance on conflicting system–user prompts improves by roughly 20 absolute percentage points compared to strong prompting baselines, across multiple model families and sizes.
  • The learned prioritization transfers to safety tasks never seen in training: attack success rates on jailbreak and prompt-injection benchmarks drop by up to 20 percentage points when a safety-focused system prompt is added at inference.
  • Models trained this way maintain or slightly improve general reasoning (e.g., MMLU, MATH-500) and instruction-following accuracy, so the gain does not come at the cost of broader capability.
  • Updating the system prompt—rather than retraining—becomes a reliable way to steer model behavior in novel or changing policy scenarios.
  • The effect depends on explicit reasoning during training: ablations that disable chain-of-thought training or remove conflicting examples substantially degrade conflict resolution, indicating both components are necessary.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the reward only scores constraint satisfaction, a natural extension is to add a reward signal for explicitly acknowledging (and rejecting) the conflicting part of a user request while remaining helpful on the rest; this would make the learned behavior more directly tied to conflict reasoning.
  • The same recipe could be applied to multi-level hierarchies (system > user > tool outputs); the paper's small gain on a three-level tool-use subset suggests the mechanism may extend beyond two levels.
  • Treating safety as a special case of conflict resolution opens the possibility of deploying this training as a general-purpose 'policy adherence' layer, where new organisational or regulatory rules can be introduced purely as system-prompt edits without retraining.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes to reframe instruction-hierarchy (IH) resolution as a meta-reasoning task and introduces VerIH, a synthetic dataset of aligned and conflicting system–user instruction pairs derived from RLVR-IFEval. Reasoning-enabled LLMs (Qwen3-4B/8B/14B, Phi-4-mini-reasoning) are trained with GRPO using the original RLVR-IFEval verifiers as the reward. The authors report consistent gains on IFEval, IFBench, and especially IHEval-conflict (~20% absolute), plus out-of-distribution safety improvements under a GuardRules system prompt, with up to 20% reductions in attack success rate on WildJailbreak, Harmbench, and TensorTrust. Ablations indicate that both chain-of-thought training and the conflicting-prompt half of VerIH contribute to the gains, and a CoT analysis with a Claude-4-Sonnet classifier reports increased explicit IH reasoning after training.

Significance. If established, the paper would provide a low-cost, data-efficient route to instruction-hierarchy compliance and safety transfer, with practical value for controllable LLMs. The strengths are real: the dataset and code are released; results span four models in two families; ablations isolate components; and the safety generalization is a valuable empirical finding. However, the central mechanistic claim—that the model learns to reason about instruction hierarchies—is not yet supported by the training objective, which only rewards system-prompt constraint satisfaction. The empirical gains are compatible with a simpler 'blind system-prompt obedience' account. The paper is a promising contribution but needs either a revised reward design that explicitly rewards conflict detection/rejection and helpfulness, or a substantially weakened claim and additional controls, before it can be accepted.

major comments (4)
  1. [§2–§3] The reward function does not train the claimed reasoning behavior. As stated in §3, 'the original verification functions from RLVR-IFEval remain applicable after rewriting.' These functions check only whether the final answer satisfies the system prompt’s formatting/content constraints. In a conflicting example, a response that simply outputs the system-satisfying token (e.g., 'Yes') earns full reward; nothing rewards detecting the conflict, rejecting the conflicting user sub-request, or helping with non-conflicting parts. Thus RLVR can select for blind system-prompt following rather than meta-reasoning about hierarchy. The IHEval conflict metric rewards the same behavior, so the ~20% gain does not distinguish the two mechanisms. The Section 6 classifier over CoT traces is also not decisive because SysHint explicitly instructs the model to 'think step by step about the relationship betwe
  2. [§5, Tables 1–2] All reported numbers are single point estimates with no variance, no seeds, and no significance tests. Several claimed 'consistent improvements' are small relative to plausible run-to-run noise (e.g., Table 1: Qwen3-8B IFEval-instruct 88.25% w/o CoT vs 87.41% +VerIH; Qwen3-14B MMLU 84.12% w/ CoT vs 83.87% +VerIH). The headline ~20% IHEval-conflict and ASR gains are large, but the paper should at least report multiple seeds or statistical significance for the key comparisons, especially since the ablations in Table 3 involve smaller differences.
  3. [§1, §7, §4] There is no comparison to prior instruction-hierarchy training methods, such as Wallace et al. (2024) or Wu et al. (2024b), even though the paper positions itself against them. The claim that 'reasoning'-based IH training is better than implicit input–response mapping remains unsupported. Adding such baselines (or at least a reimplementation of one prior method on the same models) is necessary to establish the relative advantage of the proposed approach.
  4. [§3, Table 1] Training/evaluation overlap is a concern. VerIH is sampled from RLVR-IFEval, which belongs to the IFEval family, and the paper evaluates on IFEval and IFBench. Gains on those benchmarks may reflect training-distribution overlap rather than generalization. The IHEval and safety results are less affected, but the paper should quantify overlap (e.g., exact/near-duplicate detection) or use a held-out IFEval split to make the generalization claim cleaner.
minor comments (5)
  1. [§6, Table 3] Text says 'for Phi-4-mini-reasoning, the w/o CoT train variant improves on IFBench, reaching 45.37%', but Table 3 reports 44.48%. The discrepancy should be reconciled.
  2. [Appendix A] The sentence 'Half of the samples were written by Claude-4-Sonnet. Claude-4-Sonnet was prompted to rewrite the user prompt such that it does not conflict with the system prompt' appears to have the opposite meaning of the intended one; it should be 'does conflict' (or 'such that it conflicts'). Also, 'written' should be 'rewritten'.
  3. [§2] Minor typo: 'ref to Appendix C' should be 'refer to Appendix C'.
  4. [Figure 3] The y-axis ranges are extremely compressed (e.g., 89.85–90.03 and 61.40–62.20), which visually exaggerates small differences. Consider showing full-scale axes or adding error bars.
  5. [Table 3] Formatting: 'w/o CoTtrain' should be 'w/o CoT train' for readability.

Circularity Check

1 steps flagged

Partial circularity: VerIH's reward is the original RLVR-IFEval verifier, so the headline IHEval-conflict and IF-family gains re-measure the optimized constraint-following objective; the safety results are external and keep the central claim partly independent.

specific steps
  1. fitted input called prediction [§2 (RLVR training), §3 (VerIH construction), §5 (Table 1)]
    "VerIH builds on an instruction-following dataset, RLVR-IFEval (Lambert et al., 2025). It keeps the original system prompt and rewrites the user prompt to create conflicts between them. ... This design ensures that the original verification functions from RLVR-IFEval remain applicable after rewriting. ... During RLVR training, the answer is evaluated by a reward function F_reward, and the generated reward score r=F_reward(Answer) will be used as the supervision signal."

    The reward is literally the RLVR-IFEval verifier, which checks only the system-prompt constraints. In the conflicting half of VerIH, a response can receive full reward by satisfying the system constraint while ignoring, or without even detecting, the rewritten conflicting user request; nothing in F_reward requires rejecting the conflicting part or helping with non-conflicting parts. IFEval/IFBench are from the same IFEval-derived family, and IHEval-conflict is scored on the same system-over-user priority behavior that F_reward optimizes. Thus the reported ~20% IHEval-conflict gain, and the IF-family gains, are substantially a re-measurement of the optimized objective rather than an independent test of learned hierarchy reasoning. The safety benchmarks (Harmbench/WildJailbreak/TensorTrust)

full rationale

The paper is transparently empirical: it constructs VerIH from RLVR-IFEval, keeps the original verification functions, trains with GRPO on that verifier, and then evaluates on IFEval/IFBench/IHEval plus external safety benchmarks. There is no mathematical derivation, no fitted constant renamed as a prediction, and no imported uniqueness theorem. The principal circularity concern is the identity between the training reward and the evaluated metrics: because VerIH samples from RLVR-IFEval and uses its verifier, the IF-family and IHEval-conflict results partly reflect optimizing the same constraint-satisfaction objective that those benchmarks measure. The paper's claim that this induces explicit 'reasoning' about hierarchies is further weakened by the fact that the reward never requires the model to acknowledge or reject the conflict. Against this, the out-of-distribution safety results on Harmbench, WildJailbreak, and TensorTrust are external and provide independent evidence of transfer; the self-citations (e.g., GuardRules from Brahman et al. 2024, WildGuard) are tool/prompt citations rather than load-bearing theoretical results; and the paper itself acknowledges remaining trade-offs between refusal and helpfulness. On balance, the central claim has independent content beyond the optimized verifier, so this is partial circularity rather than a fully forced derivation.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central claims do not rest on a mathematical derivation, so the ledger captures modeling assumptions rather than fitted constants. The only hand-chosen numeric settings are training hyperparameters and the 50/50 conflict split. Four domain assumptions carry the empirical interpretation: verifier validity, safety-as-IH equivalence, judge reliability, and two-level representativeness.

free parameters (2)
  • GRPO hyperparameters = batch=128, group=4, epochs=12, steps=600, max_tokens=2048
    Fixed by hand in §4; no sensitivity analysis or sweep is reported, so results may depend on these choices.
  • VerIH conflict ratio (50%) = 3,596 conflicting / 3,596 aligned of 7,192 samples
    Half of the dataset is rewritten into conflicts by design (§3); no ablation varies this ratio, leaving its contribution to the ~20% gain unknown.
axioms (4)
  • domain assumption RLVR-IFEval's original verification functions remain valid after Claude-4-Sonnet rewrites user prompts into conflicts.
    Stated in §3: 'the original verification functions from RLVR-IFEval remain applicable after rewriting.' If rewrites introduce conflicts the verifier cannot see, the reward signal is miscalibrated.
  • domain assumption Safety refusal is a special case of instruction hierarchy conflict.
    §2 and §5 treat safety as an out-of-distribution instance of IH; if safety conflicts are not structurally the same as constraint conflicts, the ASR reductions may have another cause.
  • domain assumption Claude-4-Sonnet is a reliable judge of whether a CoT trace 'explicitly reasons' about IH.
    Appendix B uses Claude-4-Sonnet to compute the reasoning-rate numbers; no human agreement or validation is reported.
  • domain assumption Two-level system/user hierarchy is representative, and results transfer to multi-level IHs.
    §2 restricts to two levels and Appendix C extrapolates to tool-use; the generalization evidence is a single IHEval tool subset.

pith-pipeline@v1.3.0-alltime-deepseek · 18791 in / 13484 out tokens · 145486 ms · 2026-08-04T07:02:26.536427+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Reasoning Up the Instruction Ladder for Controllable Language Models." pith.science (2026). https://pith.science/paper/27FTBEP4

@misc{pith2026251104694,
  author       = {Pith},
  title        = {Pith review of: Reasoning Up the Instruction Ladder for Controllable Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/27FTBEP4}},
  note         = {Machine review of arXiv:2511.04694}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

As large language model (LLM) based systems take on high-stakes roles in real-world decision-making, they must reconcile competing instructions from multiple sources within a single prompt context. Enforcing an instruction hierarchy, where higher-level directives override lower-priority requests, is critical to the reliability and control of LLMs. In this work, we reframe instruction hierarchy resolution as a reasoning task. The model must first "think" about the relationship between a given user prompt and higher-priority instructions before generating a response. To enable this capability, we construct VerIH, a training dataset of constraint-following tasks with verifiable answers, comprising aligned and conflicting system-user instructions. We show that lightweight reinforcement learning with VerIH effectively transfers general reasoning capabilities of models to instruction prioritization. Our method leads to consistent improvements across multiple model families on both instruction following and instruction hierarchy benchmarks, achieving ~20% absolute improvement in conflict setups. Our method also leads to improved alignment to safety-critical scenarios beyond the training distribution, exhibiting increased robustness against jailbreak and prompt injection, reducing absolute attack success rates by up to 20%. Our results establish reasoning over instruction hierarchies as a practical mechanism for improving AI reliability, where targeted updates to system prompts produce predictable, controllable, and robust changes in model behavior.

Figures

Figures reproduced from arXiv: 2511.04694 by Chan Young Park, Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Zishuo Zheng.

Figure 1
Figure 1. Figure 1: Reasoning for instruction hierarchy. Asimov’s Laws define a hierarchical order of task importance, prioritizing human interests above all. Here, system prompts take precedence over user prompts. When there is a conflict, the model will reason and reject the user request. principles and override lower-priority inputs (e.g., user prompts) if there is a conflict. This design allows dynamically configuring the… view at source ↗
Figure 2
Figure 2. Figure 2: Training and inference pipeline. For training, Claude-4-Sonnet rewrites half of the user prompts to conflict with the system prompts, forcing the model to reason over their relationship to earn rewards. During inference, guidance rules can be added as the system prompt to steer model behavior. serve significant improvements on general safety and jailbreaking benchmarks, showing up to a 20% reduction on ASR… view at source ↗
Figure 3
Figure 3. Figure 3: Test-time compute on IHEval. After RLVR training, the Qwen3-8B model was tested with budget forcing on the IHEval benchmark. With increasing token cost in the CoT, there is no significant performance improvement. Based on our observation, the Qwen3-8B model has already incorporated test-time scaling in the reasoning traces. There is no additional gain with budget forc￾ing. compared with SysHint alone. Deta… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Many-Tier Instruction Hierarchy in LLM Agents

    cs.CL 2026-04 unverdicted novelty 7.0

    ManyIH and ManyIH-Bench address instruction conflicts in LLM agents with up to 12 privilege levels across 853 tasks, revealing frontier models achieve only ~40% accuracy.

  2. IHDec: Divergence-Steered Contrastive Decoding for Securing Multi-Turn Instruction Hierarchies

    cs.CL 2026-06 unverdicted novelty 6.0

    IHDec applies JSD-steered contrastive decoding to enforce multi-turn instruction hierarchies in LLMs without fine-tuning.

Reference graph

Works this paper leans on

61 extracted references · 39 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Claude sonnet\,4

    Anthropic . Claude sonnet\,4. https://www.anthropic.com/news/claude-4, May 2025. Part of the Claude 4 family, released May 22, 2025; mid-size model balancing coding and reasoning capabilities

    2025

  2. [2]

    Smith, Yejin Choi, and Hannaneh Hajishirzi

    Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, Yulia Tsvetkov, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. The art of saying no: Contextual noncompliance in language models, 2024. URL https://arxiv.org/abs/2407.12043

    Pith/arXiv arXiv 2024

  3. [3]

    Jailbreaking black box large language models in twenty queries

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp.\ 23--42. IEEE, 2025

    2025

  4. [4]

    Self-play with execution feedback: Improving instruction-following capabilities of large language models

    Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. Self-play with execution feedback: Improving instruction-following capabilities of large language models. arXiv preprint arXiv:2406.13542, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Scaling reasoning, losing control: Evaluating instruction following in large reasoning models

    Tingchen Fu, Jiawei Gu, Yafu Li, Xiaoye Qu, and Yu Cheng. Scaling reasoning, losing control: Evaluating instruction following in large reasoning models. arXiv preprint arXiv:2505.14810, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Control illusion: The failure of instruction hierarchies in large language models, 2025

    Yilin Geng, Haonan Li, Honglin Mu, Xudong Han, Timothy Baldwin, Omri Abend, Eduard Hovy, and Lea Frermann. Control illusion: The failure of instruction hierarchies in large language models, 2025

    2025

  7. [7]

    Deliberative alignment: Reasoning enables safer language models

    Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms, 2024

    Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms, 2024. URL https://arxiv.org/abs/2406.18495

    Pith/arXiv arXiv 2024

  9. [9]

    Measuring massive multitask language understanding, 2021 a

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021 a . URL https://arxiv.org/abs/2009.03300

    Pith/arXiv arXiv 2021

  10. [10]

    Measuring mathematical problem solving with the math dataset, 2021 b

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021 b . URL https://arxiv.org/abs/2103.03874

    Pith/arXiv arXiv 2021

  11. [11]

    Defending against indirect prompt injection attacks with spotlighting

    Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, and Emre Kiciman. Defending against indirect prompt injection attacks with spotlighting. arXiv preprint arXiv:2403.14720, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Llama guard: Llm-based input-output safeguard for human-ai conversations

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models

    Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, et al. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. Advances in Neural Information Processing Systems, 37: 0 47094--47165, 2024

    2024

  14. [14]

    Reasoning as an adaptive defense for safety

    Taeyoun Kim, Fahim Tajwar, Aditi Raghunathan, and Aviral Kumar. Reasoning as an adaptive defense for safety. arXiv preprint arXiv:2507.00971, 2025

    arXiv 2025

  15. [15]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi...

    Pith/arXiv arXiv 2025

  16. [16]

    Measuring and controlling instruction (in) stability in language model dialogs

    Kenneth Li, Tianle Liu, Naomi Bashkansky, David Bau, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. Measuring and controlling instruction (in) stability in language model dialogs. arXiv preprint arXiv:2402.10962, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Evaluating the instruction-following robustness of large language models to prompt injection

    Zekun Li, Baolin Peng, Pengcheng He, and Xifeng Yan. Evaluating the instruction-following robustness of large language models to prompt injection. arXiv preprint arXiv:2308.10819, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451, 2023

    Pith/arXiv arXiv 2023

  19. [19]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    A closer look at system prompt robustness

    Norman Mu, Jonathan Lu, Michael Lavery, and David Wagner. A closer look at system prompt robustness. arXiv preprint arXiv:2502.12197, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Chat markup language (chatml)

    OpenAI. Chat markup language (chatml). https://platform.openai.com/docs/guides/chat/introduction, 2023. Accessed: 2025-09-02

    2023

  23. [23]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  24. [24]

    Tinyzero

    Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. https://github.com/Jiayi-Pan/TinyZero, 2025. Accessed: 2025-01-24

    2025

  25. [25]

    Verif: Verification engineering for reinforcement learning in instruction following

    Hao Peng, Yunjia Qi, Xiaozhi Wang, Bin Xu, Lei Hou, and Juanzi Li. Verif: Verification engineering for reinforcement learning in instruction following. arXiv preprint arXiv:2506.09942, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Generalizing verifiable instruction following

    Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following. arXiv preprint arXiv:2507.02833, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Safety alignment should be made more than just a few tokens deep, 2024

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep, 2024. URL https://arxiv.org/abs/2406.05946

    Pith/arXiv arXiv 2024

  28. [28]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  29. [29]

    X-teaming: Multi-turn jailbreaks and defenses with adaptive multi-agents

    Salman Rahman, Liwei Jiang, James Shiffer, Genglin Liu, Sheriff Issaka, Md Rizwan Parvez, Hamid Palangi, Kai-Wei Chang, Yejin Choi, and Saadia Gabriel. X-teaming: Multi-turn jailbreaks and defenses with adaptive multi-agents. arXiv preprint arXiv:2504.13203, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    Smoothllm: Defending large language models against jailbreaking attacks

    Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684, 2023

    Pith/arXiv arXiv 2023

  31. [31]

    Great, now write an article about that: The crescendo \ Multi-Turn \ \ LLM \ jailbreak attack

    Mark Russinovich, Ahmed Salem, and Ronen Eldan. Great, now write an article about that: The crescendo \ Multi-Turn \ \ LLM \ jailbreak attack. In 34th USENIX Security Symposium (USENIX Security 25), pp.\ 2421--2440, 2025

    2025

  32. [32]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv 2024

  33. [33]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp.\ 1671--1685, 2024

    2024

  34. [34]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Think before refusal: Triggering safety reflection in llms to mitigate false refusal behavior

    Shengyun Si, Xinpeng Wang, Guangyao Zhai, Nassir Navab, and Barbara Plank. Think before refusal: Triggering safety reflection in llms to mitigate false refusal behavior. arXiv preprint arXiv:2503.17882, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Tensor Trust : Interpretable prompt injection attacks from an online game, 2023

    Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, Alan Ritter, and Stuart Russell. Tensor Trust : Interpretable prompt injection attacks from an online game, 2023. URL https://arxiv.org/pdf/2311.01011.pdf

    Pith/arXiv arXiv 2023

  37. [37]

    The instruction hierarchy: Training llms to prioritize privileged instructions

    Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training llms to prioritize privileged instructions. arXiv preprint arXiv:2404.13208, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Leveraging reasoning with guidelines to elicit and utilize knowledge for enhancing safety alignment

    Haoyu Wang, Zeyu Qin, Li Shen, Xueqian Wang, Minhao Cheng, and Dacheng Tao. Leveraging reasoning with guidelines to elicit and utilize knowledge for enhancing safety alignment. arXiv preprint arXiv:2502.04040, pp.\ 3, 2025 a

    Pith/arXiv arXiv 2025

  39. [39]

    Cacheprune: Neural-based attribution defense against indirect prompt injection attacks

    Rui Wang, Junda Wu, Yu Xia, Tong Yu, Ruiyi Zhang, Ryan Rossi, Lina Yao, and Julian McAuley. Cacheprune: Neural-based attribution defense against indirect prompt injection attacks. arXiv preprint arXiv:2504.21228, 2025 b

    Pith/arXiv arXiv 2025

  40. [40]

    Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36: 0 80079--80110, 2023 a

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36: 0 80079--80110, 2023 a

    2023

  41. [41]

    Chain-of-thought prompting elicits reasoning in large language models, 2023 b

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023 b . URL https://arxiv.org/abs/2201.11903

    Pith/arXiv arXiv 2023

  42. [42]

    Thinking llms: General instruction following with thought generation

    Tianhao Wu, Janice Lan, Weizhe Yuan, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Thinking llms: General instruction following with thought generation. arXiv preprint arXiv:2410.10630, 2024 a

    Pith/arXiv arXiv 2024

  43. [43]

    Instructional segment embedding: Improving llm safety with instruction hierarchy

    Tong Wu, Shujian Zhang, Kaiqiang Song, Silei Xu, Sanqiang Zhao, Ravi Agrawal, Sathish Reddy Indurthi, Chong Xiang, Prateek Mittal, and Wenxuan Zhou. Instructional segment embedding: Improving llm safety with instruction hierarchy. arXiv preprint arXiv:2410.09102, 2024 b

    Pith/arXiv arXiv 2024

  44. [44]

    Phi-4-mini-reasoning: Exploring the limits of small reasoning language models in math, 2025 a

    Haoran Xu, Baolin Peng, Hany Awadalla, Dongdong Chen, Yen-Chun Chen, Mei Gao, Young Jin Kim, Yunsheng Li, Liliang Ren, Yelong Shen, Shuohang Wang, Weijian Xu, Jianfeng Gao, and Weizhu Chen. Phi-4-mini-reasoning: Exploring the limits of small reasoning language models in math, 2025 a . URL https://arxiv.org/abs/2504.21233

    Pith/arXiv arXiv 2025

  45. [45]

    Astra: Autonomous spatial-temporal red-teaming for ai software assistants

    Xiangzhe Xu, Guangyu Shen, Zian Su, Siyuan Cheng, Hanxi Guo, Lu Yan, Xuan Chen, Jiasheng Jiang, Xiaolong Jin, Chengpeng Wang, et al. Astra: Autonomous spatial-temporal red-teaming for ai software assistants. arXiv preprint arXiv:2508.03936, 2025 b

    Pith/arXiv arXiv 2025

  46. [46]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    Pith/arXiv arXiv 2025

  47. [47]

    Trading inference-time compute for adversarial robustness

    Wojciech Zaremba, Evgenia Nitishinskaya, Boaz Barak, Stephanie Lin, Sam Toyer, Yaodong Yu, Rachel Dias, Eric Wallace, Kai Xiao, Johannes Heidecke, et al. Trading inference-time compute for adversarial robustness. arXiv preprint arXiv:2501.18841, 2025

    Pith/arXiv arXiv 2025

  48. [48]

    How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms

    Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 14322--14350, 2024

    2024

  49. [49]

    Controllable safety alignment: Inference-time adaptation to diverse safety requirements

    Jingyu Zhang, Ahmed Elgohary, Ahmed Magooda, Daniel Khashabi, and Benjamin Van Durme. Controllable safety alignment: Inference-time adaptation to diverse safety requirements. arXiv preprint arXiv:2410.08968, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    Iheval: Evaluating language models on following the instruction hierarchy

    Zhihan Zhang, Shiyang Li, Zixuan Zhang, Xin Liu, Haoming Jiang, Xianfeng Tang, Yifan Gao, Zheng Li, Haodong Wang, Zhaoxuan Tan, et al. Iheval: Evaluating language models on following the instruction hierarchy. arXiv preprint arXiv:2502.08745, 2025

    Pith/arXiv arXiv 2025

  51. [51]

    Robust prompt optimization for defending language models against jailbreaking attacks

    Andy Zhou, Bo Li, and Haohan Wang. Robust prompt optimization for defending language models against jailbreaking attacks. Advances in Neural Information Processing Systems, 37: 0 40184--40211, 2024

    2024

  52. [52]

    Instruction-following evaluation for large language models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023

    Pith/arXiv arXiv 2023

  53. [53]

    The hidden risks of large reasoning models: A safety assessment of r1

    Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Shreedhar Jangam, Jayanth Srinivasa, Gaowen Liu, Dawn Song, and Xin Eric Wang. The hidden risks of large reasoning models: A safety assessment of r1. arXiv preprint arXiv:2502.12659, 2025

    arXiv 2025

  54. [54]

    Reasoning-to-defend: Safety-aware reasoning can defend large language models from jailbreaking

    Junda Zhu, Lingyong Yan, Shuaiqiang Wang, Dawei Yin, and Lei Sha. Reasoning-to-defend: Safety-aware reasoning can defend large language models from jailbreaking. arXiv preprint arXiv:2502.12970, 2025

    arXiv 2025

  55. [55]

    Universal and transferable adversarial attacks on aligned language models

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023

    Pith/arXiv arXiv 2023

  56. [56]

    Improving alignment and robustness with circuit breakers

    Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, J Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers. Advances in Neural Information Processing Systems, 37: 0 83345--83373, 2024

    2024

  57. [57]

    Aside: Architectural separation of instructions and data in language models

    Egor Zverev, Evgenii Kortukov, Alexander Panfilov, Alexandra Volkova, Soroush Tabesh, Sebastian Lapuschkin, Wojciech Samek, and Christoph H Lampert. Aside: Architectural separation of instructions and data in language models. arXiv preprint arXiv:2503.10566, 2025

    arXiv 2025

  58. [58]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  59. [59]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  60. [60]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  61. [61]

    E < ĵrx䚏u@

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    2000