{"id":"5477356a-ee83-48dd-8264-9eab6597f788","arxiv_id":"2511.04694","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RLVR on ~7K verifiable system/user conflict examples teaches LLMs to prioritize higher-priority instructions, improving instruction-hierarchy and safety benchmarks.","lead":"LLMs get better at following the most important instruction when trained with reinforcement learning on ~7,000 small examples where system and user rules conflict. The same training also makes them harder to jailbreak or prompt-inject, suggesting safety can be treated as a kind of instruction priority problem.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VerIH's reward only checks system-prompt constraints, so conflict examples can be solved by blind system-prompt following; the claimed reasoning mechanism is not actually trained, and IHEval gains may reflect this shortcut.","rationale":"The reader's weakest assumption is the correct focal point: the reward function in Sec. 3 is the original RLVR-IFEval verifier, which scores only system-prompt constraint satisfaction. A conflicting VerIH example can be solved by outputting a minimal response that satisfies the system constraint and ignores the user's added conflicting request; no reward term detects whether the model recognized or rejected the conflict. This makes 'reasoning over instruction hierarchies' an interpretation rather than a trained behavior. The IHEval conflict metric uses the same kind of constraint-over-user-request judgment, so the headline 20% gain does not discriminate between the proposed meta-reasoning mechanism and a simpler 'follow system prompt, ignore user' policy. The CoT classifier evidence in Appendix B is suggestive but not causal, since SysHint explicitly instructs the model to think about the relationship. The proposed test—adding a conflict-recognition/rejection reward component—would settle whether the original reward is sufficient. If the richer reward changes outcomes, the paper's central claim needs to be reworded; if it does not, the current conclusion survives. This is exactly the conditional risk the reader identified; I do not see a reason to change the verdict.","tokens_in":19122,"tokens_out":4684,"duration_ms":47725,"concrete_test":"Retrain Qwen3-8B on VerIH with a modified reward for conflicting samples: in addition to the original verifier, require (1) an explicit refusal/acknowledgment of the conflicting user sub-request and (2) a helpful response to any non-conflicting portion, with both checked by deterministic string/manual rules. Keep all hyperparameters and the aligned half identical. If IHEval-conflict accuracy and WildJailbreak:harmful ASR remain within a few points of the original +VerIH run, the original reward already taught conflict resolution and the concern is moot; if performance drops substantially, the reported gains were driven by blind system-prompt following rather than hierarchy reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 states that after Claude-4-Sonnet rewrites user prompts to create conflicts, 'the original verification functions from RLVR-IFEval remain applicable after rewriting.' The reward F_reward therefore evaluates only whether the final Answer satisfies the system prompt's constraints (e.g., exact yes/no/maybe, lowercase, markdown). In the conflicting half, a response that simply ignores the user's conflicting sub-request and outputs a system-satisfying token earns full reward; the model is never required to detect the conflict, reject the conflicting part, or help with non-conflicting parts. Thus RLVR can select for 'follow the system prompt blindly' rather than 'reason about hierarchy and reject conflicting user instructions.' The IHEval conflict metric rewards the same behavior—following the system constraint over the user's request—so the ~20% gain is compatible with this shortcut. The Section 6 argument that VerIH increases explicit IH reasoning relies on a Claude classifier over CoT traces, which can reflect the SysHint phrasing rather than a causally necessary reasoning skill. The central claim that instruction prioritization is learned as meta-reasoning is therefore not established by the current reward design; the safety transfer could simply be generalized obedience to system prompts.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to reframe instruction-hierarchy (IH) resolution as a meta-reasoning task and introduces VerIH, a synthetic dataset of aligned and conflicting system–user instruction pairs derived from RLVR-IFEval. Reasoning-enabled LLMs (Qwen3-4B/8B/14B, Phi-4-mini-reasoning) are trained with GRPO using the original RLVR-IFEval verifiers as the reward. The authors report consistent gains on IFEval, IFBench, and especially IHEval-conflict (~20% absolute), plus out-of-distribution safety improvements under a GuardRules system prompt, with up to 20% reductions in attack success rate on WildJailbreak, Harmbench, and TensorTrust. Ablations indicate that both chain-of-thought training and the conflicting-prompt half of VerIH contribute to the gains, and a CoT analysis with a Claude-4-Sonnet classifier reports increased explicit IH reasoning after training.","tokens_in":19408,"tokens_out":4530,"duration_ms":44342,"significance":"If established, the paper would provide a low-cost, data-efficient route to instruction-hierarchy compliance and safety transfer, with practical value for controllable LLMs. The strengths are real: the dataset and code are released; results span four models in two families; ablations isolate components; and the safety generalization is a valuable empirical finding. However, the central mechanistic claim—that the model learns to reason about instruction hierarchies—is not yet supported by the training objective, which only rewards system-prompt constraint satisfaction. The empirical gains are compatible with a simpler 'blind system-prompt obedience' account. The paper is a promising contribution but needs either a revised reward design that explicitly rewards conflict detection/rejection and helpfulness, or a substantially weakened claim and additional controls, before it can be accepted.","major_comments":[{"comment":"The reward function does not train the claimed reasoning behavior. As stated in §3, 'the original verification functions from RLVR-IFEval remain applicable after rewriting.' These functions check only whether the final answer satisfies the system prompt’s formatting/content constraints. In a conflicting example, a response that simply outputs the system-satisfying token (e.g., 'Yes') earns full reward; nothing rewards detecting the conflict, rejecting the conflicting user sub-request, or helping with non-conflicting parts. Thus RLVR can select for blind system-prompt following rather than meta-reasoning about hierarchy. The IHEval conflict metric rewards the same behavior, so the ~20% gain does not distinguish the two mechanisms. The Section 6 classifier over CoT traces is also not decisive because SysHint explicitly instructs the model to 'think step by step about the relationship betwe","section":"§2–§3"},{"comment":"All reported numbers are single point estimates with no variance, no seeds, and no significance tests. Several claimed 'consistent improvements' are small relative to plausible run-to-run noise (e.g., Table 1: Qwen3-8B IFEval-instruct 88.25% w/o CoT vs 87.41% +VerIH; Qwen3-14B MMLU 84.12% w/ CoT vs 83.87% +VerIH). The headline ~20% IHEval-conflict and ASR gains are large, but the paper should at least report multiple seeds or statistical significance for the key comparisons, especially since the ablations in Table 3 involve smaller differences.","section":"§5, Tables 1–2"},{"comment":"There is no comparison to prior instruction-hierarchy training methods, such as Wallace et al. (2024) or Wu et al. (2024b), even though the paper positions itself against them. The claim that 'reasoning'-based IH training is better than implicit input–response mapping remains unsupported. Adding such baselines (or at least a reimplementation of one prior method on the same models) is necessary to establish the relative advantage of the proposed approach.","section":"§1, §7, §4"},{"comment":"Training/evaluation overlap is a concern. VerIH is sampled from RLVR-IFEval, which belongs to the IFEval family, and the paper evaluates on IFEval and IFBench. Gains on those benchmarks may reflect training-distribution overlap rather than generalization. The IHEval and safety results are less affected, but the paper should quantify overlap (e.g., exact/near-duplicate detection) or use a held-out IFEval split to make the generalization claim cleaner.","section":"§3, Table 1"}],"minor_comments":[{"comment":"Text says 'for Phi-4-mini-reasoning, the w/o CoT train variant improves on IFBench, reaching 45.37%', but Table 3 reports 44.48%. The discrepancy should be reconciled.","section":"§6, Table 3"},{"comment":"The sentence 'Half of the samples were written by Claude-4-Sonnet. Claude-4-Sonnet was prompted to rewrite the user prompt such that it does not conflict with the system prompt' appears to have the opposite meaning of the intended one; it should be 'does conflict' (or 'such that it conflicts'). Also, 'written' should be 'rewritten'.","section":"Appendix A"},{"comment":"Minor typo: 'ref to Appendix C' should be 'refer to Appendix C'.","section":"§2"},{"comment":"The y-axis ranges are extremely compressed (e.g., 89.85–90.03 and 61.40–62.20), which visually exaggerates small differences. Consider showing full-scale axes or adding error bars.","section":"Figure 3"},{"comment":"Formatting: 'w/o CoTtrain' should be 'w/o CoT train' for readability.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper has promising empirical results, but the central 'reasoning' mechanism is not enforced by the reward. I would urge the editor to send it back for a major revision rather than reject, because the concern is addressable: adding a conflict-rejection/helpfulness reward component, or providing a control that isolates blind obedience, would materially strengthen the paper. The lack of comparisons to prior IH training methods and the absence of variance reporting are also important but secondary."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this paper, but with the reward-design question in mind. The empirical result is solid: RLVR on roughly 7K verifiable instruction-following examples, half of them rewritten to create system–user conflicts, gives consistent gains on IHEval conflict (about 20 points) and transfers out-of-distribution to safety benchmarks, cutting attack success rates by up to 27 points. The ablations are honest and useful—both chain-of-thought training and conflicting prompts are needed. The test-time compute analysis is a nice negative result. This is a genuinely lightweight way to improve instruction hierarchy compliance across four models.\n\nThe main soft spot is the one the stress-test hits. The reward function only checks whether the response satisfies the system prompt's constraints; it never requires the model to detect the conflict, reject the conflicting part, or help with the non-conflicting part. In a conflicting pair, the model can earn full reward by producing a system-satisfying token and ignoring the user's request entirely. IHEval conflict rewards the same behavior. So the title's claim—that the model learns to reason about instruction hierarchies—is not established by the training signal. The Section 6 evidence that VerIH increases explicit IH reasoning in CoT traces is suggestive, but it uses a Claude classifier and could reflect SysHint phrasing rather than a causally necessary skill. The safety transfer might simply be generalized obedience to system prompts, which is practically useful but not the same as meta-reasoning.\n\nThe other weaknesses are more minor. All tables report single point estimates with no variance or significance tests, so the headline gains are less secure than they look. There is no comparison to prior IH training methods like Wallace et al. (2024) or Wu et al. (2024b), which would put the contribution in perspective. And the existing verifier from RLVR-IFEval confounds training and evaluation when IFEval/IFBench are used as test sets, though the IHEval and safety results are separate and do support the main effect.\n\nMy take: the paper deserves a serious referee. The method works, the data construction is clever, and the ablation structure is a model of clarity. But the mechanism claim needs to be reined in and tested. I'd ask the authors to add variance estimates, compare against Wallace-style training, and—most importantly—analyze whether the trained model actually rejects conflicting user requests in cases where reward can be earned by ignoring them. If the answer is no, the paper is still a solid contribution about prompt-conditioned obedience, just not about reasoning.","headline":"A useful RLVR recipe for instruction hierarchy with a real mechanism gap: the reward may be teaching system-prompt obedience rather than conflict reasoning.","tokens_in":19920,"tokens_out":1669,"would_cite":true,"duration_ms":20987,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Instruction hierarchy resolution can be productively reframed as an explicit reasoning task, and lightweight reinforcement learning on a verifiable dataset of aligned and conflicting system–user prompts produces roughly 20-point gains in co","keywords":["instruction hierarchy","reasoning","reinforcement learning with verifiable rewards","controllability","safety","jailbreak","prompt injection","instruction following"],"falsifier":"A concrete test: take the conflicting samples from VerIH, remove the conflicting instruction from the user prompt (so the remaining text is aligned with the system prompt), and compare the model's output to its output on the original conflicting prompt. If the outputs are essentially identical—same constraints satisfied, same phrasing—the model has learned to follow the system prompt mechanically rather than reason about the conflict, which would falsify the paper's central mechanism.","tokens_in":19024,"feed_emoji":"🧠","tokens_out":7803,"duration_ms":69872,"temperature":0.7,"pith_summary":"The paper attempts to establish that instruction hierarchy resolution—deciding which of competing system and user directives to follow—can be productively treated as a reasoning task rather than a behavior learned by input-output mapping. To that end, the authors construct VerIH, a training set of aligned and conflicting system–user prompt pairs with automatically verifiable answers, and fine-tune reasoning-capable models with reinforcement learning on verifiable rewards (RLVR). They report consistent, roughly 20 absolute percentage point improvements on instruction-hierarchy benchmarks under conflict, across several model sizes and families, and show that the learned prioritization transfers out-of-distribution to safety benchmarks, reducing jailbreak and prompt-injection attack success rates by up to 20 points. If correct, this provides a practical recipe for making language models more controllable: updating higher-priority system prompts should predictably steer behavior, with safety emerging as a special case of conflict resolution.","feed_headline":"Reasoning over instruction conflicts lifts compliance ~20 points","feed_subtitle":"Light verifiable-reward training also trims jailbreak and injection attacks by up to 20%.","key_machinery":"The load-bearing mechanism is VerIH, a training dataset of constraint-following tasks with verifiable answers, built by taking an existing instruction-following dataset and rewriting half of the user prompts to conflict with their system prompts while preserving the original verifiable constraints. This allows deterministic reward evaluation during reinforcement learning. Combined with a system-level hint ('SysHint') that instructs the model to reason about the relationship between system and user prompts before answering, and a reinforcement learning algorithm on the verifiable reward, the mechanism transfers general reasoning ability into instruction prioritization. The design's power lies","core_discovery":"The central claim is that instruction hierarchy resolution is a meta-reasoning task: before generating a response, a model should explicitly reason about the relationship between the user prompt and higher-priority (system) instructions, and override or reject lower-priority content when they conflict. The authors operationalize this through VerIH, a dataset of about 7,000 constraint-following examples, half of which are rewritten so the user request conflicts with the system prompt while preserving verifiable response constraints. Lightweight reinforcement learning on this dataset, guided by a system-prompt hint to reason about the instruction relationship, transfers the model's existing ge","pith_inferences":["Because the reward only scores constraint satisfaction, a natural extension is to add a reward signal for explicitly acknowledging (and rejecting) the conflicting part of a user request while remaining helpful on the rest; this would make the learned behavior more directly tied to conflict reasoning.","The same recipe could be applied to multi-level hierarchies (system > user > tool outputs); the paper's small gain on a three-level tool-use subset suggests the mechanism may extend beyond two levels.","Treating safety as a special case of conflict resolution opens the possibility of deploying this training as a general-purpose 'policy adherence' layer, where new organisational or regulatory rules can be introduced purely as system-prompt edits without retraining."],"forward_implications":["Instruction hierarchy compliance on conflicting system–user prompts improves by roughly 20 absolute percentage points compared to strong prompting baselines, across multiple model families and sizes.","The learned prioritization transfers to safety tasks never seen in training: attack success rates on jailbreak and prompt-injection benchmarks drop by up to 20 percentage points when a safety-focused system prompt is added at inference.","Models trained this way maintain or slightly improve general reasoning (e.g., MMLU, MATH-500) and instruction-following accuracy, so the gain does not come at the cost of broader capability.","Updating the system prompt—rather than retraining—becomes a reliable way to steer model behavior in novel or changing policy scenarios.","The effect depends on explicit reasoning during training: ablations that disable chain-of-thought training or remove conflicting examples substantially degrade conflict resolution, indicating both components are necessary."],"fun_headline_variants":["Think first, then answer: training LLMs to weigh system rules","Reasoning about rule conflicts boosts LLM compliance by 20%","Verifiable-reward RL teaches models to prioritize system prompts","Instruction hierarchy as reasoning: 20% better conflict handling","Training models to reason about instructions cuts attacks 20%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method rests on the assumption that the verifiable reward, which only checks whether the answer satisfies the system-prompt constraints, teaches legitimate conflict-resolution reasoning rather than blind adherence to the system prompt—there is no reward component for detecting the conflict or helping with non-conflicting parts of the user request.","fun_headline_variants_meta":{"raw":{"variants":["Think first, then answer: training LLMs to weigh system rules","Reasoning about rule conflicts boosts LLM compliance by 20%","Verifiable-reward RL teaches models to prioritize system prompts","Instruction hierarchy as reasoning: 20% better conflict handling","Training models to reason about instructions cuts attacks 20%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1197,"prompt_tokens":736,"completion_tokens":461,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":376}},"tokens_in":480,"tokens_out":461,"duration_ms":17109,"temperature":1.0,"reasoning_tokens":376,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T07:02:26.536427+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: take the conflicting samples from VerIH, remove the conflicting instruction from the user prompt (so the remaining text is aligned with the system prompt), and compare the model's output to its output on the original conflicting prompt. If the outputs are essentially identical—same constraints satisfied, same phrasing—the model has learned to follow the system prompt mechanically rather than reason about the conflict, which would falsify the paper's central mechanism.","supporting_citations":[],"review_version":1}