{"id":"e4e05efc-862f-4613-8cd7-e53b4055deb5","arxiv_id":"2602.13562","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ASCL lets Qwen3 models retrieve safety rules on demand through a tool, and IFPO rebalances RL advantages so the model doesn't over-consult the tool; together they improve the safety/over-refusal trade-off.","lead":"This paper lets LLMs retrieve safety rules from an external policy tool only when needed, instead of embedding rules into every reasoning trace, and adds a reinforcement-learning reweighting step (IFPO) to keep models from over-consulting the tool. On Qwen3-4B/8B/14B it reports improved safety with lower over-refusal, but the evaluation judge is the same model used as the training reward.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim rests on a reward/judge feedback loop; independent judge and held-out over-refusal data are needed.","rationale":"The reader's weakest assumption—that Qwen3-Guard-8B being both reward model and evaluator undermines the safety/over-refusal measurements—is exactly the load-bearing concern I identify. The same model supplies the training signal and the outcome metric, so the reported Pareto improvement may reflect alignment to the judge rather than robust safety behavior. The OR-Bench train/eval family overlap compounds this for the over-refusal axis, and the single-run evaluations provide no variance estimate. The surprising identical IFPO over-refusal numbers across 4B and 8B add a concrete reason to distrust the table as reported. The paper has genuine strengths: the ASCL mechanism is well-motivated, the ablation against surrogate CoT data is informative, the authors are transparent about the lack of generalization to general reasoning, and code is public. But the central empirical claim cannot be accepted until an independent judge and held-out over-refusal data confirm it. The reader's CONDITIONAL verdict is appropriate; no verdict change is needed, but the conditions should specifically require the independent-judge re-evaluation and overlap-filtered OR-Bench-Hard evaluation.","tokens_in":20562,"tokens_out":4575,"duration_ms":42955,"concrete_test":"Re-run the Table 1 evaluation on Qwen3-4B and Qwen3-8B using an independent judge (e.g., WildGuard or Llama-Guard-2) instead of Qwen3-Guard-8B, and on an over-refusal set where OR-Bench-Hard has been filtered to remove exact/near-duplicate prompts present in OR-Bench-80k training data. Recompute the IFPO-vs-BC and IFPO-vs-STAR-1-mix differences in safety and over-refusal. If either claimed Pareto gain shrinks by more than 10 percentage points or reverses, the central claim is not independently supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim—ASCL+IFPO performs best on both safety and over-refusal—depends entirely on Qwen3-Guard-8B as the measurement instrument, but that same model is also the reward signal optimized during RL (Section 4.2: safety reward Rs and compliance reward Rc). Sections 3.2, 5.1, and Appendix D confirm Qwen3-Guard is the judge for every safety and over-refusal result. This creates a circular evaluation loop: the policy is trained to satisfy this judge, and then judged by it. The reported Pareto improvement may therefore reflect reward hacking rather than a genuine safety–utility trade-off. The over-refusal result is additionally vulnerable because training data for sensitive queries are drawn from OR-Bench-80k (Section 4.1) while a major evaluation component is OR-Bench-Hard (Appendix D); the large OR-Bench-Hard drop (e.g., 47.23 to 5.99 for Qwen3-4B in Table 1) could be partly in-distribution gain. The paper also reports single-run evaluations with no error bars, so the significance of any margin is unknown. A further concrete red flag: in Table 1, the IFPO over-refusal rows for Qwen3-4B and Qwen3-8B are numerically identical across all five sub-benchmarks (7.60, 2.38, 4.34, 9.67, 5.99), which is implausible across model scales and suggests an evaluation or reporting issue that must be checked. Because the central claim is empirical and the empirical backbone is this judge, this is the load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ASCL (Adaptive Safe Context Learning), a framework that recasts safety alignment as a multi-turn tool-use problem: the model can decide whether to consult an external safety-policy document through retrieval, and then reason on the retrieved context. It combines behavior cloning (BC) with reinforcement learning (RL), and introduces Inverse Frequency Policy Optimization (IFPO) to reweight advantages in order to prevent the policy from over-consulting the retrieval tool. Experiments on Qwen3-4B/8B/14B claim that ASCL+IFPO achieves the best safety and over-refusal performance among a set of baselines, while general reasoning benchmarks show no consistent gains—a limitation the authors acknowledge. Ablations compare ASCL against surrogate CoT data and against a simpler rule-penalty reward.","tokens_in":20954,"tokens_out":4049,"duration_ms":39017,"significance":"The core idea—decoupling rule retrieval from reasoning so that safety rules are consulted adaptively rather than memorized—is timely and potentially valuable for mitigating the safety–utility trade-off in reasoning models. The paper offers a detailed pipeline, is transparent about its training setup, releases code, and includes experiments across three model sizes and multiple benchmarks. The self-reported limitation that general reasoning gains are not demonstrated is a useful honest check. However, the empirical validation has serious weaknesses: the evaluation judge is identical to the reward model used in RL, results come from a single run with no variance, and over-refusal training data and one of the key evaluation benchmarks come from the same OR-Bench family. These issues directly affect the strength of the central claim that ASCL+IFPO 'performs the best on both safety and over-refusal benchmarks.' The paper is therefore not yet ready for publication as is, but the framework and method are sufficiently interesting that the concerns could be addressed with additional experiments and independent evaluation.","major_comments":[{"comment":"The reward model used during RL (Qwen3-Guard-8B for safety reward Rs and compliance reward Rc, Eq. (1)) is the same model used as the judge for all safety and over-refusal evaluations (Section 3.2 and Section 5.1). This creates a feedback loop: the policy is optimized to satisfy this judge and then evaluated by it. The reported improvements may substantially reflect the policy conforming to the judge’s scoring criteria rather than genuine progress on safety/over-refusal. The authors should evaluate with an independent judge (e.g., WildGuard, Llama Guard, or human annotation) and, ideally, measure on data not used in training. Without this, the central empirical claim is not independently verified.","section":"§4.2 vs §5.1"},{"comment":"All evaluations are executed exactly once and averaged at the dataset level; no error bars or confidence intervals are provided. Since the headline claim rests on comparative percentages (e.g., IFPO vs BC vs baselines), it is unclear whether the reported margins are meaningful or within noise. The authors should provide multiple runs, bootstrap intervals, or at minimum a variance estimate for the key safety and over-refusal numbers. This is especially important because the differences across methods are sometimes small (e.g., safety averages in Table 1, Qwen3-14B: BC 97.62 vs IFPO 98.64).","section":"§3.2, Table 1"},{"comment":"Training data for sensitive/benign queries are sampled exclusively from OR-Bench-80k (Section 4.1), while a major component of the over-refusal evaluation is OR-Bench-Hard (Appendix D), which comes from the same OR-Bench family. The large improvement on OR-Bench-Hard (e.g., Qwen3-4B: BC 47.23 vs IFPO 5.99 in Table 1) may therefore be partly an in-distribution gain. To support the claim of generalizable over-refusal mitigation, the authors should either exclude OR-Bench-Hard from the main average or supplement with an independent over-refusal benchmark that is clearly disjoint from the training distribution.","section":"§4.1, Appendix D"},{"comment":"The IFPO over-refusal rows for Qwen3-4B and Qwen3-8B are numerically identical across all five sub-benchmarks (7.60, 2.38, 4.34, 9.67, 5.99). It is implausible that two different model sizes produce exactly the same refusal rates on all benchmarks. This strongly suggests a copy/paste error or an evaluation/reporting bug. The authors must check and correct this, and rerun if necessary. This issue undermines confidence in the accuracy of the reported numbers.","section":"Table 1"}],"minor_comments":[{"comment":"The wording 'novelly decouples' and 'inutility of incorporating safety rules' is awkward; consider 'novel decoupling' and 'the limited benefit of incorporating'.","section":"§1, §2.1"},{"comment":"Inconsistent capitalization: 'Safechain' appears in the 8B and 14B sections while the baseline is referred to as 'SafeChain' in the 4B section and in the text.","section":"§5.2, Table 1"},{"comment":"The hallucination penalty Ph is described as binary but its role in Eq. (1) as a multiplier is not explicitly defined. Please clarify that Ph=0 for hallucinated tool calls and Ph=1 otherwise, and similarly for Pf.","section":"§4.2, Eq. (1)"},{"comment":"The hyperparameters τ, wmin, wmax are introduced without a sensitivity analysis. Given that IFPO’s benefits are attributed to this weighting, the authors should at least report how the results vary with τ or provide a brief justification for the chosen values.","section":"§4.3, Eq. (3)"},{"comment":"Typo: 'deleberately' should be 'deliberately'.","section":"Appendix C"},{"comment":"The OR-Bench-Hard dataset size is listed as 1319, which is the same as the GSM8K evaluation size in the next column; please verify this is not a copy-paste error.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting, but the empirical evaluation is currently not convincing due to the judge/reward overlap, single-run results, and the OR-Bench training/evaluation overlap. The identical IFPO rows in Table 1 are a clear red flag. I believe the paper can be made publishable if the authors add an independent judge, report error bars, and address the OR-Bench issue. The framing also leans heavily on 'safety-utility trade-off' without a formal notion of Pareto optimality; a more rigorous empirical protocol would strengthen the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core mechanism here is genuinely new: ASCL turns safety-rule consultation into an optional tool-use action inside a ReAct loop, and IFPO reweights policy-gradient advantages by action frequency to stop the model from over-retrieving during RL. That is a real idea, distinct from the self-reminder / in-context safety prompting line and from Deliberate Alignment / SafeChain. The zero-shot comparison, the ablation that swaps in a surrogate CoT dataset, and the penalty-vs-IFPO comparison are all useful pieces of evidence. The paper is also honest that it shows no generalization to general reasoning—that limitation is stated plainly in Section 5.3 rather than buried.\n\nWhere the paper is soft is exactly where the reader's stress-test lands. Qwen3-Guard-8B is used as both the reward model in RL (Section 4.2) and the judge for every safety and over-refusal number in Table 1 (Sections 3.2 and 5.1). So a good chunk of the reported Pareto improvement could be the policy conforming to the judge's scoring function rather than an independently measured trade-off. The over-refusal training set (OR-Bench-80k) and a major evaluation set (OR-Bench-Hard) come from the same benchmark family, which makes the large OR-Bench-Hard drops partly in-distribution gains. All evaluations are single-run with no error bars, so we have no idea whether the margins are meaningful. These are real limitations, not manufactured ones.\n\nOne concrete red flag: in Table 1, the IFPO over-refusal rows for Qwen3-4B and Qwen3-8B are numerically identical across all five sub-benchmarks (7.60, 2.38, 4.34, 9.67, 5.99). That is implausible across model scales and most likely a reporting error, but it needs to be checked and corrected. This kind of slip matters because it chips away at trust in the tables.\n\nAll that said, the paper is not a sham. The framework is clearly described, the code is public, the ablations are thoughtful, and the authors flag their own limitations. The central claim may survive if the evaluation is redone with an independent judge, held-out over-refusal data, and multiple seeds. This deserves a serious peer review rather than a desk reject—the method-level contribution is strong enough that referees should push for the additional experiments rather than dismissing the work.\n\nI'd bring it to a reading group interested in LLM safety alignment, and I'd cite the IFPO reweighting idea in my own work. The paper needs revision, but the core is solid.","headline":"Novel and well-motivated alignment method, but the headline empirical claim is weakened by a reward-judge loop, single-run evals, and a suspicious Table 1 duplicate; still worth refereeing.","tokens_in":21478,"tokens_out":1579,"would_cite":true,"duration_ms":16822,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that moving safety rules to an on-demand retrieval tool yields a better safety-utility trade-off than embedding them in chain-of-thought training data.","keywords":["safety alignment","over-refusal","reasoning models","chain-of-thought","tool-use","reinforcement learning","inverse frequency policy optimization","context learning"],"falsifier":"Run the ASCL-trained and baseline models on safety and over-refusal sets scored by human annotators or a different judge; if ASCL no longer has the best combined scores, the trade-off win is judge-specific. A second concrete check: evaluate on a newly collected set of benign but superficially sensitive prompts that were not used in any training or benchmark family; if the large over-refusal drop does not transfer, part of the gain is in-distribution.","tokens_in":20404,"feed_emoji":"🛡️","tokens_out":5129,"duration_ms":49528,"temperature":0.7,"pith_summary":"This paper is trying to show that a reasoning model can be made safer without becoming over-cautious if safety rules are moved out of its memorized chain-of-thought and into an external policy tool it consults only when needed. The training recipe has two parts: behavior cloning that teaches when to call the safety-retrieval tool, and an inverse-frequency reinforcement-learning update that stops the model from over-calling it. Across three model scales, the authors report that this combination beats existing safety-training methods on both safety benchmarks and over-refusal benchmarks, while leaving general reasoning accuracy about unchanged. If true, this means the cost of safety alignment can be lowered by treating rule consultation as a decision, not a reflex.","feed_headline":"Safety alignment works best when rules are a tool, not a memory","feed_subtitle":"Letting models fetch safety policies on demand cuts over-refusal while keeping safety high.","key_machinery":"The ASCL loop: the model operates in a multi-turn tool-use setting with a retrieval tool, safety-policy-retrieve_policy, backed by a document of 107 safety terms across 21 categories; the model decides whether and when to call it, then reasons over the returned snippets. IFPO: after a rollout group, each response's advantage is multiplied by an inverse-frequency weight, w_i = clip((|G|/N_{m_i})^τ / average, w_min, w_max), where m_i marks whether the response used the tool and N_{m_i} is how many group members share that choice; this makes rare choices get bigger updates, decoupling update size from sampling frequency. The RL reward combines safety and compliance scores from a guard model wit","core_discovery":"The paper's central claim is that the safety-utility trade-off in reasoning LLMs is not intrinsic; it is created by training that welds safety rules into the chain of thought. The proposed ASCL separates the two: safety rules live in a policy document that the model can query through a retrieval tool during generation, and a post-training phase (behavior cloning followed by reinforcement learning with Inverse Frequency Policy Optimization) teaches the model when to query and when to reason without it. The result, on three model sizes, is the best combined safety and over-refusal performance among the baselines, with the trade-off curve shifted outward. A qualitative example shows the mechani","pith_inferences":["Editorial: Because one guard model supplies both the training reward and the evaluation score, the reported safety gains may be partially an optimization toward that judge; independent judges or human annotators would make the Pareto claim stronger.","Editorial: The largest over-refusal drop is reported on the same benchmark family used to build the over-refusal training data, so some of the gain may be in-distribution; fresh benign prompts would test robustness.","Editorial: The same decoupling could be transferred to other cost-sensitive guardrails, such as privacy, copyright, or illegal-content moderation, where indiscriminate refusal is expensive, though the paper only studies safety categories.","Editorial: The paper reports no improvement on general reasoning, so the mechanism is not a general reasoning enhancer; its value is specific to refusal decisions."],"forward_implications":["Models can be aligned without embedding safety rules into every chain-of-thought; a tool call that returns policy text is enough to keep safety high.","Over-refusal falls because the model can consult rules and then decide they do not apply, as in the benign-but-sensitive prompt example.","IFPO corrects reinforcement learning's tendency to over-use the retrieval tool, keeping general reasoning accuracy from dropping.","The safety rules live in a retrievable document detached from the model's reasoning, so the model's behavior shifts from memorization to selective consultation."],"fun_headline_variants":["Safety rules as tools, not memory, improves LLM alignment","Adaptive rule retrieval cuts LLM over-refusal while keeping safety","LLM safety-utility trade-off eased by on-demand safety policy queries","Models that fetch safety rules on demand beat welded CoT training"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central empirical result depends on the judgments of a single guard model being a correct measure of both safety and over-refusal; if that judge is biased, the claimed Pareto improvement may reflect fitting the judge rather than genuinely balancing safety and utility.","fun_headline_variants_meta":{"raw":{"variants":["Safety rules as tools, not memory, improves LLM alignment","Adaptive rule retrieval cuts LLM over-refusal while keeping safety","LLM safety-utility trade-off eased by on-demand safety policy queries","Models that fetch safety rules on demand beat welded CoT training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1040,"prompt_tokens":696,"completion_tokens":344,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":270}},"tokens_in":440,"tokens_out":344,"duration_ms":3416,"temperature":1.0,"reasoning_tokens":270,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T23:29:07.033608+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the ASCL-trained and baseline models on safety and over-refusal sets scored by human annotators or a different judge; if ASCL no longer has the best combined scores, the trade-off win is judge-specific. A second concrete check: evaluate on a newly collected set of benign but superficially sensitive prompts that were not used in any training or benchmark family; if the large over-refusal drop does not transfer, part of the gain is in-distribution.","supporting_citations":[],"review_version":1}