{"id":"0a90fe47-cf65-4a8a-acb6-2d46432ac5c7","arxiv_id":"2608.13304","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Matched harmful and benign wrapper groups plus a refusal-score consistency regularizer allow fine-tuned LLMs to refuse harmful intent under surface-form variation while reducing benign over-refusal.","lead":"This paper introduces training methods that teach LLMs to refuse harmful requests by recognizing underlying intent rather than surface wrappers. The best configuration cuts benign over-refusal on OR-Bench from 25.7% to 17.4% while preserving strong harmful-request refusal.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The below-base over-refusal claim rests on a margin configuration selected on the test benchmark, with only a single seed reported and no validation protocol or error bars. This makes the 17.4% OR figure unreliable as a headline estimate.","rationale":"The reader's weakest assumption concerned the A-GCRT decision score (Eq. 3). That is a real concern, and the paper's own Table 22 shows the score is a weak label-side separator (AUROC 0.453) with frequent missing markers. However, the score is explicitly used only as a training-time regularizer, and the central empirical claim could in principle survive a noisy proxy. By contrast, the margin-selection protocol determines whether the paper's headline number (17.4 OR) is an honest estimate at all. A-GCRT-M5 is one point from a scan over five margin/anchor configurations, and it is the only configuration below the base model's OR. Since all runs use seed 42 and no validation split is reported, the 17.4 could be the result of choosing the configuration and the seed that best fit the test benchmark. This is precisely the kind of test-set selection that inflates results in small hyperparameter scans. The reader's conditional verdict already asks for error bars and a validation protocol; my concern reinforces that condition rather than changing the verdict. I therefore recommend UNCHANGED, with the condition that the authors provide a validation-based margin-selection protocol and multi-seed error bars for the headline operating points.","tokens_in":25311,"tokens_out":8136,"duration_ms":86194,"concrete_test":"Construct validation and test splits from the 1319 OR-Bench-Hard prompts (or use XSTest safe-prompt refusal as a validation criterion). For each margin setting in Table 21 (m = 2.5, 5, 7.5, 10 with γ = 1 and γ = 2), train A-GCRT with 5 seeds and select the configuration with the lowest mean validation OR. Evaluate the selected configuration on the held-out test split and report mean ± std for OR and SB-avg5, together with the same statistics for the base model. If the selected configuration's mean test OR is not below the base model's mean test OR (roughly 25.7), the headline over-refusal reduction is a test-selection or seed artifact. As a secondary check, report XSTest unsafe refusal for the selected configuration to confirm the margin selection is not simply weakening refusal overall.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The low-over-refusal result that anchors the benign-compliance half of the central claim is selected on the test benchmark, and only a single seed is reported. Section 6.5 (Table 21) scans A-GCRT margins m = 2.5, 5, 7.5, 10 with anchor weights γ = 1 and γ = 2, and Table 1 reports A-GCRT-M5 as the headline point because it has the lowest OR-Bench-Hard value (17.4) on the same OR-Bench-Hard test set used to state the claim. The m = 2.5 configuration already gives OR 29.3, above the base model's 25.7, and larger margins give 49.4 or 61.3. Appendix B.2 (Table 9) lists seed 42 for all Qwen runs, and no held-out validation split for margin selection is described anywhere in the paper, even though Section 6.5 calls the margins 'validation-selected operating controls.' Without a pre-registered validation protocol or seed averaging, the 17.4 figure is the minimum of a small scan and is not yet a reliable estimate of the method's expected over-refusal. The Llama setting does not reproduce below-base OR, which further limits the generality of this specific operating point.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes that LLM safety tuning should supervise refusal at the level of intent groups rather than individual prompt forms. It introduces WIFA, a data-augmentation method that pairs wrapped harmful prompts with structurally matched wrapped benign prompts under shared wrapper families, and two training routes built on this data layer: WIFA-Boost, a two-stage high-safety recipe, and A-GCRT, which regularizes refusal/compliance decision scores within intent groups and anchors harmful and benign groups on opposite sides of a margin. The experiments cover Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct across seven benchmarks and a 15-family unseen-attack suite. The headline claims are that WIFA-Boost reaches the strongest transformed-harmful refusal (63.7 SORRY-Bench mutation-average in Qwen) and that A-GCRT-M5 reduces OR-Bench over-refusal from 25.7% to 17.4%, with reproduced baselines failing to match these operating points. The paper also reports extensive ablations on data structure, benign/harmful ratio, stage order, A-GCRT components, margins, and decision-score diagnostics, and it explicitly notes that Llama does not reproduce below-base over-refusal.","tokens_in":25640,"tokens_out":5509,"duration_ms":64364,"significance":"If verified, the paper would offer a concrete, teacher-free way to reduce surface-form shortcuts in safety tuning and to separate the high-safety and low-over-refusal operating points. The WIFA construction is a useful idea: pairing wrapped harmful and benign intents makes the wrapper non-diagnostic of the decision, and the two-stage and margin-based recipes provide a principled way to think about safety versus over-refusal trade-offs. Strengths include the release of code and sanitized artifacts, the breadth of ablations, the honest reporting of the Llama boundary, and the inclusion of decision-score diagnostics that are negative in part. The central limitation is verification quality: the headline below-base over-refusal figure comes from a test-set margin scan with a single seed, and the A-GCRT score that drives the objective is shown to be a weak label-side separator with frequent missing markers. These issues do not make the idea circular or obviously wrong, but they mean the quantitative claims as stated are not yet reliable.","major_comments":[{"comment":"The central over-refusal result is not supported as a general estimate because the 17.4% figure is the best point of a margin/anchor scan performed on the same OR-Bench-Hard test set used to state the claim. Table 21 scans m = 2.5, 5, 7.5, 10 with gamma = 1 and gamma = 2, and Table 1 reports A-GCRT-M5 as the headline point; all runs use seed 42 per Appendix B.2. No held-out validation split for margin selection is described anywhere in the paper, despite Section 6.5 calling these 'validation-selected operating controls.' The adjacent configuration m = 2.5 gives OR 29.3, above the base model's 25.7, so the conclusion reverses under a nearby hyperparameter choice. Please provide a pre-specified validation protocol, seed-averaged results with confidence intervals, or revise the abstract and Section 5.1 to state that 17.4 is the best-scanned point rather than a robust estimate.","section":"§6.5, Table 21, Table 1, Appendix B.2"},{"comment":"The mechanism underlying A-GCRT depends on the decision-position score s_theta(x) being a usable refusal/compliance signal, but the manuscript's own diagnostics show this score is a weak label-side separator: Table 22 reports label AUROC 0.453 for A-GCRT-M5, and missing-marker rates in free generation range from 14.3% to 33.7%. Table 23's target-forced diagnostic supplies the intent-analysis marker and uses train-sampled groups, so it matches the training-time objective but does not establish that optimizing Lvar and Lanchor transfers to actual generated refusal/compliance behavior. The paper properly cautions that the score is not an inference-time classifier, yet the objective's gradient signal is still this same score. Please provide a direct analysis of how changes in the regularization loss translate into changes in generated refusal/compliance behavior, or temper the claim that A-GCRT shapes the refusal boundary through group-consistent decision scores.","section":"§3.3, Eq. (3), Appendix C.6, Table 22, Table 23"},{"comment":"All headline numbers in Table 1 come from single training runs with no seed variation or uncertainty estimates, and several comparisons that support the paper's operating-point narrative are numerically small, such as WIFA-SFT versus WIFA-Boost on SB-avg5 (63.6 versus 63.7). Without multiple seeds, bootstrap confidence intervals, or a stated evaluation-variance protocol, the paper cannot support claims that one method 'reaches' a point or that reproduced baselines 'do not match' it in a statistically meaningful sense. Reporting uncertainty for at least the headline metrics and the margin-scan table would address this concern.","section":"§5.1, Table 1"}],"minor_comments":[{"comment":"The corrected capability protocol applies a fixed benign intent-analysis prefix to MMLU and GSM8K for all methods, including base models; please provide a short comparison with the standard protocol for at least the base model so readers can quantify how the prefix changes capability estimates.","section":"§4.3 and Figure 6"},{"comment":"Several tables, especially Table 1, do not explicitly mark the direction of every column in the main text; adding arrows such as 'SB-avg5 ↑' and 'OR ↓' consistently would reduce the risk of misreading.","section":"Table 1 and Appendix C tables"},{"comment":"The wording 'Behavior Agree' and 'Label Accuracy' in Table 22 is not defined in the main text; please define these metrics and explain why label-side accuracy is computed on a presumably balanced or imbalanced sample.","section":"Appendix C.6"},{"comment":"The sentence 'we use this score only as a training-time regularization proxy, not as a calibrated inference-time classifier' is helpful, but the same paragraph should note explicitly that the score is also not a clean training-time separator, as later shown in Table 22, so readers are not misled about its quality.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is interesting and the empirical scope is broad, but the headline below-base over-refusal claim is currently a test-set-selected single-seed minimum, and the A-GCRT score diagnostics raise an unresolved question about whether the objective's mechanism is doing what is claimed. These are fixable with additional experiments and a more careful claim structure. I would not recommend rejection, but I would require a validation protocol and seed-averaged results before accepting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a credible training-time method paper, and the central idea—supervising refusal at the level of matched harmful/benign wrapper groups rather than individual prompt forms—is new and worth building on. The WIFA data construction is elegant: no external teacher, wrapper form made non-diagnostic, and the two training routes (WIFA-Boost for high safety, A-GCRT for low over-refusal) cleanly separate two goals that are usually conflated in refusal tuning.\n\nWhat it does well: the ablation set is genuinely careful. The data-structure ablation rules out 'just more benign data,' the ordering ablation rules out a pure exposure effect, and the learning-rate and single-component ablations rule out 'stronger SFT' as an explanation for A-GCRT's behavior. There is also a target audit, a cross-judge check on the OR result, and a second model setting (Llama) that the authors use to delimit, not inflate, their claim. They explicitly say they are not claiming universal below-base over-refusal. That is honest empirics.\n\nThe main soft spot is the headline number itself. A-GCRT-M5's 17.4% OR-Bench is the minimum of a margin scan (m=2.5, 5, 7.5, 10; γ=1,2) run on the same OR-Bench-Hard test set used to state the claim. The m=2.5 point already sits above the base model's 25.7%, and larger margins give 49.4 and 61.3. Only a single seed is reported, and no held-out validation split for margin selection is described. So the 17.4 figure should be read as an in-sample operating point, not a reliable estimate of expected over-refusal. That does not kill the paper, but it moves the below-base claim from 'established' to 'plausible, needs more runs.'\n\nThe decision-score proxy is the second soft spot, but a smaller one. The label-side AUROC is poor (0.453 for M5) and free generation often drops the analysis marker (14–34%). The authors are upfront about this and explicitly scope the score as a training-time regularizer, not an inference classifier; that is fair. It does, however, mean the gradient signal is only loosely tied to actual refusal/compliance, so the mechanism is less clearly verified than the behavioral results suggest.\n\nOverall: the central qualitative claim—intent-group supervision improves wrapper robustness and changes the safety–over-refusal trade-off relative to reproduced baselines—survives my reading. The quantitative operating point needs validation before anyone should rely on it. This deserves peer review, with a request for multi-seed runs, error bars, and a pre-registered or at least explicitly held-out margin-selection protocol. I would cite the WIFA construction in my own work.","headline":"Solid, honestly-reported empirical safety paper; the matched-wrapper intent-group idea is genuinely useful, but the headline below-base over-refusal number is a single-seed minimum of a test-set margin scan and should be treated as a demonstration until multi-seed validation is shown.","tokens_in":26151,"tokens_out":2451,"would_cite":true,"duration_ms":24950,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Safety tuning that supervises intent groups, not prompt forms, lets LLMs refuse wrapped harmful requests without over-refusing benign ones.","keywords":["LLM safety alignment","intent-group supervision","wrapper robustness","over-refusal","refusal tuning","group-consistent regularization","surface-form shortcuts","jailbreak robustness"],"falsifier":"Retrain A-GCRT with the decision score computed from shuffled or randomly chosen prefix token sets and check whether OR-Bench over-refusal still drops from 25.7% to roughly 17.4%; if the drop survives, the score-based group regularization is not what carries the result. A complementary check: measure free-generation refusal rates across many wrappers of the same harmful intent group; if refusal still swings sharply across wrappers, decisions track form rather than intent, and the central claim fails.","tokens_in":25121,"feed_emoji":"🛡️","tokens_out":10280,"duration_ms":90691,"temperature":0.7,"pith_summary":"The paper takes on a specific failure of LLM safety tuning: models learn that the wrapper around a request, not the intent underneath it, determines whether to refuse, so wrapped harmful prompts slip through and similarly wrapped benign prompts get refused. It proposes supervising whole intent groups, the same underlying request written in several surface styles, paired with structurally matched benign requests, so wrapper form carries no information about the right decision. Two training routes built on that data layer make the point concrete: one maximizes refusal of wrapped harmful requests, raising SORRY-Bench mutation-average refusal from 22.1 to 63.7 in the Qwen setting, and the other cuts over-refusal on benign requests from 25.7% to 17.4% while still refusing harmful ones. A sympathetic reader would care because the methods separate two goals usually conflated in refusal tuning, harmful-request refusal and benign compliance, and because the reproduced baselines do not reach these operating points, suggesting the intent-group structure itself rather than extra data or stronger fine-tuning is doing the work.","feed_headline":"Intent-group training makes LLM refusal track intent, not form","feed_subtitle":"Matched harmful and benign wrapper groups cut OR-Bench over-refusal from 25.7% to 17.4%.","key_machinery":"The load-bearing objects are intent groups and a refusal/compliance decision score. An intent group $G(z)$ is the set of prompt forms expressing one underlying request; WIFA builds matched harmful and benign groups under shared wrapper families so that form carries no signal about the desired decision. A-GCRT then regularizes the first-token decision score $s_\\theta(x)$, defined as the maximum next-token logit over refusal prefixes minus the maximum over compliance prefixes just after the intent-analysis marker, using the loss $\\mathcal{L} = \\mathcal{L}_{\\mathrm{SFT}} + \\lambda_{\\mathrm{gcr}}(\\mathcal{L}_{\\mathrm{var}} + \\gamma \\mathcal{L}_{\\mathrm{anchor}})$, where the variance term penalizes score spread within an intent group and the anchor term pushes harmful group means above a margin $+m$ and benign group means below $-m$. This score, used only at training time, is what lets the loss reshape the refusal boundary without an auxiliary classifier or preference pairs.","core_discovery":"The paper's central claim is that safety supervision should be defined over intent groups rather than isolated prompt-response pairs. WIFA constructs, for each harmful intent, a group of wrapped variants plus direct refusal anchors, and for each benign intent a group under the same wrapper families, so the wrapper is a nuisance variable rather than a decision label. The intended consequence, demonstrated in the Qwen setting, is that decisions track intent: WIFA-Boost raises SORRY-Bench mutation-average refusal from 22.1 to 63.7 and the misrepresentation subset from 0.2 to 59.3, while A-GCRT-M5 lowers OR-Bench over-refusal from 25.7% to 17.4%, below both the base model and every reproduced defense, while still improving harmful-request refusal. The Llama results and the ablations over data structure, stage order, margins, and loss components support this intent-group interpretation without claiming universal below-base over-refusal.","pith_inferences":["Editorial inference: Because WIFA is a data layer independent of the optimizer, the same matched intent groups could feed preference-based objectives such as DPO-style or RLHF-style training; A-GCRT's variance-plus-anchor structure is orthogonal to the SFT loss it currently rides on, and the paper itself notes that ungrouped preference pairs do not supply the structure it needs.","Editorial inference: The decision score's known weakness (near-chance harmful/benign separation, with the intent-analysis marker missing in 14-34% of free generations) suggests A-GCRT's gains might be reproducible with a more reliable score, such as a generative refusal judge or a hidden-state probe, which could turn the training-time regularizer into an inference-time control.","Editorial inference: The matched-benign principle yields a practical diagnostic: measuring within-intent variance of refusal decisions across wrappers could tell a practitioner whether a safety-tuned model has learned a form shortcut, before deployment, using the paper's variance notion as a cheap inspection tool.","Editorial inference: The cross-setting gap in the paper's own numbers, Qwen below base on OR while Llama stays at 40.1% against a 28.4% base, raises the open question of whether the intent-group advantage is model- and data-source-dependent rather than a universal law."],"forward_implications":["Wrapper form becomes non-diagnostic: models trained on matched intent groups refuse wrapped harmful prompts at rates close to direct harmful prompts, and average attack success across 15 unseen attack families falls from 48.3% on the base model to 9.5% for WIFA-Boost, evidence against template memorization though not against adaptive attackers.","Over-refusal and harmful refusal can be selected as separate operating points: A-GCRT margin and anchor settings act as validation-selected controls (the M5 point gives 17.4% OR with 46.7 SB-avg5, the M10 point gives 40.7% OR with 52.0 SB-avg5), not a monotonic larger-is-better dial.","Curriculum order matters: learning the intent-form structure before plain-benign calibration is what preserves refusal on difficult mutated harmful prompts (59.3 misrepresentation refusal), while reversing the order collapses it to 0.2, showing WIFA-Boost is not merely a data-mixture effect.","Stronger fine-tuning is not a substitute for the group objective: a high-learning-rate WIFA-SFT run approaches A-GCRT on misrepresentation refusal but leaves OR near 68%, and variance-only or anchor-only components leave OR at 74.5% and 48.7%, respectively.","The safety gain is not capability-neutral: A-GCRT keeps MMLU close to the base model (69.0 versus 70.1 in Qwen) but still costs GSM8K performance, and the Llama setting shows the below-base OR result does not transfer across models and data sources."],"supporting_citations":[{"why":"Establishes the surface-form jailbreak failure mode that motivates refusing intent rather than form.","marker":"Wei et al., 2023"},{"why":"Supplies the AdvBench-style harmful seeds used to build the Qwen harmful intent groups.","marker":"Zou et al., 2023"},{"why":"SORRY-Bench is the transformed-harmful refusal benchmark carrying the main safety-side result (63.7 mutation-average refusal).","marker":"Xie et al., 2025"},{"why":"OR-Bench is the over-refusal benchmark carrying the main compliance-side result (17.4%).","marker":"Cui et al., 2024"},{"why":"XSTest provides the safe-prompt and unsafe-prompt refusal diagnostics for the trade-off analysis.","marker":"Röttger et al., 2024"},{"why":"HarmBench measures harmful refusal and supports the overlap check ruling out prompt duplication with the Qwen training seeds.","marker":"Mazeika et al., 2024"},{"why":"Consistency training is the conceptual source for the within-group variance regularization term in A-GCRT.","marker":"Xie et al., 2020"},{"why":"The Alpaca-style benign pool supplies the matched benign counterexamples in WIFA.","marker":"Taori et al., 2023"},{"why":"Intention Analysis is the reproduced prompt-time baseline whose intent-analysis format WIFA's targets echo.","marker":"Zhang et al., 2025a"}],"fun_headline_variants":["LLM over-refusal drops from 25.7% to 17.4% with intent-group training","Safety tuned on intent groups, not surface wrappers","WIFA: pair harmful and benign wrappers to teach intent refusal","Intent-group supervision: better refusal, fewer false alarms","LLMs refuse intent, not form—under wrapper augmentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The low-over-refusal result rests on a hand-built training signal, a single-position score comparing the model's next-token preference for refusal phrases versus polite-answer phrases, being a faithful proxy for real refusal behavior, even though the paper's own diagnostics show the score separates harmful from benign prompts at only about chance level and is often uncomputable because the marker it depends on is missing from the model's output.","fun_headline_variants_meta":{"raw":{"variants":["LLM over-refusal drops from 25.7% to 17.4% with intent-group training","Safety tuned on intent groups, not surface wrappers","WIFA: pair harmful and benign wrappers to teach intent refusal","Intent-group supervision: better refusal, fewer false alarms","LLMs refuse intent, not form—under wrapper augmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000656,"raw_usage":{"total_tokens":3012,"prompt_tokens":962,"completion_tokens":2050,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":1958}},"tokens_in":578,"tokens_out":2050,"duration_ms":13149,"temperature":1.0,"reasoning_tokens":1958,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:06:41.429462+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain A-GCRT with the decision score computed from shuffled or randomly chosen prefix token sets and check whether OR-Bench over-refusal still drops from 25.7% to roughly 17.4%; if the drop survives, the score-based group regularization is not what carries the result. A complementary check: measure free-generation refusal rates across many wrappers of the same harmful intent group; if refusal still swings sharply across wrappers, decisions track form rather than intent, and the central claim fails.","supporting_citations":[],"review_version":1}