{"id":"24a273ae-0200-4e46-a8dc-a2ae5a155c48","arxiv_id":"2502.02153","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A decoding-time method called TSDI estimates and removes the context-free refusal bias caused by safety alignment, improving helpfulness while keeping safety.","lead":"This paper finds that safety training makes language models refuse harmless questions too often, by pushing up the probability of negative words like 'sorry' and 'unfortunately' no matter what the user asked. The authors propose a simple decoding-time fix that estimates this bias from random prompts and subtracts it, improving helpfulness while keeping safety scores high.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central transfer assumption is unverified: the bias b_i estimated on random word-salad prompts may not equal the alignment bias on real harmful prompts, so subtracting it could suppress needed refusals; the paper's safety evaluation does not isolate this.","rationale":"The reader's conditional verdict and weakest assumption identify the same load-bearing concern: the method is a linear logit correction, and its validity for safety depends on the estimated bias being both context-free and not removing tokens that are functionally necessary for refusal. I see no internal inconsistency in Proposition 1; the algebra is a straightforward asymptotic identity. The weakness is interpretive: the paper needs E_{~rho}[g_theta] to be a pure side effect, but DPO-trained models can have nonzero g_theta on arbitrary inputs for reasons unrelated to safety (e.g., token priors, OOD degeneration), and the appendix's own examples of generation corruption under strong alignment show that random-input behavior is not always representative. The empirical evidence for safety preservation is consistent across two evaluators and three categories, which is real support, but it is not a direct test of the transfer assumption. For these reasons the paper should not be accepted as-is; the conditional verdict stands pending the proposed bias-source comparison. This is a normal reproducibility and validity check, not a claim of fraud or internal contradiction.","tokens_in":31312,"tokens_out":8004,"duration_ms":87607,"concrete_test":"Estimate three bias vectors from (a) random word-salad prompts as in Sec. C.4, (b) natural harmless prompts from AlpacaEval/MMLU, and (c) harmful SALAD-Bench prompts, all with L=20 and |D|=500. Compare the refusal-token components (sorry, cannot, unfortunately, no) at positions 1-5. Then run the Sec. 5 evaluations (MD-Judge, Llama Guard 3, GPT-4 win rate) using each b_i on both harmless and harmful sets. If b_i^(random) differs materially in sign or magnitude from b_i^(harmful), or if b_i^(harmless) fails to reproduce the Pareto-front gain, the transfer premise is falsified and safety preservation is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"TSDI's central premise (Sec. 4.1-4.3) is that b_i = E_{~rho}[f_{pi_theta} - f_{pi*_r}] over 500 random word-salad prompts (Sec. C.4) is a pure context-free side effect of safety alignment, so the same vector can be subtracted from every real prompt. Proposition 1 only proves an algebraic identity: after debiasing, g_theta is replaced by g_theta - E_{~rho}[g_theta]. It does not establish that E_{~rho}[g_theta] is a side effect rather than an out-of-distribution artifact, nor that the debiased g_theta still assigns refusal tokens high safety value on harmful inputs. The Sec. 4.3 argument that random prompts have no overlap with D and hence g_theta should be zero there is not implied by DPO training: parameter sharing lets the safety update alter logits on arbitrary inputs. If the random-prompt expectation overstates the refusal-token bias actually needed on red-team prompts, uniform subtraction suppresses refusals on harmful queries. The reported SALAD-Bench safety scores do not isolate this effect, and the compliance-rate improvement is partly mechanical because the method lowers the very refusal keywords counted in Sec. C.6; the GPT-4 win-rate gain is the non-circular evidence, so the transfer premise is the load-bearing link between that gain and the safety claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies safety-aligned language models and argues that safety alignment induces a context-free token-level bias toward negative/refusal tokens, which hurts helpfulness on harmless prompts. It proposes TSDI, a learning-free decoding method that estimates per-position logit biases on randomly constructed prompts and subtracts them at generation time. Experiments with SACPO-style DPO safety alignment on an Alpaca-7B replica show improvements in compliance rate and GPT-4 helpfulness win rate while aggregate safety scores on SALAD-Bench remain high, yielding a claimed improvement of the safety-helpfulness Pareto front.","tokens_in":31627,"tokens_out":9264,"duration_ms":93300,"significance":"If validated, TSDI would be a simple, training-free intervention that improves the safety-helpfulness trade-off for DPO-style safety-aligned models using only logit access. The paper has notable strengths: the Proposition 1 algebra is clean; the qualitative Pareto-front improvement is reported across two safety evaluators, two token pools, several β/λ settings and iteration counts, and three seeds; and the ablations on L and token-pool choice are useful. The main limitations are that the central transfer assumption is not independently verified, the compliance-rate metric is partly aligned with the mechanism TSDI suppresses, and the safety evaluation does not isolate whether debiasing erodes refusals on harmful prompts.","major_comments":[{"comment":"The load-bearing premise of TSDI is that the per-position bias vector b_i estimated from 500 random word-salad prompts (Section 4.1, Appendix C.4) is a context-free side effect of safety alignment that transfers to all real prompts, including harmful red-team prompts. Proposition 1 establishes only the algebraic identity that debiasing replaces g_theta by g_theta - E_{\\tilde rho}[g_theta]; it does not show that E_{\\tilde rho}[g_theta] is an unintended bias rather than an out-of-distribution artifact, nor that the debiased policy still assigns high safety value to refusal tokens on harmful inputs. The argument in Section 4.3 that random prompts have no overlap with training distribution D and hence should have zero g_theta is not implied by DPO training, because parameter sharing lets the safety update alter logits on arbitrary inputs. I ask for a direct test of the transfer premise, for example comparing b_i estimated on random prompts with b_i estimated on harmful prompts, and an analysis of whether debiasing flips any refusals on harmful queries into unsafe completions. Without such a test, the 'maintaining safety' claim is not fully established.","section":"Sections 4.1-4.3"},{"comment":"The compliance-rate metric is partly circular with TSDI. Appendix C.6 lists 53 keywords including 'I'm sorry', 'Sorry', 'I apologize', 'I cannot', 'I do not', 'Unfortunately', and 'No,', which are exactly the refusal-token families whose logits TSDI is designed to suppress. Consequently, the large compliance-rate improvement in Figure 5a is expected by construction and should not be presented as independent evidence of helpfulness. The GPT-4 helpfulness win rate is the non-circular outcome and should be the primary helpfulness measure; please report its confidence intervals and explicitly state that the compliance metric is a mechanism sanity check rather than a helpfulness evaluation.","section":"Section 5.1"},{"comment":"The safety-maintenance claim relies on aggregate per-category safety scores from MD-Judge and Llama Guard 3, but no confidence intervals are reported for these scores or for the GPT-4 win rates. The hypervolume significance test in Tables 3-4 reports zero standard deviation for the without-TSDI condition on two of three categories, which is not explained. More importantly, an aggregate safety score can remain high while debiasing erodes refusals on a subset of harmful prompts; the red-team examples in Appendix D.1 are anecdotal and show identical w/o and w/ TSDI outputs for several categories. Please provide per-category or per-prompt safety changes before and after debiasing, and a test of whether any previously safe refusal becomes unsafe after debiasing.","section":"Section 5.2 and Appendix C.8"}],"minor_comments":[{"comment":"The text says 66 safety categories with 68 prompts per category, giving 4488 prompts, but Figure 1 and Table 5 report only 16 category numbers; please clarify the relationship between subcategories and the 16 macro categories used in the figures.","section":"Section 3.1"},{"comment":"There is a typo in the second paragraph: 'Speficially' should be 'Specifically'.","section":"Section 2.1"},{"comment":"The category label 'Influence 0perations' contains a typo ('0' for 'O').","section":"Table 5"},{"comment":"The softmax function is denoted σ in Eq. (6) but SOFTMAX earlier in Section 4.1; please unify the notation.","section":"Equation (6)"},{"comment":"The example random prompt is word-salad; the statement that this construction is 'reasonably close to the normal distribution of text' is overstated. Please soften this or provide a quantitative justification.","section":"Appendix C.4"},{"comment":"The AlpacaEval dataset is used but not cited; please add the appropriate reference.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"I am sympathetic to the core idea and the empirical pattern is suggestive. The main gate for publication is verification of the transfer premise and a less circular helpfulness evaluation. I would also encourage the authors to release code and evaluation details, as the current appendix gives good detail but no runnable artifact."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core result is that a token-level refusal bias induced by DPO-style safety alignment can be estimated from random word-salad prompts and subtracted at decoding time, yielding a modest but consistent improvement in the safety-helpfulness trade-off across several settings. That finding holds up better than the stress-test note suggests, though the compliance metric is partly circular and the inference-time transfer assumption deserves closer scrutiny.\n\nWhat is actually new: TSDI itself. Prior inference-time methods learn a reward model or require hidden-state access; TSDI only needs logits and a sample of random prompts. The observation that safety alignment inflates logits for refusal tokens on random inputs is well documented with token-level logit plots, including the surprising finding that removing refusal-token responses from the training data does not remove the bias. The category-level vulnerability analysis of Safe RLHF and SACPO is also useful: they show the adult-content vulnerability is not predictable from the SFT model or dataset quality.\n\nThe evaluation is unusually thorough: two safety evaluators, three categories, four beta/lambda values, three iteration counts, three seeds for hypervolume, robustness checks on L and token pool, and comparisons to SafeDecoding and Self-reminder. The GPT-4 win rate improving from 0.59 to 0.67 for the main setting is the non-circular evidence that helpfulness genuinely improves. Proposition 1 is straightforward algebra, but it is correct and clarifies what the debiasing does.\n\nNow the soft spots. The compliance-rate metric is partly circular: the 53 keywords are the very refusal tokens TSDI suppresses, so it is not surprising that compliance improves. The paper should present the GPT-4 win rate as the primary helpfulness metric and treat compliance as supporting. The transfer assumption is the real gap. The authors argue that random prompts are outside the preference dataset, so any logit shift is a side effect; but parameter sharing means DPO updates can affect arbitrary inputs. The method subtracts a constant vector estimated on random prompts, and if harmful prompts need a larger refusal-token boost than the random-prompt average captures, subtracting it could erode refusals. Their SALAD-Bench safety scores do not show erosion, which is reassuring, but they never directly measure refusal-token bias on harmful prompts. That missing measurement is the clearest fixable weakness. Also, no code or bias vectors are released, which slows reproduction.\n\nThis is a practical contribution for people working on safety alignment and inference-time control; it does not change the alignment paradigm. The paper deserves a serious referee. I would send it out, asking the authors to add a direct test of the transfer assumption and to reframe the compliance metric as secondary. With those changes, this is a solid, citable result.","headline":"TSDI is a useful, simple debiasing trick for DPO-style safety alignment, with solid but partly circular evaluation; the transfer assumption needs a direct test, but the core Pareto-front improvement is credible.","tokens_in":32174,"tokens_out":4188,"would_cite":true,"duration_ms":43399,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Safety alignment can plant an input-independent bias toward rejective tokens, and the paper's TSDI method estimates and subtracts that bias at decoding time to improve helpfulness without sacrificing safety.","keywords":["safety alignment","token-level debiasing","logit bias","over-refusal","helpfulness-safety trade-off","direct preference optimization","inference-time alignment","SALAD-Bench"],"falsifier":"Compute the per-position bias $b_i$ from random prompts and separately from 500 ordinary harmless prompts; if the two vectors diverge enough that subtracting the random-prompt $b_i$ increases the refusal rate on the harmless set or produces unsafe completions on a red-team suite, TSDI's transfer premise is falsified.","tokens_in":31090,"feed_emoji":"🛡️","tokens_out":9182,"duration_ms":83728,"temperature":0.7,"pith_summary":"The paper sets out to show that DPO-style safety alignment does more than suppress harmful outputs: it plants an input-independent bias that makes the model prefer negative, rejective tokens even on harmless prompts. Because that bias sits in the output logits, the authors argue it can be estimated cheaply and removed at inference time without retraining. Their Token-level Safety-Debiased Inference (TSDI) estimates a per-position bias vector from 500 random word-salad prompts, then subtracts it from logits during generation. Across the PKU-SafeRLHF/SACPO setup, they report that TSDI raises compliance and helpfulness win rates while keeping MD-Judge and Llama Guard 3 safety scores at the same level, improving the safety-helpfulness trade-off. If true, this gives a plug-in remedy for over-refusal in safety-aligned models that only requires logit access.","feed_headline":"Subtracting a refusal bias makes safety-aligned LLMs more helpful","feed_subtitle":"A training-free logit correction estimated from random prompts lifts helpfulness without lowering safety scores.","key_machinery":"The load-bearing object is the per-position logit bias vector $b_i$, defined as the average difference between the safety-aligned policy's logits and the reference policy's logits over randomly constructed prompt-response pairs $\\tilde{D}$: $b_i = \\frac{1}{|\\tilde{D}|}\\sum_{(x,y)\\in\\tilde{D}} [f_{\\pi_\\theta}(x\\oplus y_{1:i-1}) - f_{\\pi_r^*}(x\\oplus y_{1:i-1})]$. TSDI subtracts $b_i$ from the aligned model's logits at the $i$-th generated token, i.e. it samples from $\\mathrm{softmax}(f_{\\pi_\\theta}(x\\oplus y_{1:i-1}) - b_i)$. Proposition 1 shows this is equivalent to replacing the implicitly learned safety function $g_\\theta$ by a version centered with respect to the random-prompt distribution, removing the part of the safety signal that is independent of the input. The random prompts are built by sampling tokens from an MMLU-derived pool, and the paper uses $L=20$ and $|\\tilde{D}|=500$; the bias is recomputed for each trained model.","core_discovery":"The central discovery is that safety alignment introduces a measurable, context-free shift in next-token logits: after DPO-based safety alignment, the logits of negative tokens such as 'sorry', 'unfortunately', 'none', and 'cannot' rise substantially even when the input is a randomly assembled token salad. The size of the shift grows as the KL penalty shrinks and varies by generation position. TSDI treats this shift as a bias vector and subtracts it during decoding, which the authors formalize in Proposition 1 as re-centering the implicit safety function used by the aligned policy. In their experiments on an Alpaca-7B replica aligned via SACPO, TSDI improves the compliance rate on harmless prompts without reducing safety scores, and the helpfulness win rate against the SFT model rises from 0.59 to 0.67 for the $\\beta/\\lambda = 0.025$, 200-iteration model while safety is preserved. They also report that the same pattern holds under Llama Guard 3 evaluation and that the method is stable to the choice of debiasing length and token pool.","pith_inferences":["Editorial inference: because the bias is estimated once per model and then applied indiscriminately, the same technique should transfer to other tasks and prompt distributions, and its failure modes would show up as a mismatch between the random-prompt bias and the actual bias on real prompt distributions.","Editorial inference: the paper records cases where removing the initial negative tokens still leaves an unhelpful continuation, so token-level debiasing is a partial fix; a natural next step is to apply the same re-centering idea to hidden states or later decoding positions.","Editorial inference: a cheap test of the transfer assumption would be to compute the bias vector on ordinary harmless prompts and compare it with the random-prompt vector; if they diverge, TSDI could be restricted to prompts that are already classified as safe."],"forward_implications":["Safety-aligned models trained by DPO-style safety alignment carry an input-independent bias toward rejection tokens, so even harmless prompts can draw openings such as 'I\\'m sorry' or 'Unfortunately'.","Subtracting the estimated per-position bias at decoding raises the compliance rate on harmless prompts without lowering MD-Judge or Llama Guard 3 safety scores.","The helpfulness win rate versus the SFT model improves while safety is maintained, for example from 0.59 to 0.67 for the debiased model trained with $\\beta/\\lambda = 0.025$ for 200 iterations.","Because TSDI needs only logit access, it can be applied to any logit-accessible safety-aligned model without retraining or fitting a separate reward model.","Similar results hold for debiasing lengths $L=5,10,20$ and for token pools built from either MMLU or MS MARCO, indicating the bias estimate is not tied to one prompt-construction recipe."],"supporting_citations":[{"why":"supplies the Safe RLHF models and evaluation setup whose category-level vulnerabilities the paper re-measures.","marker":"[Dai et al., 2024]"},{"why":"supplies the SACPO stepwise safety-alignment procedure whose trained policies TSDI debiases.","marker":"[Wachi et al., 2024]"},{"why":"provides the DPO objective underlying both helpfulness and safety alignment in the experiments.","marker":"[Rafailov et al., 2024]"},{"why":"provides the PKU-SafeRLHF preference dataset used for safety alignment and the data-cleansing analysis.","marker":"[Ji et al., 2024a]"},{"why":"provides the SALAD-Bench balanced red-teaming prompts and the MD-Judge safety classifier used to score safety categories.","marker":"[Li et al., 2024a]"},{"why":"gives the 47 refusal keywords used for the compliance-rate metric and motivates why controlling initial tokens matters.","marker":"[Zou et al., 2023]"},{"why":"supplies the shallow safety alignment perspective that frames the observed over-refusal bias.","marker":"[Qi et al., 2024]"},{"why":"supplies the MMLU token pool from which the random prompts for bias estimation are constructed.","marker":"[Hendrycks et al., 2021]"}],"fun_headline_variants":["TSDI removes refusal bias, boosting helpfulness in aligned LLMs","Safety-aligned LLMs get a helpfulness boost from token-level debiasing","Training-free logit debiasing improves helpfulness of safety-aligned LLMs","Correcting refusal bias boosts alignment without hurting safety"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the bias measured on 500 random word-salad prompts is the same input-independent bias that acts on real user prompts, so subtracting it helps real responses rather than distorting them.","fun_headline_variants_meta":{"raw":{"variants":["TSDI removes refusal bias, boosting helpfulness in aligned LLMs","Safety-aligned LLMs get a helpfulness boost from token-level debiasing","Training-free logit debiasing improves helpfulness of safety-aligned LLMs","Correcting refusal bias boosts alignment without hurting safety"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001028,"raw_usage":{"total_tokens":4340,"prompt_tokens":960,"completion_tokens":3380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":3303}},"tokens_in":576,"tokens_out":3380,"duration_ms":22703,"temperature":1.0,"reasoning_tokens":3303,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T13:07:54.420109+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the per-position bias $b_i$ from random prompts and separately from 500 ordinary harmless prompts; if the two vectors diverge enough that subtracting the random-prompt $b_i$ increases the refusal rate on the harmless set or produces unsafe completions on a red-team suite, TSDI's transfer premise is falsified.","supporting_citations":[{"cited_title":"Stepwise Alignment for Constrained Language Model Policy Optimization","cited_arxiv_id":"2404.11049","evidence_quote":"supplies the SACPO stepwise safety-alignment procedure whose trained policies TSDI debiases."}],"review_version":1}