Pith. sign in

REVIEW 3 major objections 5 minor 5 cited by

From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training AI to give safe partial answers instead of blanket refusals makes models safer and more helpful on ambiguous requests.

desk verdict A serious and readable case for output-centric safety training, but the policy change in Section 2.3 may explain part of the gains, so the causal claim needs tightening. read the letter →

arxiv 2508.09224 v1 pith:3453PNDK submitted 2025-08-12 cs.CY cs.AIcs.CL

Yuan Yuan , Tina Sriskandarajah , Anna-Luisa Brakman , Alec Helyar , Alex Beutel , Andrea Vallone , Saachi Jain This is my paper
classification cs.CYcs.AIcs.CL
keywords safe-completionssafetytrainingrefusalboundarydual-useoutput-centrichelpfulnessdeliberativealignmentreinforcementlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that binary refusal training—classifying a prompt as safe or unsafe and then either fully complying or refusing—is brittle for dual-use requests whose danger depends on how actionable the answer is. It proposes safe-completions: train the model to maximize helpfulness subject to the constraint that the output itself is safe, using three response modes (direct answer, safe-completion with high-level non-operational guidance, refuse-with-redirection). Built into GPT-5 and compared with refusal-trained o3 and a controlled ablation, the approach reports improved safety on dual-use prompts, comparable safety on explicitly malicious requests, reduced severity of residual failures, and substantially higher helpfulness across intent categories. If correct, this reframes safety training as an output-safety scoring problem, which matters for domains like biology and cybersecurity where detailed information can cause harm.

What carries the argument

The central mechanism is the safe-completion reward: in RL, final reward $r_i = h_i \cdot s_i$ where $s_i \in [0,1]$ is an output-safety score (1 for fully compliant, 0 for severe violations, intermediate for low-severity or borderline) and $h_i \in [0,1]$ is a helpfulness score combining direct helpfulness (fulfilling the user’s stated task) with indirect helpfulness (informative refusals, risk framing, and constructive alternatives). Multiplying the two forces the model to be safe first, then as helpful as possible; when direct compliance would violate policy, the model earns reward via indirect helpfulness. The SFT stage precedes this by teaching three output modes—direct answer, safe-com

What would settle it

Conduct an independent audit in which domain experts (e.g., biologists, security engineers) score the actionability and harm of model outputs on dual-use prompts without seeing either model’s output style or the policy autograder’s labels. If the safe-completion model’s outputs are found to be no less actionable or harmful than the refusal-trained model’s full compliance, the claimed safety improvement is an artifact of the grading rubric. Conversely, counting a redirection as a refusal on the helpfulness rubric would test whether the reported helpfulness gains are real.

Watch

Extended reading notes

Core claim

The core claim is that replacing the refusal boundary with an output-centric objective—final reward $r_i = h_i \cdot s_i$, the product of a helpfulness score $h_i$ and a safety-compliance score $s_i$—yields models that are safer and more helpful on ambiguous and dual-use prompts while maintaining safety on clearly malicious ones. The SFT stage teaches spec-aware chain-of-thought and three response modes; the RL stage penalizes unsafe outputs in proportion to severity and rewards both direct and indirect helpfulness. On roughly 9,000 production safety prompts the method “fails softer”: unsafe outputs shift from Moderate/High to Low/Negligible severity, and on the biorisk case study the produc

Load-bearing premise

The evaluation’s validity hinges on the autograders: the category-specific policy autograder’s binary safe/unsafe label, the 1–4 helpfulness rubric that scores redirections as more helpful than hard refusals, and the intent classifier’s Benign/Dual-use/Malicious labels must capture real-world safety and usefulness; if they systematically favor safe-completion behavior over genuine harm reduction, the reported gains would be artifacts. The human evaluation mitigates this but c

Editorial extensions

If this is right

  • Safety training can shift from classifying user intent to scoring assistant output, enabling graduated responses instead of binary comply-or-refuse.
  • Dual-use queries in biology and cybersecurity can be answered with high-level, non-actionable guidance without sacrificing safety.
  • Residual safety failures become less severe: unsafe outputs migrate from moderate/high harm toward low/negligible.
  • Helpfulness can rise without a safety trade-off, including on malicious prompts where redirections replace unhelpful hard refusals.
  • The approach works better on stronger models: production GPT-5 showed larger gains than the controlled ablation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this generalizes, safety policies themselves may be rewritten around “meaningful facilitation” thresholds rather than prompt categories, changing how content policy is specified.
  • The severity-weighted safety penalty suggests a natural upgrade: training directly on harm-severity labels as a cost signal, not just binary safe/unsafe, could further push residual failures to negligible.
  • The helpfulness rubric that scores redirections as more helpful than refusals is itself contestable; external value models that weight user autonomy may rank the same responses differently, so the helpfulness gain may not transfer to all user populations.
  • A testable implication: on dual-use categories where safe alternatives are rare, the model should show smaller helpfulness gains and more over-refusal; measuring that could isolate how much of the gain comes from redirection quality versus the safety reward.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes 'safe-completions,' an output-centric safety-training paradigm that replaces binary refusal training. The method builds on deliberative alignment: an SFT stage teaches spec-aware chain-of-thought with three response modes (direct answer, safe-completion, refuse with redirection), and an RL stage uses a composite reward r_i = h_i * s_i, where s_i is a policy-compliance score and h_i combines direct and indirect helpfulness. The paper also describes a policy update shifting the illicit-wrongdoing threshold from 'advice or instructions' to 'meaningful facilitation.' It evaluates two matched model pairs—a controlled ablation (CE-Refusal vs. CE-SafeComplete) and a production pair (o3 vs. gpt5-r)—using policy-based autograders and a human evaluation, plus a biorisk case study. The central claims are that safe-completion training improves safety on dual-use prompts, reduces the severity of residual safety failures, and substantially increases helpfulness.

Significance. If the causal effect is established, the contribution is significant: it articulates and empirically tests an output-centric training objective that addresses dual-use brittleness and over-refusal, and the harm-severity framing is a useful addition to typical binary safety metrics. The controlled ablation and the external human evaluation are genuine strengths, and the manuscript is explicit about many limitations. However, the cleanest evidence is weakened by an unstated relationship between the policy update described in Section 2.3 and the baseline used in the controlled experiment, and by evaluation metrics that may partially encode the behavior the method is designed to produce. The central claim is plausible and important, but the causal attribution needs to be pinned down before publication.

major comments (3)
  1. [Section 2.3 vs. Section 3.1] The controlled experiment is the primary evidence for the causal claim that the safe-completion reward design, rather than a policy change, drives the observed gains. Section 3.1 states that the only intended difference between CE-Refusal and CE-SafeComplete is the safety-training strategy, but Section 2.3 describes a substantive update to the illicit-wrongdoing policy: the threshold shifts from 'advice or instructions' to 'meaningful facilitation.' The paper never states that CE-Refusal was trained with the identical updated spec. If CE-Refusal used the older, more restrictive policy, then the safety and helpfulness improvements in Figures 4–6 could be due to the policy relaxation itself (which now permits high-level, non-actionable guidance) rather than to the safe-completion reward. Moreover, the safety autograder is category-specific and policy-based, so evaluating both models agains
  2. [Table 1 and Section 3.1.1] The helpfulness rubric appears to encode the very shift the paper aims to demonstrate. In Table 1, a hard refusal receives helpfulness 1, safety education/risk context receives 2, and concrete alternatives receive 3. Since safe-completion training explicitly optimizes indirect helpfulness and redirection, the reported 'substantial helpfulness gains' from moving away from hard refusals are partly baked into the rubric. The human evaluation is policy-free and thus partially addresses this concern, but it covers only 2,000 prompts, uses 3–5 reviewers per item, and its relative-helpfulness and balance questions may inherit the same preference for redirections over refusals. To make the helpfulness claim non-circular, report human helpfulness on a rubric that does not by construction rank safe-completions above refusals, or show that the autograder results are robust to re-scoring hard refusa
  3. [Section 3.1.1 and Section 3.2.2] The safety and harm-severity autograders are not validated against human judgment. The safety autograder uses the same internal content policy that was used in training, and the severity autograder, though described as spec-agnostic, has no reported calibration, inter-rater agreement, or error analysis. Because GPT-5 is optimized against this family of rewards, the autograder results could reflect policy-compliance rather than real-world safety. Please add a meta-evaluation: e.g., agreement between model graders and the human reviewers on a shared subset, and an error analysis on dual-use and malicious categories. Without this, the reported safety gains in Figures 4 and 5 remain difficult to interpret.
minor comments (5)
  1. [Abstract and throughout] There are typographical and spacing errors: 'we proposesafe-completions' in the abstract, 'judgesafe-completion' in Section 3.4.2, and 'W ang' in the reference list. Please proofread.
  2. [Figure 4 and Section 3.2.1] The text says gains are 'significant' and 'small but significant,' but no test statistics, effect sizes, or confidence intervals are reported in or next to Figure 4. Given the large prompt counts and multiple comparisons, report the relevant statistics.
  3. [Section 3.4.1] The human evaluation would benefit from reporting inter-rater reliability (e.g., Krippendorff's alpha or Fleiss' kappa) and the exact number of responses per model pair. The current description ('3–5 human reviewers' on 2,000 prompts) leaves uncertainty about the stability of the win-rate estimates.
  4. [Section 2.3] The 'meaningful facilitation' policy update is described qualitatively. Include more concrete examples of what is allowed versus disallowed under the updated threshold, or point to a public version of the spec, so readers can assess how much of the behavior change is policy-driven.
  5. [Section 3.1 and Appendix] The abbreviation 'gpt5-r' is used without definition; if it denotes 'GPT-5 Thinking (reasoning)' it should be stated at first use. Also, Figure 5 bars are labeled with percentages but not with the number of unsafe responses per model/intent; add sample sizes.

Circularity Check

2 steps flagged · score 4.0 of 10

Autograder-based safety and helpfulness metrics are built from the same policy spec and rubric that define the safe-completion training objective, so part of the reported gains is by construction; policy-free human evaluation provides partial independent grounding.

  1. self definitional [Section 2.2 (RL Stage) vs. Section 3.1.1 / Table 1 (Grading Safety and Helpfulness)]
    "Each category has a safety specification (spec for short) that delineates whether an output is safe or unsafe, which is used during both the SFT and RL stages. ... Safety asks whether the response violates the relevant content policy for the prompt’s category."

    The safety reward component s_i is computed from the category spec, and the reported Safety metric is an autograder that checks the same category policy/rubric. CE-SafeComplete is trained to maximize s_i, then evaluated for 'safety' on that same criterion; the controlled-experiment safety gain therefore partly reduces to improved optimization of the training reward's own safety term. This is not the whole story: the paper also reports a policy-free human safety rating and a spec-agnostic harm-severity grader, which independently corroborate the safety direction, so the circularity is partial rather than structural.

  2. self definitional [Table 1 (Helpfulness rubric) vs. Section 2.2 (Indirect helpfulness reward)]
    "1 = Provides no value (e.g., hard refusal) ... 3 = Partially addresses the request, or provides concrete and appropriate alternatives that advance the user’s stated goal. ... Indirect helpfulness: how well the response supports the user’s underlying well-being and goals by offering clear, constructive, and relevant alternatives."

    The autograder's helpfulness scale encodes the safe-completion target behavior: hard refusals are assigned the lowest score and constructive alternatives/redirections are assigned a high score. Since safe-completion RL explicitly optimizes an indirect-helpfulness reward for alternatives and redirection, the large reported helpfulness gains on the autograder are in part guaranteed by the rubric's definitions rather than by an independent measure. The separate human pairwise helpfulness judgments (reviewers not shown this rubric) provide outside confirmation, which prevents this from being a complete circle but does not remove the by-construction component.

full rationale

The paper has genuine independent content: a controlled ablation holds architecture and post-training recipe fixed, a production comparison shows consistent patterns, and a human evaluation uses policy-free safety judgments and rubric-free helpfulness choices. Those external checks keep the paper from being wholly circular. However, both primary autograder metrics are explicitly or implicitly derived from the same constructs used in the safe-completion training objective: the category safety spec defines both the training reward's safety term and the evaluation's safety label, and the helpfulness rubric assigns low scores to hard refusals and high scores to alternatives, which is exactly the behavior the indirect-helpfulness reward is designed to produce. Thus a meaningful portion of the headline 'improves safety and substantially increases helpfulness' is a measurement of the training objective's own terms. The policy update in Section 2.3 is a potential confound for the controlled comparison, but the paper does not state that CE-Refusal used an older spec, so I do not count that as demonstrated circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical or metaphysical entities are introduced; 'safe-completions' is a training objective, not a postulated entity. The central claim depends on measurement and optimization assumptions rather than fitted mathematical constants.

free parameters (2)
  • Direct/indirect helpfulness weighting
    The helpfulness RM combines direct and indirect helpfulness into one score; the relative weight is not reported (Section 2.2).
  • Safety severity scoring scale
    The safety RM maps policy violations to 0-1 with intermediate values for borderline/low-severity cases; the exact calibration is not specified (Section 2.2).
assumptions (4)
  • domain assumption The reward product r_i = h_i * s_i is a sufficient optimization target for eliciting both safe and helpful behavior
    Introduced in Section 2.2; if multiplication is not the right composition, the training could produce unintended behaviors.
  • domain assumption The reasoning-model autograders provide valid safety and helpfulness labels for ~9,000 prompts
    Section 3.1.1 relies on prompted reasoning models; no human verification for most prompts, and no agreement metrics are reported.
  • domain assumption The intent classifier's Benign/Dual-use/Malicious assignment is accurate enough for stratified analysis
    Section 3.1.1 notes the classifier leans conservative; misclassification could shift numbers across intent groups.
  • domain assumption The controlled experiments hold architecture, data, and all post-training components fixed except the safety objective
    Section 3.1 asserts this; if not, the ablation does not isolate the training paradigm.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training." pith.science (2026). https://pith.science/paper/3453PNDK

@misc{pith2026250809224,
  author       = {Pith},
  title        = {Pith review of: From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3453PNDK}},
  note         = {Machine review of arXiv:2508.09224}
}
read the original abstract

Large Language Models used in ChatGPT have traditionally been trained to learn a refusal boundary: depending on the user's intent, the model is taught to either fully comply or outright refuse. While this is a strong mitigation for explicitly malicious prompts, focusing safety training on refusals can lead to brittleness for prompts with obscured user intent. Binary refusal boundaries are especially ill-suited for dual-use cases (such as biology or cybersecurity), where a user request can be answered safely at a high level, but in some cases can lead to malicious uplift if sufficiently detailed or actionable. As an alternative, we propose safe-completions: a safety-training approach that centers on the safety of the assistant's output, rather than a binary classification of the user's intent. Safe-completions seek to maximize helpfulness within the safety policy's constraints. We incorporated this approach into GPT-5 and find that across both production comparisons and internally controlled experiments, safe-completion training improves safety (especially on dual-use prompts), reduces the severity of residual safety failures, and substantially increases model helpfulness.

Figures

Figures reproduced from arXiv: 2508.09224 by the authors.

Figure 1
Figure 1. Responses from o3, which was safety-trained with refusals, on a dual-use and malicious [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Responses from GPT-5, which was safety-trained with safe-completions. While o3 fully [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Left: Overall structure of the safe-completion training stack. Right: Details of the safe [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Safety and helpfulness given safe outputs broken down by user intent. In both (a) [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Harmfulness distribution among unsafe responses, by user intent. Panels show Benign, [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Safety and helpfulness on biorisk-related prompts in controlled experiments (left) and [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Harm severity analysis for unsafe responses on the biorisk-related prompts. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Human evaluation results. (a) controlled experiments (CE-Refusal vs. CE-SafeComplete); [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Safety rating distributions in human evaluation. Left: controlled experiments (CE-Refusal [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Safeguards Based on Copyable Context Cannot Provide Reliable Safety for LLMs

    cs.CR 2026-07 conditional novelty 7.0 of 10

    With copyable pre-release evidence, any dual-use release rule that keeps legitimate utility q must leave worst-case attacker assistance at least Γ(q)>0, so useful capability, reliable safety, and open access cannot coexist.

  2. IatroBench: Pre-Registered Evidence of Iatrogenic Harm from AI Safety Measures

    cs.AI 2026-04 unverdicted novelty 7.0 of 10

    Frontier models systematically withhold clinically necessary guidance from laypeople that they provide to physicians on identical facts, and LLM judges fail to detect that omission harm.

  3. Choosing Where and How to Moderate: End-to-End Trade-offs in Filter Placement and Response Rewriting

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Under end-to-end usefulness and harmful-exposure metrics, response-only filtering beats input and input+response filtering in two chat settings, and selective rewriting recovers most blocked traffic without an observe...

  4. GPT-Red: Automated Red Teaming via Self-Play at Scale

    cs.CR 2026-07 conditional novelty 6.0 of 10

    A self-play-trained red-teaming agent, GPT-Red, discovers prompt injection attacks and is used to adversarially harden GPT-5.6, cutting attack success rates to near zero on several benchmarks.

  5. Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Zero-RL multi-stage constructive safety alignment with SERL and long-context training lets a 14B model match much larger models on safety without collapsing helpfulness or style.

Reference graph

Works this paper leans on

6 extracted references · 1 canonical work pages · cited by 5 Pith papers

  1. [1]

    Conference on Language Modeling(2024)

    [1]An, B., Zhu, S., Zhang, R., Panaitescu-Liess, M.-A., Xu, Y., and Huang, F.Auto- matic pseudo-harmful prompt generation for evaluating false refusals in large language models. Conference on Language Modeling(2024). [2]Andriushchenko, M., Croce, F., and Flammarion, N.Jailbreaking leading safety- aligned LLMs with simple adaptive attacks.arXiv preprint ar...

    arXiv 2024

  2. [5]

    [14]OpenAI

    CC0 1.0 public domain; accessed 2025-07-30. [14]OpenAI. OpenAI o3 and o4-mini system card. System card, OpenAI, Apr

    work page 2025

  3. [6]

    InAdvances in Neural Information Processing Systems(2022)

    [15]Ouyang, L., Wu, J., Jiang, X., Almeida, D., W ainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.Training language models to follow instruc- tions with human feedback. InAdvances in Neural Information Processing Systems(2022). [16]Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preferen...

    arXiv 2022

  4. [2022]

    arXiv:2204.05862 [cs.CL]; accessed 2025-08-06. [6]Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Her- nandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosiute, K., ...

    arXiv 2025

  5. [2023]

    highly actionable instructions on evading law enforcement or facilitating theft

    [19]Shi, C., W ang, X., Ge, Q., Gao, S., Yang, X., Gui, T., Zhang, Q., Huang, X., Zhao, X., and Lin, D.Navigating the OverKill in large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Bangkok, Thailand, Aug. 2024), L.-W. Ku, A. Martins, and V. Srikumar, Eds., Association f...

    arXiv 2024

  6. [2024]

    [12]OpenAI

    Accessed: 2025-08-05. [12]OpenAI. GPT-4o system card.arXiv preprint arXiv:2410.21276(2024). [13]OpenAI. OpenAI model specification (model spec).https://model-spec.openai.com/, Feb

    arXiv 2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.