{"id":"e360871b-42e7-4f6f-bd2e-9fa6bc3cf8a1","arxiv_id":"2507.14987","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"AlphaAlign uses pure reinforcement learning with a verifiable safety reward to make LLMs refuse harmful requests with explicit reasoning while preserving helpfulness on benign queries.","lead":"Researchers trained LLMs to be safer using reinforcement learning with a simple rule-based reward: if a prompt is flagged harmful, the model must explain its reasoning and refuse, and if benign, it should answer helpfully. The method needs only binary harmful/benign labels and very few training steps, and it aims to reduce both dangerous answers and overly cautious refusals.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The refusal verifier used for training (Eq. 4) is also the first stage of ASR evaluation, so the reported safety gains may reflect phrase-list overfitting rather than genuine harmfulness recognition.","rationale":"The reader's weakest assumption identifies the same mechanism: Vr is a hand-crafted phrase matcher, it is the training reward in Eq. 4, and it is also the first stage of the safety evaluation. This is the most load-bearing concern because every quantitative safety claim in Tables 2 and 4 uses this metric. If the phrase list is too permissive or too narrow, the training reward and evaluation metric move together, so the model can maximize the reported ASR by copying the canonical refusals rather than learning to recognize harmfulness. The paper's own Limitations section concedes that Vr is a string-matching verifier, and no independent safety evaluation is reported; the second-stage Llama-Guard check is only reached for responses not already classified as refusals by the phrase list. The utility claim is also partially contradicted by Llama3.2-3B-Instruct (GSM8K -8.3, MMLU -2.1 in Table 3), but that is a narrower issue affecting one backbone, whereas the verifier circularity affects the headline safety-utility trade-off claim across all backbones. A concrete independent evaluation is feasible and would either confirm the method's robustness or show that its measured gains are partly an artifact of optimizing the evaluation's own refusal detector. Since the paper is already conditional with high correctness risk in the reader's verdict, this stress-test does not change the verdict, but it sharpens the primary condition that needs to be met.","tokens_in":20985,"tokens_out":3364,"duration_ms":34696,"concrete_test":"Recompute all ASR numbers in Table 2 using Llama3-Guard-8B as the sole independent judge on every response, removing the phrase-matching verifier Vr from the evaluation pipeline, and additionally have a human annotator label a stratified random sample of 100 responses per model-condition as 'genuine refusal' vs. 'non-refusal'. If AlphaAlign's ASR advantage over SCoT and Circuit Breaker shrinks or reverses under independent judgment, the central safety claim is an artifact of optimizing Vr.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that AlphaAlign improves refusal of harmful content while reducing over-refusals—rests on the refusal verifier Vr. In Eq. 4, Vr is the training reward: harmful prompts receive +ra if Vr(y)=1, and benign prompts receive -ra if Vr(y)=1. Vr is implemented as a soft match against the handcrafted phrase list in Appendix B.1, Table 4 (e.g., \"Sorry, I can't comply\", \"I cannot\"). The same Vr is applied first in the ASR evaluation pipeline (Section 4.1: \"evaluated by refusal verifier first, then by Llama3-Guard-8B\"), so a response matching one of the listed phrases is classified as a safe refusal without requiring a substantive refusal or an independent judgment of why the prompt is harmful. PPO training can exploit this by making the final answer literally echo one of the 25 canonical phrases—the template in Table 1 already instructs the model to use \\boxed{Sorry, I can't comply} for unsafe queries—so the reward signal is substantially a test of string matching, not of safety understanding. The CKAS analysis in Section 4.5 and the qualitative case studies do not break this circularity, because an optimized policy can emit safety-sounding rationales together with a canonical refusal phrase. Consequently, the reported safety gains over SCoT and Circuit Breaker may largely reflect that AlphaAlign overfits the evaluation's own first-stage classifier.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AlphaAlign, a pure reinforcement learning framework for LLM safety alignment. The method uses a dual reward: a verifiable safety reward that rewards correctly formatted and explicit refusals for harmful prompts and penalizes over-refusals for benign prompts, and a normalized helpfulness reward that encourages high-quality responses to benign queries. Training requires only binary prompt-safety labels and uses PPO with fewer than 200 steps. Experiments on Qwen2.5 and Llama3.2 backbones report improved refusal of harmful and jailbroken prompts, reduced over-refusal, maintained or improved utility, and evidence of proactive safety reasoning, compared with direct refusal, Circuit Breaker, and SCoT baselines.","tokens_in":21342,"tokens_out":4407,"duration_ms":48179,"significance":"If the empirical claims hold, AlphaAlign would be a simple and scalable safety-alignment recipe: it needs only binary labels, uses a verifiable reward, and reports convergence in fewer than 200 RL steps. The framework is clearly presented, the code is released, and the comparison against several existing alignment methods is useful. However, the central safety evaluation shares its refusal verifier with the training reward, and the utility table contains an unexplained constant AlpacaEval score, so the headline claims are not yet established by the reported evidence.","major_comments":[{"comment":"The ASR evaluation pipeline is not independent of the training reward. Eq. (4) rewards the model for matching the refusal verifier Vr on harmful prompts, and Section 4.1 states that ASR is 'evaluated by refusal verifier first, then by Llama3-Guard-8B.' Because Vr is a soft match against the handcrafted phrase list in Appendix B.1, Table 4, and the prompt template in Table 1 explicitly instructs the model to answer unsafe queries with \\boxed{Sorry, I can't comply}, the low ASR numbers may reflect phrase-list overfitting rather than genuine harmfulness recognition. The authors should report ASR with Llama3-Guard-8B alone (or with an independent judge) on the final-answer content, and clarify whether Llama3-Guard is applied at all when Vr already classifies a response as a refusal.","section":"Section 4.1 and Eq. (4)"},{"comment":"The AlpacaEval column is exactly 50.00 for all three backbones, while the parenthetical deltas are +6.7, +10.0, and +7.9. These two representations are inconsistent: if 50.00 is the AlphaAlign score, the baseline cannot be 50.00 with a nonzero delta; if 50.00 is the baseline reference score, the table does not report AlphaAlign's score. The authors should state explicitly which model is the AlpacaEval reference and provide the actual AlphaAlign win rates. As printed, this table does not support the claim that AlphaAlign maintains or improves instruction-following ability.","section":"Table 3"},{"comment":"All safety and utility results are reported as single runs with no error bars, confidence intervals, or seed variation. Given that many ASR values are near zero (e.g., 0.0 in several columns of Table 2) and some utility deltas are small (e.g., -0.1 MMLU), the paper cannot distinguish a real effect from training noise. At least for one backbone, the authors should run multiple seeds and report the mean and variance.","section":"Sections 4.2-4.4"},{"comment":"The CKAS metric is a heuristic defined in this paper and is not validated against any human judgment or an established safety-reasoning metric. The claim that AlphaAlign achieves 'deep alignment via proactive safety reasoning' relies on CKAS and on qualitative case studies in Appendix D, which show canonical refusal phrases but not that the safety reasoning is causally responsible for the refusal. The authors should either validate CKAS against human annotations or provide an independent behavioral test of safety reasoning, for example by measuring whether the safety rationale survives paraphrasing or whether the model can explain its refusal in its own words.","section":"Section 4.5 and Appendix C.5.1"}],"minor_comments":[{"comment":"\"We proposeAlphaAlign\" is missing a space; should read \"We propose AlphaAlign.\"","section":"Abstract"},{"comment":"The ASR description \"evaluated by refusal verifier first, then by Llama3-Guard-8B\" should be expanded to specify exactly how the two stages are combined, because the interpretation affects the validity of the reported safety numbers.","section":"Section 4.1"},{"comment":"The phrase list contains a typo: \"I'\\m sorry\" should be \"I'm sorry\"; also \"\\u201cI'm sorry\" appears to contain a stray escape.","section":"Appendix B.1, Table 4"},{"comment":"The column header \"Jailbreaktrigger\" is inconsistently formatted and should be \"JailbreakTrigger\" to match the benchmark name.","section":"Table 2"},{"comment":"The sentence \"This avoids penalizing lower-quality but safe outputs with negative rewards\" is slightly misleading: Eq. (5) assigns zero reward to non-refusal outputs below the mean, not a negative reward; the text can be clarified accordingly.","section":"Section 3.2"},{"comment":"Figure 2b reports curves for ASR, format compliance, and over-refusal, but the exact evaluation conditions for ASR (which verifier and which dataset) are not stated in the caption; this should be specified.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is interesting and the method is simple, but the evaluation circularity is the central issue: the safety metric is the same verifier that is optimized in training. This is fixable by re-running the safety evaluation with an independent judge and reporting the resulting ASR. The AlpacaEval table must also be corrected. If these points are addressed, the paper could be suitable for publication; as it stands, the headline claims are not supported by the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: AlphaAlign is a clean, cheap RL approach to safety alignment that deserves a referee, but the central safety evaluation is weakened by using the same refusal verifier for training and evaluation, and the utility numbers are not as clean as the abstract claims.\n\nWhat is new: the specific combination of RLVR-style verifiable rewards (refusal matching) with a GRPO-normalized helpfulness reward, and the demonstration that a base model can develop safety reasoning in under 200 PPO steps with only binary labels. The AlphaAlign-Zero result on Qwen2.5-3B is genuinely interesting, and the case studies are suggestive. The paper is well situated against SCoT and circuit breakers, and the ablation showing the helpfulness reward matters for utility is useful.\n\nThe main soft spot is real: Vr (Eq. 4) is a phrase-list matcher used as the training reward, and the same Vr is the first stage of ASR evaluation (Sec 4.1). The template in Table 1 already instructs the model to emit \"Sorry, I can't comply\" for unsafe queries, so the model can earn high safety scores by copying canonical phrases. The second-stage Llama-Guard check does not fix this because a canonical refusal is exactly what Llama-Guard would classify as safe. So the ASR numbers likely overstate genuine harmfulness recognition. The CKAS analysis is a heuristic and does not break the circularity. To the authors' credit, they acknowledge the string-matching verifier in the limitations, but the main claim has not been tested independently.\n\nUtility numbers are rockier than the abstract suggests. AlpacaEval is exactly 50.0 for all three backbones, with the reported \"improvements\" relative to untuned baselines; the exact 50.0 across models is suspicious, possibly an artifact of the length-controlled evaluation. GSM8K drops 8.3 points on Llama3.2, contradicting the \"maintaining or even improving\" claim for that backbone. There are no error bars or multiple seeds, and single-run ASR numbers near zero are hard to distinguish from noise.\n\nSo the paper is a promising recipe but not a demonstrated breaking of the safety-utility trade-off. The right fixes are: independent safety evaluation (human ratings or a separate classifier applied to all responses, plus an analysis of refusal quality beyond the phrase list), reported seeds and error bars, and corrected or clearer utility metrics.\n\nFor whom: people working on practical safety alignment and RLVR. It deserves peer review because the method is simple, cheap, and potentially useful; it just needs stronger validation.","headline":"A simple RLVR-style safety alignment recipe with real promise, but the safety metric is partly circular and the utility claims are overstated.","tokens_in":21838,"tokens_out":3037,"would_cite":false,"duration_ms":31008,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AlphaAlign claims that a pure reinforcement learning setup with a verifiable safety reward and a normalized helpfulness reward can make LLMs refuse harmful queries and stop over-refusing, while keeping or improving utility, using only…","keywords":["safety alignment","reinforcement learning with verifiable rewards","refusal training","jailbreak robustness","over-refusal","safety-utility trade-off","proactive safety reasoning","large language models"],"falsifier":"Take a model trained by AlphaAlign and attack it with jailbreaks that instruct it to answer harmfully while explicitly avoiding all canonical refusal phrases, then score the outputs with a held-out judge that was never used in training; if harmful content appears despite near-zero phrase-based ASR, the safety gain is an artifact of the verifier rather than latent safety awareness.","tokens_in":20809,"feed_emoji":"🛡️","tokens_out":8558,"duration_ms":81120,"temperature":0.7,"pith_summary":"AlphaAlign tries to establish that a large language model's latent safety understanding can be turned into reliable safe behavior by pure reinforcement learning, without any supervised safety reasoning data. It does this with a verifiable safety reward that checks only whether the final answer is a refusal, plus a normalized helpfulness reward that protects quality on benign prompts. The paper reports that across four backbones this recipe improves refusal of harmful and jailbroken queries, lowers over-refusal, and preserves or improves instruction-following and mathematical reasoning within fewer than 200 RL steps. If true, safety alignment would need only binary prompt labels and a small verifier, and the usual safety-versus-utility trade-off would not be a fixed cost of alignment.","feed_headline":"Binary-label RL reduces harmful answers and over-refusal","feed_subtitle":"Paper argues pure RL with binary safety labels can replace supervised safety fine-tuning, keeping or improving utility.","key_machinery":"The load-bearing object is the dual-component reward. The verifiable safety reward $R_s$ (Eq. 4) combines a format verifier, which checks that the output has `<safety_reasoning>` and `<answer>` tags, with a refusal verifier $V_r$, which checks the final answer against a canonical phrase list; it gives positive credit for refusing labeled harmful prompts and negative credit for refusing labeled benign ones. The normalized helpfulness reward $R_h$ (Eq. 5) applies only to benign prompts, scores each answer against the batch mean, and returns positive reward only for above-average non-refusal answers while returning zero to refusals. This separates what is rewarded (the safety label match of the final answer) from how the model reasons, so the reasoning trajectory is free to become whatever the model's latent safety awareness supports.","core_discovery":"The central claim is that a structurally prompted model can discover proactive safety reasoning when PPO maximizes a reward that verifies only the final answer: positive for refusal of labeled harmful prompts, negative for refusal of labeled benign prompts, plus a GRPO-normalized helpfulness component that rewards above-average non-refusal answers to benign prompts. Nothing in the reward specifies how to reason; the model must find its own path to the correct answer format. The paper reports that the resulting policies generate explicit safety rationales, refuse harmful and jailbreak queries, stop over-refusing, and hold or improve utility, all in under 200 steps and without safety-specific SFT.","pith_inferences":["A direct extension, flagged by the paper as unexplored, is replacing the string-matching refusal verifier with a learned safety judge; the training signal would then track genuine harmfulness rather than phrase matching.","The same conditional-reward structure could be applied to other binary properties of model output, such as factual consistency or format compliance, by swapping in the corresponding verifier.","Because AlphaAlign-Zero works on a base model, the recipe might shorten conventional post-training pipelines that start from instruction-tuned models, if the result carries to larger scales.","If the verifier's coverage is the true ceiling of the method, then iterating the RL loop against a growing set of refusal paraphrases should directly raise robustness to adaptive jailbreaks."],"forward_implications":["Safety alignment can be driven by binary prompt labels and a small verifier, with no curated refusal demonstrations or safety reasoning data.","Refusal of harmful and jailbroken prompts can improve at the same time as over-refusal falls, so the safety-utility trade-off is not forced by the RL objective.","A base model without instruction tuning can acquire substantial safety behavior from the safety reward alone, as shown by AlphaAlign-Zero.","Fewer than 200 RL steps with 8 rollouts per prompt suffice for substantial safety gains, making the alignment loop short enough to iterate on.","The presence of safety-relevant keywords and reduced jailbreak-trigger tokens in early generation indicates the learned behavior is deeper than memorized refusal patterns."],"supporting_citations":[{"why":"Supplies the RLVR paradigm that AlphaAlign transfers from math and code reasoning to safety reasoning.","marker":"DeepSeek-AI et al. [2025]"},{"why":"Contributes the group-relative normalization that AlphaAlign reuses for the helpfulness reward.","marker":"Shao et al. [2024]"},{"why":"Provides the PPO update rule used to maximize the dual reward.","marker":"Schulman et al. [2017]"},{"why":"Provides Circuit Breaker as a comparison baseline and 5k of the harmful training prompts.","marker":"Zou et al. [2024]"},{"why":"Supplies the SCoT baseline and the safety-harmfulness data source, and represents the reasoning-based alignment approach AlphaAlign contrasts with.","marker":"Yang et al. [2025]"},{"why":"Establishes the shallow-alignment failure mode that motivates AlphaAlign's depth evaluation.","marker":"Qi et al. [2024a]"},{"why":"Documents failure of refusal training to generalize, motivating the need for proactive safety reasoning.","marker":"Andriushchenko and Flammarion [2024]"}],"fun_headline_variants":["Pure RL, binary labels: less harmful output, less over-refusal","Minimal RL steps with safety reward improve refusal and utility","Safety alignment via RL without SFT, keeping utility","AlphaAlign: RL with verifiable reward fosters safety reasoning","Binary safety labels + PPO yields proactive refusal reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-written refusal-phrase list used by verifier $V_r$ faithfully identifies safe refusals, because that same verifier provides the training reward and the first stage of the ASR evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Pure RL, binary labels: less harmful output, less over-refusal","Minimal RL steps with safety reward improve refusal and utility","Safety alignment via RL without SFT, keeping utility","AlphaAlign: RL with verifiable reward fosters safety reasoning","Binary safety labels + PPO yields proactive refusal reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000728,"raw_usage":{"total_tokens":3241,"prompt_tokens":909,"completion_tokens":2332,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":2250}},"tokens_in":525,"tokens_out":2332,"duration_ms":17613,"temperature":1.0,"reasoning_tokens":2250,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:43:07.562156+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a model trained by AlphaAlign and attack it with jailbreaks that instruct it to answer harmfully while explicitly avoiding all canonical refusal phrases, then score the outputs with a held-out judge that was never used in training; if harmful content appears despite near-zero phrase-based ASR, the safety gain is an artifact of the verifier rather than latent safety awareness.","supporting_citations":[{"cited_title":"Zico Kolter, Matt Fredrikson, and Dan Hendrycks","cited_arxiv_id":null,"evidence_quote":"Provides Circuit Breaker as a comparison baseline and 5k of the harmful training prompts."}],"review_version":1}