{"id":"38fb94ad-bead-4d8f-81e0-60066693929f","arxiv_id":"2505.24369","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"APL iteratively trains an attacker to generate adversarial prompt rewrites and a defender to resist them, using the defender's own preference probabilities as the attack signal.","lead":"This paper introduces Adversarial Preference Learning (APL), a method that automatically generates attack prompts to test and harden a language model's safety guardrails. The authors report large reductions in harmful outputs and attack success rates on two open models, while noting a rise in false refusals.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing assumption is that Eq. 5's intrinsic reward is a valid proxy for harmfulness; the paper's own XS-Test doubling suggests the defender instead learns broad refusal, so the central robustness/utility claim is not yet established.","rationale":"The reader's weakest assumption matches my own reading: the entire APL method is built on Eq. 5, and the only thing that makes APL different from ordinary DPO-on-rewritten-prompts is the claim that R is an intrinsic, classifier-free measure of attack effectiveness. That claim is load-bearing because the objective is optimized against R at every iteration (Algorithm 1 lines 5-8) and because the paper's headline safety gains are presented as evidence for the method's mechanism. If R is not a harmfulness metric, the training signal is just DPO on prompts chosen for high preference-probability imbalance; then Table 1's XS-Test jump (20.00 to 40.89) suggests the defender satisfies the objective by refusing broadly. This is not an ad hominem or a disagreement with consensus; it is a correctness risk in the central claim's external validity. The paper itself flags the false-refusal increase in Limitations, which strengthens rather than weakens the concern. I therefore recommend keeping the reader's CONDITIONAL verdict: the mechanism is plausible, but the decisive test of R's validity has not been run. If the AUROC test passes, the concern is resolved; if it fails, the headline claim should be downgraded. My agreement_with_reader is 'agree' because the same assumption is the weakest point.","tokens_in":17875,"tokens_out":9839,"duration_ms":141163,"concrete_test":"On a held-out balanced set of 200 HarmBench and 200 XS-Test prompts, run the final APL attacker to generate K=8 variants per prompt and compute the Eq. 5 reward R under the final defender and reference model. Compute the AUROC of R in predicting independent LLaMA-Guard-2 harmfulness labels of those generated variants. If AUROC < 0.8, R is not a valid harmfulness proxy, and the reported robustness gains cannot be attributed to the paper's intrinsic-reward mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 4.3 ('significantly enhances model safety by improving adversarial robustness while maintaining competitive utility') requires that the attack-effectiveness reward R(x'; y_pre, y_dis) in Eq. 5 measure harmfulness, not just preference-probability imbalance. R compares the defender's likelihood of the dataset's dispreferred versus preferred completion on a rewritten prompt; it contains no independent signal about whether x' elicits harmful content. An attacker maximizing R can therefore win by producing prompts that merely make the defender unsure between y_pre and y_dis, and the defender's DPO update (beta_def=0.01, Fig. 3) then suppresses such uncertainty globally. The manuscript's own XS-Test results (Table 1: Mistral false-refusal rises from 20.00% to 40.89%) are direct evidence that a broad refusal policy is being learned, and the Limitations section concedes 'increased false refusal rates.' Since HarmBench ASR is judged by a refusal-oriented classifier and the harmlessness win rate by GPT-4o comparing refusal-like outputs, the reported robustness gains may largely reflect over-refusal rather than genuine adversarial robustness. This is a correctness risk in the external validity of the mechanism, not an internal inconsistency: the equations are coherent, but the proxy's validity is unestablished by the evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Adversarial Preference Learning (APL), an iterative framework in which a generative attacker rewrites prompts and a defender is trained via DPO on the most effective adversarial prompt, with attack effectiveness measured by the defender's log-probability ratio of dispreferred to preferred responses relative to a reference model. The authors report experiments on Mistral-7B-Instruct-v0.3 and Llama-3-8B-Instruct showing reduced HarmBench attack success rates, higher harmlessness win rates, lower unsafe scores, and roughly maintained MT-Bench/LC-WinRate. The central claim (Section 4.3) is that APL 'significantly enhances model safety by improving adversarial robustness while maintaining competitive utility.'","tokens_in":18164,"tokens_out":6855,"duration_ms":83760,"significance":"If valid, APL would be an appealing self-contained alignment method: no external reward model is needed during training, the attacker and defender co-evolve automatically, and the framework is simple enough to build on DPO. The ablation study (Figure 3, Table 2) and cross-model results (Table 1) are useful. However, the significance is conditional on the validity of the intrinsic reward as a harmfulness proxy and on the stability of the small-sample evaluations; the paper's own XS-Test results raise a concrete risk that the method is partly learning broad refusal rather than targeted robustness. The paper does not include code or machine-checked proofs, so reproducibility rests on the appendix details.","major_comments":[{"comment":"The attack-effectiveness reward R(x'; y_pre, y_dis) is defined as the defender's log-probability ratio between the dispreferred and preferred completions (minus a reference term), which measures preference-probability imbalance, not harmfulness. An attacker maximizing R can succeed by producing prompts that merely increase the defender's uncertainty between y_pre and y_dis, and the defender's DPO update can lower R by refusing broadly rather than by becoming robust to the specific harmful content. Table 1 (Mistral XS-Test rising from 20.00% to 40.89%, the worst among all methods) is direct evidence of broad refusal, and since HarmBench ASR is judged by a refusal-oriented classifier while the GPT-4o win rate rewards harmless (often refusal-like) outputs, the reported robustness gains may partly reflect over-refusal. I ask for a disentangling analysis: refusal rates on HarmBench prompts, ASR computed conditional on non-refusal responses, and evidence that R tracks harmfulness (e.g., correlation with external harmfulness labels) rather than general uncertainty.","section":"§3.3, Eq. (5); Table 1"},{"comment":"The headline harmlessness win rate is computed on only 60 prompts from HH-RLHF, and the paper reports no error bars, confidence intervals, or significance tests for any comparison in Table 1. Moreover, the main hyperparameters (β_att=0.03, β_def=0.01, α=0.2) appear to be selected using the same test-set metrics that support the central claim (Figure 3 and Table 2), which risks overfitting the evaluation. Please report variance across at least a few seeds, bootstrap confidence intervals for the win rate, and a clear separation between validation and test sets for hyperparameter selection, or at a minimum disclose the selection protocol.","section":"§4.2, Table 1; Figure 3; Table 2"},{"comment":"The defender is trained with DPO on (x'_w, y_pre, y_dis), reusing preference labels from the original prompt x without re-annotation for the adversarial rewrite x'_w. This assumes that y_pre/y_dis remain valid and semantically appropriate for x'_w, which is not obvious because x'_w is generated to be 'more harmful.' If y_pre is a refusal for a harmful original prompt, this training directly reinforces refusal on adversarial prompts, providing another mechanism for the XS-Test increase. The authors should validate the label transfer, for example by sampling and manually/LLM-checking whether y_pre is still the preferred response to x'_w, or by comparing against a variant that re-annotates responses for adversarial prompts.","section":"§3.4, Algorithm 1; §4.1"}],"minor_comments":[{"comment":"The term 'direct harmfulness metric' overstates what Eq. (5) computes; consider renaming it 'attack-effectiveness metric' to avoid conflating a likelihood ratio with harmfulness.","section":"Abstract; §3.3"},{"comment":"Algorithm 1 uses α and β as update coefficients in lines 7–8, while α is the reward weight in Eq. (5) and β_att/β_def are DPO regularization coefficients; this notation clash should be fixed.","section":"Algorithm 1; Eq. (5)"},{"comment":"Section 4.1 says the attacker is initialized with an 'untrained base model,' but Algorithm 1 sets π_att ← π_ref; please clarify what 'untrained' means and how the prompt-based generation in Appendix D.1 is initialized.","section":"§4.1; Algorithm 1"},{"comment":"The XS-Test column is not discussed in the main text despite APL having the highest false-refusal rate on Mistral; please address this directly when claiming 'competitive utility.'","section":"Table 1; §4.3"},{"comment":"Equation (4) writes the inner problem as a minimization over x' ∈ A(x), but the algorithm samples K candidates and selects the argmax of R (Eq. 6); the text should clarify that Eq. (4) is a conceptual formulation rather than the implemented optimization.","section":"Eq. (4); Eq. (6)"},{"comment":"Table 2's caption says 'defined in 5' instead of 'Eq. (5)'; also the α values in the table should be referenced consistently with the main text.","section":"Table 2; §4.4"}],"recommendation":"major_revision","confidential_remarks":"This is a promising framework, but the principal risk is that the intrinsic reward induces over-refusal rather than targeted robustness; the manuscript's own XS-Test data makes this concern concrete. I would support a major revision that adds refusal-disentangled evaluations and proper uncertainty quantification. The paper's fit with the journal is acceptable if these issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on APL. The core idea—using the defender's own preference probabilities as the attack reward, then co-training attacker and defender with DPO—is clean and genuinely useful. It removes the need for a separate reward model or classifier, and the ablation against APL(RM) suggests the intrinsic signal carries real information. The paper is honest: the limitations section calls out increased false refusals and sensitivity to beta and alpha. I'd give credit for that.\n\nThe problems are in the evaluation, not the method. The headline 83.33% win rate comes from 60 prompts with no error bars or significance tests. The authors picked the best hyperparameters (beta_att=0.03, beta_def=0.01, alpha=0.2) directly on the test metric, so the result is optimistic. More importantly, XS-Test false refusal on Mistral doubles from 20% to 40.89%. That's not a side effect; it's the main event. Eq. 5's reward measures preference-probability imbalance, and the attacker can in principle win by making the model unsure, not necessarily by eliciting harmful content. The defender then suppresses that uncertainty, which is exactly the broad refusal behavior you'd expect. Since HarmBench ASR and LLaMA-Guard are refusal-oriented judges, part of the \"safety\" gain is likely just refusals.\n\nNone of this is an internal contradiction—the equations are coherent and the pipeline is reproducible in principle. But the central claim, \"significantly enhances model safety while maintaining competitive utility,\" is not established. On utility, MT-Bench of 6.59 vs 6.78 is fine, but the XS-Test increase is a direct utility cost the paper underweights. I'd also like to see comparisons against MART or SEAS, which it cites but doesn't benchmark.\n\nWho should read it: people working on red-teaming and alignment with limited annotation budgets. It deserves a serious referee—the idea is worth engaging with, and the negative result on refusal is useful. For a top venue, though, the authors need held-out hyperparameter selection, error bars, and a decomposition of safety gains into genuine robustness versus refusal. As a desk decision, I'd send it to review and let referees push on that. The current version wouldn't be my accept, but it's a solid conditional.","headline":"APL offers a clever self-contained adversarial alignment idea, but the refusal-flavored safety gains and test-set tuning make the headline claim premature.","tokens_in":18758,"tokens_out":4110,"would_cite":true,"duration_ms":54879,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"Adversarial preference learning makes LLMs far more resistant to jailbreaks in the paper's tests, with a small utility cost.","keywords":["adversarial training","preference learning","LLM safety","jailbreak robustness","direct preference optimization","red teaming","iterative training"],"falsifier":"Run the APL training loop on Mistral-7B and then evaluate the trained defender on a suite of benign prompts that are phrased in unusual or adversarial-sounding ways; if the false-refusal rate rises sharply while the attack-success rate falls, that would show the reward signal is being gamed by over-refusal rather than by improved discrimination. A cleaner test would be to pit the trained defender against a held-out, never-seen attack family and check whether the reduction in attack success rate persists out of distribution.","tokens_in":17689,"feed_emoji":"🛡️","tokens_out":2477,"duration_ms":23761,"temperature":0.7,"pith_summary":"The paper tries to show that language models can be made substantially more resistant to adversarial jailbreak attacks without relying on external classifiers, expensive human annotation, or fixed attack templates. The proposed method, Adversarial Preference Learning, turns the alignment problem into an iterative game: a generative attacker rewrites prompts to exploit the defender's weaknesses, and the defender is trained on the most effective attacks. On Mistral-7B, harmful outputs drop from 5.88% to 0.43% and attack success rates fall by up to 65%, while an MT-Bench score of 6.59 stays close to the baseline's 6.78.","feed_headline":"Adversarial preference learning cuts LLM jailbreak success by up to 65 percent","feed_subtitle":"An iterative attacker-defender loop using the model's own preferences keeps utility nearly flat while sharply reducing harmful outputs.","key_machinery":"The core mechanism is an iterative minimax training loop between a generative attacker and a defender, both trained with Direct Preference Optimization. The attack-effectiveness reward is the defender's own log preference ratio between the dispreferred and preferred response, minus a reference-model baseline; this intrinsic signal replaces external classifiers. The attacker and defender co-evolve over 30 iterations, with the most and least effective adversarial prompts selected each round to train attacker and defender respectively.","core_discovery":"The paper's central claim is that an iterative adversarial training loop using the model's own preference probabilities as the attack-effectiveness signal can significantly improve safety against diverse attacks while keeping utility competitive. Concretely, the defender's harmlessness win rate over the base model reaches 83.33%, harmful outputs drop from 5.88% to 0.43%, and attack success rates on HarmBench fall by 21% to 65% across Direct Request, Zero-Shot, Few-Shot, and GCG attacks on Mistral-7B-Instruct-v0.3. The method also improves the already-aligned Meta-Llama-3-8B-Instruct.","pith_inferences":["The reported robustness gains may partly reflect increased over-refusal rather than genuine harmlessness; the paper's own XS-Test false-refusal rate rises to 40.89%, so an external judge could drive the internal reward up by simply refusing more broadly. ","A natural testable extension is to replace the intrinsic preference ratio with a calibrated harmlessness classifier and compare whether the iterative game still helps; that would reveal how much of the gain is due to adversarial co-evolution versus the reward signal. ","The framework could in principle be transferred to other preference-based alignment objectives such as SimPO or IPO, since only the reward signal in Eq. 5 and the DPO update need swapping."],"forward_implications":["If the gains hold, adversarial robustness could be improved without training a separate reward model or safety classifier, reducing annotation cost. ","An iterative attacker that adapts to the defender's current weaknesses may cover more of the adversarial input space than static attack libraries. ","The method's usefulness on an already-aligned model suggests it could serve as a post-alignment hardening step before deployment. ","The framework's closed-loop design points toward fully automated red-teaming and safety-tuning pipelines, provided the intrinsic reward signal remains trustworthy."],"supporting_citations":[{"why":"Direct Preference Optimization is the core learning objective for both attacker and defender.","marker":"Rafailov et al., 2024"},{"why":"HH-RLHF harmless dataset supplies the preference pairs used in all experiments.","marker":"Bai et al., 2022a"},{"why":"HarmBench supplies the four attack strategies and the classifier that measures attack success rates.","marker":"Mazeika et al., 2024"},{"why":"Provides the zero-shot and few-shot attack baselines used in evaluation and the red-teaming method that generative attackers build on.","marker":"Perez et al., 2022"},{"why":"GCG token-optimization attack is one of the four evaluation attacks and a representative gradient-based threat model.","marker":"Zou et al., 2023"},{"why":"MT-Bench is one of the primary utility metrics used to check that safety gains do not destroy general capability.","marker":"Zheng et al., 2023"},{"why":"XS-Test measures exaggerated safety behavior and provides the false-refusal numbers that show the safety-utility trade-off.","marker":"Röttger et al., 2024"}],"fun_headline_variants":["Self-adversarial training slashes jailbreak success by up to 65%","APL: iterative self-attacks cut harmful outputs by over 90%","Using its own preferences, APL cuts jailbreaks by up to 65%","New training loop reduces harmful LLM outputs from 5.88% to 0.43%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the defender's own preference probability is a trustworthy measure of how harmful an attack is, so that optimizing against it improves genuine safety rather than merely teaching the model to refuse anything unusual.","fun_headline_variants_meta":{"raw":{"variants":["Self-adversarial training slashes jailbreak success by up to 65%","APL: iterative self-attacks cut harmful outputs by over 90%","Using its own preferences, APL cuts jailbreaks by up to 65%","New training loop reduces harmful LLM outputs from 5.88% to 0.43%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001361,"raw_usage":{"total_tokens":5515,"prompt_tokens":935,"completion_tokens":4580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":4490}},"tokens_in":551,"tokens_out":4580,"duration_ms":35289,"temperature":1.0,"reasoning_tokens":4490,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:24:58.629995+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the APL training loop on Mistral-7B and then evaluate the trained defender on a suite of benign prompts that are phrased in unusual or adversarial-sounding ways; if the false-refusal rate rises sharply while the attack-success rate falls, that would show the reward signal is being gamed by over-refusal rather than by improved discrimination. A cleaner test would be to pit the trained defender against a held-out, never-seen attack family and check whether the reduction in attack success rate persists out of distribution.","supporting_citations":[],"review_version":1}