{"id":"7862c992-a862-4cf0-ae26-926c7f780733","arxiv_id":"2412.15623","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"JailPO uses preference optimization to train attack models that generate covert jailbreak questions and templates, achieving high attack success on aligned LLMs with far fewer queries.","lead":"This paper introduces JailPO, a method that trains language models to automatically craft jailbreak prompts that bypass the safety filters of other chatbots. If it works as reported, it offers a low-cost way to probe and expose alignment weaknesses in commercial AI systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reward and primary metric for JailPO are the same ClassJudge detector, so reported ASR may reflect classifier gaming rather than genuine harmful responses.","rationale":"The reader correctly identified the circular use of ClassJudge as both training reward and evaluation metric. I agree that this is the most load-bearing concern: all quantitative claims about attack effectiveness, universality, and robustness depend on the ASR/QSR numbers, which are computed with automated classifiers. The paper's inclusion of Rob-lg as a second evaluator and Llama Guard for high-risk analysis provides partial independent evidence, but neither is a human judgment and both are automated classifiers with unknown alignment to human notions of harm. Because the training signal is explicitly ClassJudge, the method could overfit to that detector; without a human-evaluation spot-check, the central claim remains unverified. The paper is otherwise coherent, with clear methodology and useful patterns, so a conditional verdict requiring human validation is appropriate. My read does not change the reader's verdict; I would keep CONDITIONAL/UNCHANGED pending that validation.","tokens_in":14297,"tokens_out":5067,"duration_ms":43642,"concrete_test":"Sample 100 jailbreak attempts per target model from JailPO that ClassJudge labels as successful (S=1). Have two independent human annotators, blinded to the classifier output, judge whether the response actually provides the requested harmful information or is a refusal/evasive non-answer. Compute Cohen's κ and the false-positive rate of ClassJudge relative to human labels. Also compare with Rob-lg and Llama Guard scores on the same sample. If the false-positive rate exceeds 10% or κ < 0.8, the reported ASR is not trustworthy and the central claim of effective automatic jailbreaks is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that ClassJudge (Section 'Problem Formulation') is a faithful measure of whether a target LLM response contains harmful content. ClassJudge is used twice: (i) as the reward signal in Eq. (1) to construct preference pairs for SimPO, and (ii) as the primary ASR metric in Table 1. This circularity means the attack models are directly optimized to maximize the exact score used to claim success. If ClassJudge has blind spots—for example, if it marks certain refusal phrasings as 'correct answers' or is sensitive to superficial prompt features—the reported ASR numbers (e.g., 55.6% on Mistral, 15.2% on GPT-3.5) would overstate the actual jailbreak capability. The paper does include a second evaluator (Rob-lg) and reports those numbers, which partially mitigates the concern. However, Rob-lg is also a RoBERTa-based classifier and may share the same learned biases; neither is validated against human judgments in this paper. The absence of error bars despite 'repeat each experiment five times' further weakens the quantitative claims. A human-evaluation spot-check is therefore essential to establish that JailPO's outputs are genuinely harmful, not just classifier-positive.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces JailPO, a black-box jailbreak framework that trains two attack models—QEM and TEM—using supervised fine-tuning followed by Simple Preference Optimization (SimPO). Preference pairs are constructed by scoring responses of a fixed aligned LLM (Llama2-7B) with a RoBERTa-based binary detector, ClassJudge. The resulting attack models generate covert questions (QEPrompt), templates (TemplatePrompt), and a combined pattern (MixAsking). The method is evaluated on four aligned LLMs (Llama2, Mistral, Vicuna, GPT-3.5) plus three larger models, using ClassJudge and a second RoBERTa-large detector (Rob-lg) as automatic evaluators. The paper claims superior attack success rate, question success rate, efficiency, universality, and robustness against perplexity and LLM-Guard defenses, and includes ablations, query-iteration analyses, and tests on advanced models.","tokens_in":14542,"tokens_out":4961,"duration_ms":42457,"significance":"If the reported results are trustworthy, JailPO constitutes a meaningful advance in automated red-teaming: it generates jailbreak prompts with very few queries, transfers across target models, and is far more query-efficient than token-level optimization baselines such as GCG. The experimental coverage is extensive, including ablations, defense evaluations, and tests on current strong models (GPT-4, Llama3-70B, Command-R). However, the main effectiveness metric is the same detector used to score the preference data, so the validity of the headline attack-success numbers rests on an independent demonstration that JailPO's prompts elicit genuinely harmful content rather than classifier-positive outputs. The secondary RoBERTa evaluator and the Llama Guard high-risk analysis provide partial but incomplete mitigation, since they are also automatic classifiers and are not validated against human judgment in this paper.","major_comments":[{"comment":"The circularity of using ClassJudge as both the reward signal for preference optimization and the primary attack-success metric is a load-bearing concern. Eq. (1) constructs preference pairs by scoring target-model responses with ClassJudge, and Table 1 reports ASR with the same detector. Consequently, the reported success rates may reflect JailPO's ability to exploit classifier blind spots rather than to elicit genuinely harmful content. The second evaluator, Rob-lg, is also a RoBERTa-based classifier and may share the same learned biases; no human evaluation is provided. The Llama Guard high-risk analysis (Figure 4) is reported separately and is not integrated into the main success-rate claim. Please add a human-annotation spot-check or an independent, non-classifier-based benchmark to validate the ASR numbers.","section":"Problem Formulation, Eq. (1), Table 1"},{"comment":"The query count for MixAsking appears inconsistent. Table 1 lists QN=520 for all JailPO patterns, but Algorithm 1 shows that MixAsking can issue up to two queries per question: first QEPrompt, and then TemplatePrompt if PatternJudge flags non-compliance. Thus the actual number of LLM calls for MixAsking is at most 1040, not 520. Please clarify whether QN counts questions or LLM queries; if it counts questions, the efficiency comparison against baselines should use the worst-case query count. This affects the claimed 'two orders of magnitude fewer queries' relative to TemplateJailbreak.","section":"Table 1 and Algorithm 1 (Appendix A.5)"},{"comment":"The paper states that each experiment was repeated five times, yet no standard deviation, confidence interval, or significance test is reported anywhere. Without variance information, the observed differences in ASR/QSR—such as the 3.26% vs. 6.21% difference between QEPrompt and TemplatePrompt on Llama2 in Table 1—cannot be judged for statistical significance. Since many of the headline comparisons involve small percentage-point gaps, please report means with error bars or a significance analysis, at least for the main results in Tables 1 and 2 and Figures 4 and 5.","section":"Experiments Settings (Settings paragraph)"}],"minor_comments":[{"comment":"The reference to SimPO is incorrectly attributed to (Wei, Haghtalab, and Steinhardt 2023); the correct citation is Meng, Xia, and Chen (2024).","section":"Introduction"},{"comment":"'Defense Rassing Rate' appears to be a typo; the metric should likely be 'Defense Passing Rate' or 'Defense Bypass Rate.'","section":"Experiments Settings (Metrics)"},{"comment":"The variable l in Eq. (1) is not defined in the main text. Please state explicitly that l is the number of queries to the target LLM used to evaluate each prompt.","section":"Eq. (1)"},{"comment":"The PatternJudge used in MixAsking is only briefly described as a refusal-pattern detector from Zou et al. (2023). Please provide the exact set of refusal patterns or a precise description so the pattern is reproducible.","section":"Prompts setup"},{"comment":"The construction of the 6000 preference pairs for QEM is unexplained. With 522 questions and 10 responses each, the number of possible preference pairs is much larger; please specify how the pairs were selected or sampled.","section":"Enhanced Model Construction"},{"comment":"The preliminary experiments use 50 questions from Deng et al. (2023), while the main training set uses 522 questions from multiple sources. Please clarify whether the test questions (AdvBench) are guaranteed disjoint from the training questions, since the paper states they are distinct but does not show the overlap analysis.","section":"Preliminary Experiments"}],"recommendation":"major_revision","confidential_remarks":"This paper has a well-designed framework and a broad experimental study, but the central reported metric is the same detector used to generate the training signal. The lack of any human evaluation or independent harmfulness benchmark, combined with absent error bars and a query-count inconsistency for MixAsking, makes the headline claims difficult to verify in their current form. I believe these issues are fixable: a human spot-check on a sample of successful attacks, a variance analysis, and a correction/clarification of the MixAsking query count would substantially strengthen the paper. I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"JailPO is a worthwhile paper with a real circularity problem at its core. The authors apply preference optimization to jailbreak prompt generation, training two Llama-2-based attackers—one that rephrases questions covertly, one that generates complex templates—and combine them across three attack patterns. That combination is genuinely new, and the paper is well-structured and thorough.\n\nThe strongest part is the ablation and the multi-evaluator setup. Preference optimization adds a measurable boost over fine-tuning alone, the method transfers across open and closed models, and the high-risk analysis with Llama Guard provides some independent grounding beyond the main detector. The efficiency story—520 queries for 520 questions with competitive ASR—is compelling.\n\nThe soft spots are real. ClassJudge is both the reward signal for the preference pairs and the headline ASR metric. That is circular, and it means the reported numbers likely overstate true jailbreak capability. Rob-lg and Llama Guard help, but Rob-lg is also a RoBERTa-based classifier and neither is validated against human judgments here, so the actual attack strength remains uncertain. There is also a concrete inconsistency: MixAsking is described as a two-stage fallback, yet Table 1 lists a query count of 520 for 520 questions, which only makes sense if the fallback never happens or the count is misreported. The paper also omits error bars despite claiming five repetitions, and it skips stronger black-box baselines like PAIR and GPTFuzzer. No code or data is released.\n\nOn balance, the central idea holds up—the method probably works—but the headline numbers need independent verification. A human-evaluation spot-check on a sample of successful attacks would settle the circularity question. This deserves a serious referee, but with the expectation of major revision.\n\nRecommendation: send to peer review, not desk reject.","headline":"New preference-optimized jailbreak attackers are promising, but the circular reward/metric choice makes the headline ASR untrustworthy.","tokens_in":15063,"tokens_out":3624,"would_cite":true,"duration_ms":27043,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A preference-optimized model can automatically generate jailbreak prompts that work across multiple aligned black-box LLMs with very few queries, replacing handcrafted templates and white-box optimization.","keywords":["jailbreak attack","preference optimization","black-box attack","LLM alignment","SimPO","universal adversarial prompts","LLM safety evaluation"],"falsifier":"Take the set of prompts that JailPO generated and were scored as successful by ClassJudge, present the target LLM's responses to independent human annotators (or a separately trained harmfulness classifier), and measure the agreement rate; if agreement is low, or if the prompts' 'success' depends on phrasing that exploits the classifier, the central claim of genuine jailbreak effectiveness is not supported.","tokens_in":14102,"feed_emoji":"⚔️","tokens_out":5027,"duration_ms":40756,"temperature":0.7,"pith_summary":"The paper proposes JailPO, a framework that trains small language models to produce jailbreak prompts automatically. It claims that preference optimization—the same family of techniques used to align LLMs—can teach an attack model to rewrite harmful questions into covert forms and to build complex templates that make aligned black-box LLMs answer instead of refuse. The central promise is scale and portability: one training run yields a generator that works across Llama2, Mistral, Vicuna, and GPT-3.5 (and later Llama3-70B, GPT-4, Command-R) with as few as one query per question, replacing handcrafted templates and costly white-box token optimization. If true, this turns LLM safety evaluation into a cheap, automated probe, while also lowering the bar for real attackers.","feed_headline":"Preference training yields automatic universal jailbreaks","feed_subtitle":"A two-stage fine-tuned Llama2 generator probes seven LLMs with a few queries per question.","key_machinery":"The load-bearing mechanism is a two-stage training pipeline: supervised imitation of GPT-3.5 rewrites, followed by preference optimization with SimPO. The preference signal comes from ClassJudge, a binary classifier that scores whether a target model's response to a candidate prompt is an affirmative answer; prompts with higher scores are treated as preferred completions. Because the same classifier is also the primary evaluation metric, the whole method's reported effectiveness rests on that detector's judgments.","core_discovery":"The paper's central claim is that jailbreak prompting can be learned end-to-end. Starting from a small aligned model (Llama2-7B), it first supervises fine-tuning on GPT-3.5-rewritten versions of 522 harmful questions and 78 existing templates, then applies Simple Preference Optimization (SimPO) on preference pairs where the reward is whether a target LLM's response is judged a positive answer by the RoBERTa-based ClassJudge detector. The resulting Question-Enhanced and Template-Enhanced models generate covert prompts that, according to the paper's measurements, outperform handcrafted and token-optimization baselines in attack success rate, question success rate, and robustness against perplexity and toxicity defenses.","pith_inferences":["Because ClassJudge is both the training reward and the reported metric, the method's measured gains might reflect optimization against this specific detector; testing with independent human annotations or a different harmfulness classifier would clarify whether the attacks elicit genuinely harmful content.","The transferability of a Llama2-trained generator to GPT-4 and Command-R suggests that jailbreak prompt styles generalize across model families, which implies that defenses will need to target prompt transformations rather than specific templates.","A natural extension is to use the same two-stage recipe with a stronger (or ensemble) reward model, or to adapt the generator to a specific target with a few preference queries, turning jailbreak generation into an on-demand service."],"forward_implications":["A single preference-optimized generator can probe multiple aligned LLMs automatically, removing the need for handcrafted templates or per-target white-box optimization.","Attack success improves with query iterations: MixAsking, which falls back to a template prompt when the target refuses, reaches the highest question success rate at acceptable query cost.","Different prompt families have different failure modes: complex templates achieve higher attack success, while covert question rewrites elicit riskier content and evade perplexity and toxicity defenses more often.","Safety alignment itself may be leveraged against the model: the same preference-optimization machinery used to align LLMs can be repurposed to break that alignment."],"supporting_citations":[{"why":"Supplies ClassJudge, the RoBERTa-based detector that scores preference pairs and serves as the primary evaluation metric.","marker":"(Xu et al. 2024)"},{"why":"Provides SimPO, the reference-free preference optimization objective used to fine-tune the attack models.","marker":"(Meng, Xia, and Chen 2024)"},{"why":"Contributes the GCG baseline, the self-instruction data augmentation recipe, and the AdvBench dataset used for evaluation.","marker":"(Zou et al. 2023)"},{"why":"Source of the handcrafted jailbreak templates used as seed data for TEM and as the TemplateJailbreak baseline.","marker":"(Liu et al. 2023)"},{"why":"Provides Llama Guard, used to measure high-risk content in responses when analyzing the three jailbreak patterns.","marker":"(Inan et al. 2023)"},{"why":"Inspires the intricate semantic transformation approach and is one of the datasets used to construct the question set.","marker":"(Deng et al. 2024)"}],"fun_headline_variants":["Preference-trained attackers automate covert jailbreaks","Learned jailbreak prompts outperform handcrafted templates","Automated jailbreaks via preference optimization","Black-box attack model yields universal jailbreak prompts","Trained generators produce robust jailbreak attacks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that the RoBERTa-based ClassJudge detector reliably identifies whether a response is a harmful answer; since ClassJudge is both the reward signal for preference optimization and the primary evaluation metric, a poorly calibrated or gameable detector would inflate both the training signal and the reported attack success rates.","fun_headline_variants_meta":{"raw":{"variants":["Preference-trained attackers automate covert jailbreaks","Learned jailbreak prompts outperform handcrafted templates","Automated jailbreaks via preference optimization","Black-box attack model yields universal jailbreak prompts","Trained generators produce robust jailbreak attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000605,"raw_usage":{"total_tokens":2796,"prompt_tokens":895,"completion_tokens":1901,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":1833}},"tokens_in":511,"tokens_out":1901,"duration_ms":13744,"temperature":1.0,"reasoning_tokens":1833,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:14:27.585088+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the set of prompts that JailPO generated and were scored as successful by ClassJudge, present the target LLM's responses to independent human annotators (or a separately trained harmfulness classifier), and measure the agreement rate; if agreement is low, or if the prompts' 'success' depends on phrasing that exploits the classifier, the central claim of genuine jailbreak effectiveness is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Inspires the intricate semantic transformation approach and is one of the datasets used to construct the question set."}],"review_version":1}