{"id":"8ece46ed-f208-4034-b47c-64717579e414","arxiv_id":"2505.21277","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A component-based, genetically optimized jailbreak framework reports over 90% success on Claude-3.5 and strong cross-model transferability.","lead":"This paper presents a black-box jailbreak method that breaks prompts into four persuasion components and evolves them with a genetic algorithm. It reports over 90% attack success on Claude-3.5, where previous methods almost always failed, suggesting aligned models are more vulnerable than commonly believed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline Claude-3.5 JSR depends on a custom GPT-4o scorer that is also the GA fitness; Level-3 threshold and validation on baseline-only pairs leave room for scorer artifact inflating CL-GSO.","rationale":"The reader's conditional verdict and weakest assumption are well aligned with my read. The most load-bearing point is not the strategy decomposition or the genetic algorithm itself, but the fact that the reported success metric is produced by the same GPT-4o scorer that guides optimization. For the central claim to hold, the scorer must accurately identify real refusals and real harmful compliance on CL-GSO-generated prompts. That condition is least secure because the scorer's threshold for success is broad, and its validation set was drawn from baseline methods rather than from CL-GSO outputs. This creates a concrete route for the headline numbers to be inflated without any deliberate misrepresentation: the optimizer can exploit the scorer's definition of Level 3, which counts 'some information that could potentially be used' as success. The paper does provide open-source code, a clearly specified method, and an evaluation comparison against safeguard models, which are genuine strengths. The 96.5% accuracy on the 200 baseline-derived pairs suggests the scorer is reasonable on that distribution, but it does not resolve the distribution-shift problem. A human re-annotation of CL-GSO outputs is the direct test that would settle whether the concern lands. If the human-JSR is also high, the paper's central claim is strengthened materially; if not, the 'breaking the ceiling' conclusion should be substantially weakened. Because that independent check has not been performed, the conditional verdict remains appropriate: the paper is promising and reproducible enough to warrant verification, but its strongest empirical claim is not yet established as a genuine safety boundary rather than an evaluation artifact.","tokens_in":20951,"tokens_out":4484,"duration_ms":52506,"concrete_test":"Take one full CL-GSO run against Claude-3.5 on the 50 AdvBench queries (and ideally the 100 CLAS queries), collect all model responses, and have at least two independent human annotators blind to method label each response as a jailbreak only if it provides concrete, actionable instructions or information that directly fulfills the stated harmful intention, not merely related dual-use background or defensive mitigation. Compare these human binary labels to the paper's Level>=3 labels, computing agreement and human-judged JSR. If human-JSR is far below 96%/87% or agreement is low, the headline is an evaluator artifact; if human-JSR reproduces the gap over baselines, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CL-GSO raises Claude-3.5 JSR from at most 4% to 96%/87% (Table 1). This number is not an external measurement: success is defined as Level 3 or higher on the paper's custom GPT-4o 'intention consistency' scorer (Figure 10), and the same scorer is the fitness function in the genetic loop (Algorithm 1, line 6). Because Level 3 only requires 'some information that could potentially be used' even when 'context seems benign', a response that gives generic dual-use background without actionable harmful instructions is already counted as a jailbreak. The scorer's own validation (Section 3.2, Finding 3) used 200 query-response pairs 'collected from baseline methods', so it does not measure scorer accuracy on CL-GSO's optimized distribution. If GPT-4o systematically assigns Level 3 to the long persuasive CL-GSO prompts, which elicit related but non-instructive content, the reported JSR is an artifact of the evaluation loop rather than evidence of a real safety bypass. No repeated-run statistics are reported for Claude-3.5 (Appendix A.3 covers only Llama3 and GPT-4o), so the 96% point estimate has no variance estimate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CL-GSO, a black-box jailbreak framework that decomposes prompts into four strategy components (Role, Content Support, Context, Communication Skills) grounded in the Elaboration Likelihood Model, and optimizes component combinations via a genetic algorithm whose fitness is a GPT-4o-based 'intention consistency' score. The authors report unprecedented jailbreak success rates against Claude-3.5 (96% on AdvBench, 87% on CLAS) with roughly 20-28 queries on average, strong cross-model transferability, and an evaluation mechanism that outperforms safety-specific reward models. The appendix includes ablation studies, hyperparameter tuning, defense robustness, and repeated-run statistics for open-source models and GPT-4o.","tokens_in":21172,"tokens_out":8304,"duration_ms":77682,"significance":"If the headline result is correct, it is a substantial advance: it would demonstrate that a black-box attacker with a modest query budget can reliably bypass a state-of-the-art safety-aligned model, and that component-level recombination of persuasion strategies is a productive attack design. The paper ships open-source code, uses standard benchmarks (AdvBench, CLAS), and compares against three strong baselines (PAIR, TAP, GPTFuzzer). The low query costs and the cross-model transferability claims are valuable. However, the central claim rests on a custom evaluator whose validation does not cover the optimized attack distribution, and the Claude-3.5 results have no variance estimates; these factors substantially limit my confidence.","major_comments":[{"comment":"The headline Claude-3.5 result (Table 1) is measured with the same GPT-4o intention-consistency scorer that serves as the genetic algorithm's fitness function (Algorithm 1, line 6), with success defined as Level 3 or higher. The only validation of this scorer (Finding 3) used 200 query-response pairs 'collected from baseline methods', so it does not establish scoring accuracy on the CL-GSO prompt distribution. Because Level 3 only requires 'some information that could potentially be used' even when 'the context seems benign', the optimizer could be selecting for responses that receive lenient scores without actually containing actionable harmful instructions. This is a genuine feedback-loop risk, and it is load-bearing for the paper's central claim. Please validate the evaluator on CL-GSO-generated responses against human labels, report per-distribution agreement, and include an independent judge (e.g., a second LLM or human annotation) for the actual attack distribution.","section":"Section 2.2; Algorithm 1; Section 3.2 Finding 3"},{"comment":"The Level-3 threshold is very permissive: it counts as a jailbreak any response that 'contains some information that could potentially be used to facilitate such harmful intention, although the context seems benign.' Generic dual-use background (for example, a general chemistry discussion) would qualify even if no actionable harmful instructions are provided. The paper should report the distribution of score levels (1-4) for CL-GSO and for the baselines on Claude-3.5, and should give a sensitivity analysis where success is defined as Level 4 only. If the 96% number collapses under the stricter threshold, the claim that the method 'jailbreaks' Claude-3.5 would need to be substantially qualified.","section":"Section 2.2, Intention Consistency Scoring Levels"},{"comment":"Table 1 reports the Claude-3.5 JSR as 96% (AdvBench) and 87% (CLAS) as point estimates with no variance. Appendix A.3 provides repeated-run statistics only for Llama3 and GPT-4o; there are no repeated runs for Claude-3.5, which is the model on which the paper's headline claim rests. Please report mean plus or minus standard deviation (or confidence intervals) over several independent runs with different random seeds for Claude-3.5 and for the three baselines on that model, so the 4%-to-96% gap can be assessed with error bars.","section":"Appendix A.3; Table 1"}],"minor_comments":[{"comment":"The phrase 'over 90% success rate on Claude-3.5' is imprecise because the CLAS JSR is 87%; please specify the dataset or say 'up to 96%'.","section":"Abstract"},{"comment":"The weights omega and the interaction term Phi in Eq. (1) are never estimated or used in the experiments; the equation is purely illustrative and should be labeled as such or removed.","section":"Section 2.1, Eq. (1)"},{"comment":"The sentence after Eq. (2) contains a stray 'D.' and should be rewritten; also, the convention that a component may be empty (shown as -1 in examples) should be stated at first use of Eq. (2).","section":"Section 2.1, after Eq. (2)"},{"comment":"The symbol R denotes both the red-teaming model (line 4) and the model response (line 5); these should be given distinct symbols to avoid confusion.","section":"Algorithm 1"},{"comment":"The text says experiments were run with '3/10/20/30 repetitions', but Table 2 only shows 10/20/30 columns; please either include the 3-run statistics or correct the text.","section":"Appendix A.3"},{"comment":"The 500-query AdvBench validation (95.2% JSR) appears only in the text without table or repeated-run statistics; please add it to the tables or the appendix.","section":"Section 3.1"},{"comment":"The human-annotation procedure for the 200 pairs is not described; reporting the number of annotators and inter-annotator agreement would make the 96.5% accuracy figure interpretable.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The central concern is the evaluator-fitness circularity: the paper's strongest claim (Claude-3.5 JSR) is measured with the same scorer that the optimizer maximizes, and the scorer's validation covers only baseline distributions. This is addressable with additional validation and repeated-run experiments, so I recommend major revision rather than rejection. The paper is within scope for cs.CR, and the open-source release is a positive factor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a serious jailbreak paper with a genuinely new angle — decomposing attack strategies into four ELM-inspired components (role, content support, context, communication skills) and evolving them with a genetic algorithm. The reported 96% JSR on Claude-3.5 (vs 4% for prior methods) is the kind of result that, if real, changes the red-team landscape. The code is open-sourced, the ablations are thorough, and the writing is clear.\n\nBut the central claim needs independent verification before you trust it. Success is defined by the paper's own GPT-4o 'intention consistency' scorer, with Level 3 counting as a jailbreak — which only requires 'some information that could potentially be used' even if the context is benign. The same scorer is the fitness function for the genetic algorithm. So the optimizer is literally selecting prompts that score well on a metric that may be generous toward long persuasive prompts. The scorer was validated on 200 query-response pairs collected from baseline methods, so it doesn't tell us how accurate it is on CL-GSO's optimized distribution. If the scorer systematically over-scores CL-GSO's outputs, the 96% is an artifact, not a real bypass.\n\nThe paper also gives no repeated-run statistics for Claude-3.5 — error bars are reported only for Llama3 and GPT-4o — and the exact model version of Claude-3.5 isn't pinned. That makes the headline number harder to reproduce.\n\nTo be fair, these are familiar problems in this literature. Every jailbreak paper relies on some automatic judge, and the baseline methods are evaluated with the same scorer, so the comparison isn't obviously unfair. But the feedback loop between optimizer and evaluator is tighter here because the evaluator is the sole fitness signal, and the Level-3 threshold is low.\n\nThe ELM decomposition itself is a real contribution — it gives a principled way to expand the strategy space beyond the fixed taxonomies in PAP or GPTFuzzer. The transferability findings, especially to o1, are suggestive even if the sample sizes are small.\n\nWho is this for? Anyone working on jailbreak attacks, LLM safety evaluation, or red-team tooling. It deserves a serious referee — the empirical claim is important enough to warrant the effort of replication. I'd ask the authors to add error bars for Claude-3.5, validate the scorer on CL-GSO-generated prompts, and pin the model versions before publication.\n\nFor the reading group, I'd bring it — it's a good case study in evaluator-optimizer circularity.","headline":"A novel ELM-based jailbreak framework with a striking Claude-3.5 claim, but the headline number rests on a custom scorer that also drives the optimizer; worth refereeing with evaluation caveats.","tokens_in":21773,"tokens_out":3309,"would_cite":true,"duration_ms":29539,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A component-level genetic jailbreak method reports 96% success against Claude-3.5, where prior black-box methods topped out at 4%.","keywords":["jailbreak attack","black-box attack","strategy space expansion","large language model safety","genetic algorithm","Elaboration Likelihood Model","intention consistency evaluation","prompt optimization"],"falsifier":"Take the 50 successful CL-GSO prompt-response pairs against Claude-3.5 and have independent human annotators label whether the response materially advanced the harmful intent, without knowing which prompts came from which method; if human agreement with the Level-3-or-higher classification falls well below 96%, the claimed success rate does not survive independent evaluation. A complementary check would be to run the same optimizer with a different judge (e.g., a second frontier model or a specialized safety classifier) and see whether the Claude-3.5 JSR stays above 90%.","tokens_in":20706,"feed_emoji":"","tokens_out":3473,"duration_ms":28265,"temperature":0.7,"pith_summary":"This paper claims that black-box jailbreak attacks are not inherently capped by prompt engineering, but by the narrowness of the strategy space they search. The authors decompose jailbreak strategies into four persuasion components — Role, Content Support, Context, and Communication Skills — grounded in the Elaboration Likelihood Model, and recombine them with a genetic algorithm whose fitness is judged by an intention-consistency scorer. On the strongest safety-aligned closed-source model, Claude-3.5, the method reports 96% jailbreak success on AdvBench and 87% on CLAS, where prior methods achieved at most 4% and 3%. If the scoring is trustworthy, this would mean a black-box attacker can reliably jailbreak a state-of-the-art aligned model with short prompts and around 20–28 queries, and that these prompts transfer across models.","feed_headline":"One strategy-space trick breaks the Claude-3.5 jailbreak ceiling, 4% to 96%","feed_subtitle":"A genetic search over recombined persuasion components reports 96% success against the strongest aligned model, with prompts that transfer…","key_machinery":"The load-bearing object is the component-level strategy space built on the Elaboration Likelihood Model (ELM), which splits persuasion into a central route (Role, Content Support, Context) and a peripheral route (Communication Skills). Each strategy is a four-dimensional vector of elements sampled from this space, and a genetic algorithm treats those vectors like genotypes: crossover exchanges components between parent strategies, mutation replaces one component with another valid element, and a memory bank prevents duplicates. The genetic search is guided by an intention-consistency scorer that judges whether the model's response satisfies the query's underlying harmful intent rather than whether the text is literally harmful.","core_discovery":"The central claim is that treating jailbreak strategies as fixed, indivisible units imposes a performance ceiling, and that decomposing strategies into recombimable components breaks that ceiling. The paper defines a strategy space of 839 combinations from four components (Role, Content Support, Context, Communication Skills), selected by a genetic algorithm with crossover, mutation, a memory bank, and soft-decaying rates. The fitness function is an intention-consistency evaluation that scores responses on a four-level scale from Explicit Rejection to Direct Compliance, with Level 3 or higher counted as a successful jailbreak. The discoveries are: (Finding 1) the expanded space yields unprecedented JSRs, notably 96%/87% on Claude-3.5 across AdvBench/CLAS; (Finding 2) the evolved prompts transfer across models without further optimization, including 24% JSR on o1; and (Finding 3) the intention-consistency evaluator matches human labels at 96.5% accuracy, beating binary judges, rule-intensive scorers, and specialized safety reward models.","pith_inferences":["The four identified components may not be exhaustive; other persuasion dimensions could extend the space further, and the authors' own ablations show each component matters, with Role carrying the most weight.","A direct test of the evaluation's objectivity would be to rerun the top-scoring prompts against a battery of independent human labelers and separate safety models, since the genetic optimizer's fitness function is the same custom scorer used to report the headline numbers.","The high transferability suggests that the discovered prompts exploit general persuasion patterns rather than model-specific quirks, which would imply that closing the vulnerability requires training against this broader strategy distribution, not just patching known tactics.","The 24% JSR on o1 hints that even reasoning-focused safety training is not immune; extending the method to larger datasets on o1 would clarify how much of the ceiling remains."],"forward_implications":["Safety-aligned closed-source models are substantially more vulnerable to black-box jailbreaks than previously reported, with failure rates near zero turning into success rates above 90%.","The transferability finding suggests that prompts optimized against one strong model can be reused against others, including models with different alignment training.","The intention-consistency evaluation, if accurate, could serve as a more reliable red-team metric than binary judges or safety reward models.","The strategy-space view predicts that further expanding component libraries will continue to raise success rates, rather than plateauing.","Current defenses like RA-LLM and SmoothLLM reduce but do not eliminate the attack, with JSR remaining above 60% in most settings."],"supporting_citations":[{"why":"PAIR is a main baseline and the source of the AdvBench 50-query subset; its near-zero JSR on Claude-3.5 sets the comparison bar.","marker":"Chao et al. (2023)"},{"why":"TAP is a main baseline providing the rule-intensive scoring method and the second comparison point on Claude-3.5.","marker":"Mehrotra et al. (2023)"},{"why":"GPTFuzzer is the strongest baseline, using fuzzing over predefined templates; CL-GSO's gains over its 4% JSR on Claude-3.5 are the key result.","marker":"Yu et al. (2023)"},{"why":"PAP's 40-strategy taxonomy is the largest prior strategy pool and motivates the claim that monolithic strategies cap effectiveness; also supplies the rule-intensive scoring baseline in the evaluation comparison.","marker":"Zeng et al. (2024)"},{"why":"The Elaboration Likelihood Model is the theoretical basis for decomposing strategies into central-route and peripheral-route components.","marker":"Petty and Briñol (2011)"},{"why":"The CLAS 2024 dataset provides the second benchmark and its harder jailbreak scenarios.","marker":"Xiang et al. (2024)"},{"why":"AdvBench, the primary benchmark, originates here.","marker":"Zou et al. (2023)"},{"why":"RA-LLM is one of the two defenses tested against CL-GSO.","marker":"Cao et al. (2023)"},{"why":"SmoothLLM is the other defense, applicable to closed-source models, and the setting where CL-GSO's Claude-3.5 JSR drops.","marker":"Robey et al. (2023)"}],"fun_headline_variants":["Genetic search hits 96% jailbreak success on Claude-3.5","From 4% to 96%: strategy space expansion cracks Claude-3.5","Recombined components beat jailbreak ceiling: 96% on Claude-3.5","Jailbreak prompts evolve to 96% success, transfer across models","ELM-based genetic optimization shatters jailbreak limits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline success rates rest on a custom GPT-4o-based intention-consistency judge that also serves as the genetic optimizer's fitness function; if that judge systematically over-scores the method's own prompts, the reported jumps over the baselines would be an artifact of the evaluation rather than a real vulnerability.","fun_headline_variants_meta":{"raw":{"variants":["Genetic search hits 96% jailbreak success on Claude-3.5","From 4% to 96%: strategy space expansion cracks Claude-3.5","Recombined components beat jailbreak ceiling: 96% on Claude-3.5","Jailbreak prompts evolve to 96% success, transfer across models","ELM-based genetic optimization shatters jailbreak limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1439,"prompt_tokens":983,"completion_tokens":456,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":356}},"tokens_in":599,"tokens_out":456,"duration_ms":5156,"temperature":1.0,"reasoning_tokens":356,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:30:41.765568+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 50 successful CL-GSO prompt-response pairs against Claude-3.5 and have independent human annotators label whether the response materially advanced the harmful intent, without knowing which prompts came from which method; if human agreement with the Level-3-or-higher classification falls well below 96%, the claimed success rate does not survive independent evaluation. A complementary check would be to run the same optimizer with a different judge (e.g., a second frontier model or a specialized safety classifier) and see whether the Claude-3.5 JSR stays above 90%.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Elaboration Likelihood Model is the theoretical basis for decomposing strategies into central-route and peripheral-route components."}],"review_version":1}