{"id":"47cadcf2-5dca-4365-b2e2-c94d071b60e8","arxiv_id":"2502.01154","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"JUMP and its handcrafted-initialized variant JUMP++ jailbreak LLMs by optimizing a set of universal multi-prompts with beam search and perplexity constraints, outperforming existing universal attack baselines on several open models.","lead":"The authors present JUMP, a method that optimizes a set of universal multi-prompts to jailbreak large language models, achieving higher attack success rates than prior universal attacks. The approach also adapts to a defense method, DUMP, which reduces the success of AutoDAN attacks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported ASR@k advantage may stem from Algorithm 3's inference-time ranking by the per-example test target y, not from the optimized universal prompts; baselines are not described as using this selection.","rationale":"The reader's weakest_assumption identifies Algorithm 3's loss-based ranking as the fragile link, and I agree that this is the right mechanism to scrutinize. My emphasis is slightly different: the more immediate threat to the central claim is not only whether the ranking transfers, but whether it is credited fairly to JUMP++ and not to the baselines. If AutoDAN and GPTFuzzer are evaluated without an equivalent target-aware selection step, Table 2's ASR@10 differences could reflect the selection rule rather than the quality of the optimized prompt set. The paper's Appendix A.2.2 only describes transforming baselines into the multi-prompt setting for training; the inference protocol is not specified. Additionally, because Algorithm 3 uses the test set's per-example y, the method is not fully 'universal' in the sense of attacking an unseen instruction without knowing a desired affirmative prefix. The proposed check—comparing all methods under identical inference protocols, plus a random-order control—would settle whether the optimization itself carries the reported gains. This does not contradict the reader's conditional verdict; it sharpens one condition that should be met before the headline claim is accepted.","tokens_in":22678,"tokens_out":12312,"duration_ms":124298,"concrete_test":"Re-run the Table 2 comparison for AutoDAN, GPTFuzzer, and JUMP++ under two identical inference protocols: (a) rank each method's optimized prompt set with Algorithm 3's L_pi(x,q,y) using the same per-example test target y; (b) control: try templates in random order, repeated over seeds. If protocol (a) brings baselines to within noise of JUMP++ or if protocol (b) collapses JUMP++'s ASR@10, the headline advantage is attributable to the target-loss ranking rather than to the universal multi-prompt optimization; if JUMP++ stays ahead under both, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 3 (Appendix A.1) selects which templates to try by sorting Q with L_pi(x,q,y), where y is the per-example affirmative target from the test set. This target-aware ranking is a substantial component of the reported ASR@k: on Llama2-7b (Table 2), JUMP++ jumps from ASR@1=15.4 to ASR@10=55.8, so most of the headline performance comes from the 10 best-loss templates rather than from a single universal prompt. The paper states (A.2.2) that AutoDAN and GPTFuzzer were 'transformed into the multi-prompt setting' for training, but it never specifies whether these baselines also received Algorithm 3's target-loss ranking at inference. If they did not, Table 2 conflates prompt-set quality with an uncredited test-time selection mechanism. The paper's own factorial ablation (Figure 3) reinforces the concern: GPTFuzzer initialized with JUMP++ seeds beats JUMP++ on Llama2-7b, so the optimization advantage is not robust across seeds. Without an ablation that removes or equalizes the inference-time ranking, the central claim 'outperforms existing techniques' is not isolated from this selection rule.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces JUMP, a beam-search-based method for optimizing a set of universal adversarial prompts (multi-prompts) to jailbreak aligned LLMs. It extends BEAST to a universal multi-prompt setting, with variants JUMP* (no perplexity control), JUMP (with perplexity constraints), and JUMP++ (with handcrafted initialization). The authors also propose DUMP, a defense variant. Experiments on AdvBench across five open models and transfer attacks to GPT-3.5/4/4o report that JUMP++ achieves higher ASR@10 than AutoDAN, GPTFuzzer, and AdvPrompter, while the paper's own ablations show sensitivity to initialization and admit limitations.","tokens_in":22956,"tokens_out":2925,"duration_ms":32283,"significance":"If the reported comparisons are fair, the paper contributes a practical method for training transferable universal multi-prompts without fine-tuning an attacker model, with code released and a held-out test set used for evaluation. The transfer results to GPT models and the DUMP defense extension broaden the contribution. However, the central 'outperforms existing techniques' claim is currently not isolated from an asymmetric inference-time selection mechanism and from initialization effects that the authors themselves document, so the significance is conditional on a fair and more controlled comparison.","major_comments":[{"comment":"Algorithm 3 sorts the optimized adversarial set Q by L_pi(x, q, y), where y is the per-instance affirmative target from the test set, and the k trials then test the templates in this sorted order. The paper states in §A.2.2 that AutoDAN and GPTFuzzer were 'transformed into the multi-prompt setting' for training, but it never states whether these baselines also receive the same target-aware test-time ranking. As a result, the Table 2 ASR@10 improvements (e.g., Llama2-7b JUMP++ 55.8 vs AutoDAN 37.5 and GPTFuzzer 26.9) may be driven by this uncredited selection rule rather than by the quality of the optimized prompt set. The authors should add an ablation that either removes the target-loss ranking from JUMP/JUMP++ or provides the same ranking procedure to all baselines, and should clarify the exact inference protocol used for each baseline.","section":"§3.3, Appendix A.1 (Algorithm 3), Table 2"},{"comment":"The seed-dependence ablation undermines the unqualified claim that JUMP++ outperforms existing techniques. On Llama2-7b, GPTFuzzer initialized with JUMP++ seeds achieves ASR 0.60 versus JUMP++ with the same seeds at 0.50, and on Llama3-8b AutoDAN initialized with JUMP++ seeds achieves 0.57 versus JUMP++ at 0.64. The authors themselves state in §4.5.4 that they 'perform worse on Llama2-7b' than GPTFuzzer. The paper should report the comparison as seed-conditional rather than as a general superiority claim, and should ideally report results averaged over multiple random seeds for all methods under identical initialization protocols.","section":"§4.5.4, Figure 3"},{"comment":"All reported ASR numbers appear to come from a single run, with no variance, confidence intervals, or significance tests. Given the stochasticity of multinomial sampling in the Mutator and Constraints steps and the small test-set sizes, differences of a few percentage points may not be meaningful. The paper should provide multiple-seed results, or at least a statistical test for its headline comparisons.","section":"Tables 2 and 3, Figure 3"},{"comment":"The headline version JUMP++ relies on handcrafted initialization, and the paper's own Limitations section concedes that 'the efficiency of our algorithm depends on the method of initialization' and that the transfer results in §4.5.5 are only satisfactory when initialized with AutoDAN prompts. This self-reported dependence is load-bearing because it changes the scope of the central claim: the contribution is not a fully automatic universal attack optimizer, but a search algorithm whose performance is strongly conditioned on human-provided seeds. The paper should either restrict its conclusions accordingly or provide evidence that the advantage over AutoDAN and GPTFuzzer persists under neutral initializations, not only under its own handcrafted seeds.","section":"§4.5.3, Table 2, §7 Limitations"}],"minor_comments":[{"comment":"The inference-time procedure described in the text ('we create a set of inputs... sorted inputs A by computing losses') is only fully specified in Algorithm 3 in Appendix A.1; the main text should include the algorithm reference or a brief description so readers can see that L_pi(x, q, y) is used at inference time.","section":"§3.3"},{"comment":"The sentence 'which refers to JUMP and JUMP*' appears to be a typo; the context indicates the authors mean JUMP and JUMP++.","section":"§A.2.2"},{"comment":"The BEAST-univ rows show ASR@10 columns as '—' while reporting only ASR@1 values; this makes it difficult to compare BEAST-univ with AdvPrompter on the same metric.","section":"Table 8"},{"comment":"The citation 'Anil et al.' lacks a year and venue; please provide the full bibliographic entry.","section":"References"},{"comment":"The heatmap labels are difficult to read, especially the numeric values in the Llama2-7b panel; please increase font size or use annotations with clearer formatting.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the code release is a plus, but the fairness of the baseline comparison is currently the main obstacle. The asymmetric use of test-target loss for ranking at inference time is a serious methodological concern that cannot be resolved without an ablation. If the authors add that ablation and report multi-seed results, the paper could become publishable. The paper's own Limitations section already acknowledges the initialization dependence, which is good intellectual honesty but also narrows the claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: this is a real, useful step in universal jailbreak attacks. The core idea is simple—instead of optimizing one universal suffix, optimize a set with min_Q sum_min_q L—and it works. The set beats single universal baselines (BEAST-univ, AdvPrompter) on the Llama models and transfers to GPT models. That multi-prompt sets outperform single universal prompts is new and non-obvious, and the authors show it with clean train/test splits and multiple evaluators.\n\nThe experiments are broad: five victim models, two evaluators, transfer to closed-source models, plus a defense adaptation (DUMP) that is a reasonable secondary contribution. Code is public. The authors are also honest about their limitations—they explicitly flag the initialization dependence and the role of handcrafted seeds in JUMP++.\n\nThe soft spots are real but not fatal. First, the reported ASR@k depends heavily on Algorithm 3's inference-time ranking: templates are sorted by cross-entropy loss on the test example's target y, then the top k are tried. That is a sensible selection rule, but the baselines are only described as \"transformed into the multi-prompt setting\"—never as receiving the same target-aware ranking. If AutoDAN and GPTFuzzer just iterate through their prompt sets in a fixed order, Table 2 conflates prompt quality with the selection mechanism. The paper needs an ablation where all methods use the same inference selection, or at least a check showing the gains persist without target-aware ranking. Second, the best variant JUMP++ relies on handcrafted seeds, and their own Figure 3 shows GPTFuzzer initialized with JUMP++ seeds beats JUMP++ on Llama2-7b. That doesn't invalidate the contribution, but it means the optimization advantage is seed-dependent. Also, there are no variance or significance estimates anywhere; with single runs, a few seeds for the main table would help.\n\nThe stress-test note is on point: the central claim \"outperforms existing techniques\" is not fully isolated from the selection rule. Still, the underlying result—multi-prompt sets beat single universal prompts—looks solid, and the paper is a serious empirical contribution to LLM safety evaluation.\n\nWho is this for? Researchers working on adversarial prompting and red-teaming evaluation. A serious referee can get value from it, especially if the authors add the missing inference-selection control. I'd send it out for review, with that ablation as a major request.","headline":"A solid extension of universal jailbreak attacks with a clean multi-prompt objective, but the headline comparison needs an ablation controlling for the inference-time target-loss ranking.","tokens_in":23449,"tokens_out":2084,"would_cite":true,"duration_ms":22593,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fixed pool of jointly optimized universal prompts, seeded with handcrafted role-play templates, jailbreaks aligned large language models on unseen harmful instructions more reliably than existing universal attacks.","keywords":["jailbreaking","universal adversarial prompts","multi-prompt attack","large language models","beam search","perplexity constraint","transfer attack","red-teaming"],"falsifier":"Re-run the ASR@10 evaluation with the ten trial prompts chosen uniformly at random instead of sorted by target-string loss; if random ordering reproduces the reported test-set attack rates (55.8% on Llama-2-7b, 82.7% on Llama-3-8b, string matching), then the loss-ranking mechanism in Algorithm 3 is not what carries the result.","tokens_in":22490,"feed_emoji":"🔓","tokens_out":13222,"duration_ms":124873,"temperature":0.7,"pith_summary":"The paper's claim is that jailbreaking can be made universal without per-request optimization or attacker fine-tuning: a fixed set of prompt templates, jointly optimized over batches of harmful instructions, can transfer to unseen requests and even to unseen models. The proposed method, JUMP, extends a beam-search attack so that a frozen attacker model proposes suffixes while candidates are scored by average cross-entropy loss on the victim model's expected affirmative completion. A perplexity constraint makes prompts less detectable, and handcrafted role-play seeds recover the attack success the constraint costs, giving JUMP++. On test instructions, JUMP++ reaches 55.8% attack success (string matching, ten trials) on Llama-2-7b and 82.7% on Llama-3-8b, exceeding the AutoDAN and GPTFuzzer baselines. The same framework inverted as DUMP is claimed to reduce AutoDAN's success on Vicuna and Mistral below no-defense and SmoothLLM baselines. A sympathetic reader would take away that a single reusable prompt pool is a practical transfer attack, and that the same optimizer can be turned toward defense.","feed_headline":"A shared prompt pool cracks unseen harmful requests at up to 83 percent","feed_subtitle":"A fixed set of optimized prompts transfers across instructions and models, and the same machinery doubles as a defense.","key_machinery":"The load-bearing object is the adversarial template pool Q: a set of M prompt templates containing a placeholder for the harmful instruction. The objective is $\\min_{Q} \\sum_{(x,y)} \\min_{q \\in Q} L_{\\pi}(x,q,y)$, where $L_{\\pi}$ is the cross-entropy loss the victim model assigns to the affirmative target $y$ given instruction $x$ and template $q$. The search alternates a Selector that samples K templates, a Mutator that asks a frozen attacker model $\\phi$ to extend each template with sampled token continuations, an Evaluator that scores each candidate by the average loss over a batch of instructions, and a Selector that keeps the best candidate per beam and merges it into the pool. A Constraints step samples from each beam with probability proportional to $e^{s_k/T}$ with $s_k = 1/\\text{ppl}_k$, biasing the pool toward lower-perplexity prompts; JUMP++ initializes the pool with handcrafted role-play templates such as 'Act as Yoda...' frames. At inference, Algorithm 3 computes $L_{\\pi}(x, q, y)$ for every template, sorts the pool, and queries the victim model with the top-k templates in that order, which is exactly what turns a static pool into an adaptive per-instance attacker.","core_discovery":"The paper's central discovery is that the universal multi-prompt objective — minimize, over a pool of templates Q, the sum over training instructions of the minimum loss to an affirmative target string — is tractable by beam search, and that the resulting pool beats existing universal attackers. Concretely, the plain version JUMP* outperforms AdvPrompter on most tested models at the cost of high perplexity; adding a perplexity-based sampling step drops attack success by more than ten points on both Llama models; and initializing the pool with AutoDAN's handcrafted prompts or with the authors' own role-play templates reverses that drop. In the enhanced JUMP++ configuration the pool outperforms AutoDAN and GPTFuzzer on the harder Llama-2-7b and Llama-3-8b targets, and when initialized with AutoDAN prompts it transfers to closed GPT models through an open-source proxy. The authors further claim the same optimization, run as DUMP against individual AutoDAN attacks, suppresses attack success on Vicuna-7b and Mistral-7b on both train and test data.","pith_inferences":["Because Algorithm 3's rank-and-query step is the only component that adapts the pool to a new instruction, the framework predicts that removing the loss-based ordering, for example by randomizing trial order, should substantially reduce ASR@10; this is a direct, untested consequence of the paper's own mechanism.","The ablation results suggest the search is a local refiner around seeds rather than a discoverer of entirely novel prompt families; a cheap experiment is to initialize JUMP++ with diverse, independently generated paraphrases and measure whether ASR is preserved, which would separate search efficiency from seed quality.","ASR@10 counts a request as jailbroken if any of ten ranked attempts succeeds, so a practical defender could blunt the attack with per-conversation query caps or escalating refusal; measuring JUMP++ under such budgets would map the metric onto real deployment risk."],"forward_implications":["A single optimized pool of a few dozen templates can replace per-request optimization: at inference the attacker only needs to rank the pool and query the victim model k times, so the cost of universal attack shrinks to the cost of k forward passes.","On the hardest open models tested, the gain is large: JUMP++'s test ASR@10 (string matching) is 55.8% on Llama-2-7b and 82.7% on Llama-3-8b, versus 37.5% and 15.4% for AutoDAN and 26.9% and 39.4% for GPTFuzzer.","With AutoDAN seed prompts, the pool transfers to closed GPT models: for example, 92.3% ASR@10 and 66.3% ASR@1 on GPT-3.5-turbo and 51.9% and 28.8% on GPT-4 when optimized on a Llama-3-8b proxy, measured by Llama Guard.","The ASR-versus-perplexity trade-off is real but steerable: perplexity constraints can cut ASR by more than ten points on both Llama models, and a well-chosen seed set restores most of the loss.","DUMP, the defense version, lowers AutoDAN's ASR on Vicuna-7b and Mistral-7b on both train and test sets relative to no defense and SmoothLLM, indicating the same universal multi-prompt optimizer can be used for hardening."],"supporting_citations":[{"why":"Supplies the AdvBench dataset of harmful instruction/target pairs, the cross-entropy loss on affirmative targets that JUMP optimizes, the string-matching refusal patterns used as one judge, and the GCG attack family the paper situates itself against.","marker":"Zou et al., 2023"},{"why":"BEAST is the beam-search attack that JUMP extends from single-prompt optimization to a universal pool of multi-prompts.","marker":"Sadasivan et al., 2024"},{"why":"AdvPrompter is the principal universal attack baseline, and its train/test split of AdvBench is reused for all universal experiments.","marker":"Paulus et al., 2024"},{"why":"AutoDAN is a prompt-based baseline, the source of the seed prompts used to initialize JUMP++ in several experiments, and the attack DUMP defends against.","marker":"Liu et al., 2023"},{"why":"GPTFuzzer is the other prompt-based baseline, converted to the same multi-prompt setting for fair comparison.","marker":"Yu et al., 2023"},{"why":"Supplies the Selector/Mutator/Constraint/Evaluator decomposition of jailbreak pipelines that organizes JUMP's design.","marker":"Zhou et al., 2024b"},{"why":"Perplexity-based detection motivates the constraint step that balances attack success against prompt naturalness.","marker":"Alon and Kamfonas, 2023"},{"why":"SmoothLLM is the defense baseline DUMP is compared with in the defense experiments.","marker":"Robey et al., 2023"},{"why":"Llama Guard is the classifier-based judge used alongside string matching, and all transfer results are reported with it.","marker":"Inan et al., 2023"}],"fun_headline_variants":["Universal prompt pool beats per-case jailbreaks","JUMP: one prompt set cracks unseen harmful requests","Shared prompts attack and defend LLMs","Optimized prompt pool transfers to new targets","Multi-prompt jailbreak outperforms single attacks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack stands on the assumption that, for an unseen harmful request, the prompt that makes the victim model most strongly predict a canned 'Sure, here is...' answer is also the prompt most likely to draw out a real harmful response.","fun_headline_variants_meta":{"raw":{"variants":["Universal prompt pool beats per-case jailbreaks","JUMP: one prompt set cracks unseen harmful requests","Shared prompts attack and defend LLMs","Optimized prompt pool transfers to new targets","Multi-prompt jailbreak outperforms single attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1301,"prompt_tokens":887,"completion_tokens":414,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":346}},"tokens_in":503,"tokens_out":414,"duration_ms":4849,"temperature":1.0,"reasoning_tokens":346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T16:22:38.520812+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the ASR@10 evaluation with the ten trial prompts chosen uniformly at random instead of sorted by target-string loss; if random ordering reproduces the reported test-set attack rates (55.8% on Llama-2-7b, 82.7% on Llama-3-8b, string matching), then the loss-ranking mechanism in Algorithm 3 is not what carries the result.","supporting_citations":[],"review_version":1}