{"id":"52ef719e-1e40-4eb3-b092-66884a0d692d","arxiv_id":"2502.01925","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PANDAS boosts many-shot jailbreak success rate on open-source LLMs using positive affirmations, negative demonstrations, and Bayesian-optimized topic sampling.","lead":"This paper presents PANDAS, an attack that improves many-shot jailbreaking of large language models by adding praise phrases, refusal-then-correction demonstrations, and topic-aware sampling to the fabricated conversation history. It reports success-rate gains across several open-source models, plus a new dataset of harmful question-answer pairs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PANDAS's headline gains are measured on the same set used to tune it: PA/ND positions are chosen as the best on AdvBench50 (Sec. 4.1) and the AS distribution is BO-optimized without a held-out split, so the Table 1 improvements are likely inflated and need re-estimation on held-out prompts.","rationale":"The central claim is an empirical superiority claim, so the most load-bearing assumption is that the reported ASR numbers are unbiased comparisons. The paper's own protocol violates this: the PA/ND configuration is chosen by best performance on AdvBench50 (Sec. 4.1) and the adaptive sampling distribution is optimized per model on the evaluation data (Sec. 3.3, App. B). This is not an internal inconsistency of the method, but it invalidates the numerical comparisons as estimates of out-of-sample performance. The missing i-MSJ on AdvBench/HarmBench narrows the evidence for superiority over the stronger baseline to exactly the dataset used for tuning. I agree with the reader's weakest_assumption. That said, the paper has real strengths: transparent component ablations (Table 3), diversity across five models and two additional datasets, and released source code. The attention analysis is correlational and should not be read as mechanistic proof, but it is presented as suggestive. A simple held-out re-evaluation would settle the issue; if the gap persists, the paper's core message survives. Therefore the reader's CONDITIONAL verdict is appropriate; no change needed.","tokens_in":21800,"tokens_out":5573,"duration_ms":50189,"concrete_test":"Perform a random 50/50 split of AdvBench50 into tuning and evaluation halves. On the tuning half only, (a) run the four PA/ND position variants at 256 shots and pick the best, and (b) run the Bayesian optimization of the topic distribution per model. Freeze the chosen configuration and distribution, then evaluate PANDAS and MSJ on the untouched evaluation half with the same 3-restart protocol. If the PANDAS-minus-MSJ ASR-L gap at 256 shots falls below roughly 5 percentage points (or within the bootstrap CI of zero), the Table 1 gains are largely a test-set artifact; if the gap remains large (e.g., >10 points), the headline claim is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states that the PA/ND insertion positions are selected as 'the configuration that achieves the highest ASR-L for 256 shots on AdvBench50,' and Table 1 then reports PANDAS performance on the same AdvBench50 split. In Section 3.3, the adaptive sampling distribution z is learned by Bayesian optimization 'performed separately for each target model,' and Appendix B describes only random exploration and 50 BO steps, with no indication of a separate tuning set. Consequently, the AdvBench50 column is a selected maximum over four PA/ND configurations plus an optimized topic distribution, whereas MSJ and i-MSJ are fixed algorithms. This selection-on-evaluation makes the reported gap (e.g., 94 vs 80 for Llama-3.1-8B at 256 shots) an upper-biased estimate of the method's true advantage. The issue is compounded because i-MSJ is evaluated only on AdvBench50 (due to cost), so the one dataset with the stronger baseline is the one used for tuning. The consistent pattern across models and datasets suggests the effect is likely real in direction, but the magnitudes in Table 1 are not trustworthy without a held-out evaluation split or a multiple-comparison correction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PANDAS, a hybrid many-shot jailbreaking method that augments the standard many-shot jailbreaking (MSJ) prompt with three modifications: positive affirmation (PA) phrases, a negative demonstration (ND) that embeds a refusal-and-correction turn, and an adaptive sampling (AS) distribution over demonstration topics, where the topic distribution is learned per target model via Bayesian optimization. The authors introduce ManyHarm, a dataset of 2,400 harmful question-answer pairs across 12 policy topics, and evaluate on five open-weight 7-8B models using AdvBench50, AdvBench, and HarmBench. The central empirical claim is that PANDAS outperforms MSJ and i-MSJ under equal shot counts (up to 256) across all evaluated models and datasets, as measured by both LLM-based (ASR-L, Llama-Guard-3) and rule-based (ASR-R) attack success rates. A structured attention analysis is used to argue that PA and ND reinforce instruction-following by increasing the fraction of attention each demonstration directs to earlier demonstrations.","tokens_in":22029,"tokens_out":10010,"duration_ms":94312,"significance":"If the reported gains survive a properly separated evaluation protocol, PANDAS would be a simple, transferable, and computationally cheap improvement to long-context jailbreaking, which is directly relevant to red-teaming and safety alignment. The PA and ND components are practically valuable because they require no additional demonstrations and no search, and the release of source code supports reproducibility. The ManyHarm dataset, with its 12-topic structure and controlled formatting, is a useful resource for controlled studies of demonstration-level properties. However, the current evaluation protocol makes the headline magnitudes optimistic: the PA/ND insertion positions are selected on the evaluation set (Sec. 4.1), the adaptive sampling distribution is optimized without a held-out split (Sec. 3.3, Appendix B), and no error bars or significance tests accompany the reported numbers (Tables 1-2). The direction of the effect is likely real, since the component-wise results in Table 3 show consistent positive signs, but the size of the improvement and the independent contribution of the adaptive sampling component remain to be established.","major_comments":[{"comment":"The reported PANDAS numbers are selected maxima over configurations tuned directly on the evaluation set. Section 4.1 states that PA/ND insertion positions are chosen as \"the configuration that achieves the highest ASR-L for 256 shots on AdvBench50,\" and Table 1 reports PANDAS on the same AdvBench50 split; Section 3.3 says the adaptive sampling distribution z is learned by Bayesian optimization \"performed separately for each target model,\" with Appendix B giving 50 BO steps and no held-out split. The AdvBench50 column is therefore a fitted value rather than a prediction, while MSJ and i-MSJ are fixed or per-prompt algorithms without benchmark-level tuning. The same concern applies to the AdvBench and HarmBench columns if z is optimized on prompts from those datasets. This makes the reported gaps (e.g., 94 vs 80 ASR-L for Llama-3.1-8B at 256 shots) upper-biased estimates of the method's advantage. The authors should tune on one split (e.g., AdvBench50) and report held-out benchmarks only, or use nested/leave-one-topic-out evaluation, and should quantify the variance of the selection procedure.","section":"Sec. 4.1, Sec. 3.3, Appendix B, Table 1"},{"comment":"The BO reward is a first-token proxy (success if the first token is not \"I\", with outputs beginning with \"1\" also counted as successful in some cases), whereas Table 1 reports ASR-L and ASR-R from full responses judged by Llama-Guard-3. The paper does not show that maximizing this proxy correlates with the reported metrics, and 50 optimization steps after 5 random evaluations are unlikely to reliably converge on a 12-dimensional simplex with noisy evaluations. Without repeated BO runs or a validation set, the adaptive-sampling contribution in Table 3 (e.g., ASR-L 87.12 vs 85.19 for MSJ on AdvBench at 128 shots) is indistinguishable from noise. Please report the proxy-to-metric correlation, the number of target prompts used per BO evaluation, the variability of the optimized distribution across independent runs, and the ASR of the final distribution on held-out prompts.","section":"Appendix B, Sec. 3.3"},{"comment":"The evaluation provides no uncertainty quantification. The protocol aggregates three restarts as success-if-any (\"the jailbreak is considered successful if any of the 3 attempts succeeds\"), which collapses variability, and all reported numbers are point estimates without confidence intervals or significance tests. For example, the Qwen-2.5-7B AdvBench50 differences (e.g., 22 vs 10 at 256 shots) are based on 50 prompts with success-if-any aggregation, so differences of several percentage points are within the expected noise. The abstract's \"significantly outperforms\" is not supported by any statistical analysis. Please report per-restart ASR distributions, confidence intervals, or a paired bootstrap over the prompt set for the main comparisons.","section":"Tables 1 and 2, Sec. 4.1"},{"comment":"The partition definition in the attention analysis is internally inconsistent. The paper defines P_{i,j} for 1≤i≤j≤n with k∈[N_i,N_{i+1}), q∈[N_j,N_{j+1}), and k≤q. Under the stated convention that rows of A correspond to queries and each row sums to 1, causal attention imposes q≤k (a token attends to itself and earlier positions), so the condition k≤q selects the wrong triangle within a segment and makes P_{i,j} identically zero for i<j. Moreover, the subsequent definition R_i = 1 − S_{i,i} = Σ_{j=1}^{i−1} S_{i,j} requires S_{i,j} for j<i, which is not covered by the stated range 1≤i≤j≤n. As written, Eqs. (5)-(6) and Figures 4-5 cannot be reproduced from the definitions. The index ranges and the triangular condition should be corrected (probably j≤i with q≤k), and the reported reference scores should be recomputed or confirmed against the corrected definition.","section":"Sec. 4.5, Eq. (5)"}],"minor_comments":[{"comment":"The supplied version contains typographical and formatting issues: \"Maliclious\" in Figure 1, and several PANDAS rows in Table 1 show values concatenated with inconsistent spacing (e.g., \"96.0098.0094.00\"). Please ensure the camera-ready version aligns all numbers correctly.","section":"Figure 1, Table 1"},{"comment":"The selection rule in Sec. 4.1 (\"the configuration that achieves the highest ASR-L for 256 shots on AdvBench50\") is not reconciled with Table 3, where PA is \"added after random demonstrations\" and ND \"after the first demonstration.\" Please state explicitly whether Table 3 uses the selected configuration or a different fixed configuration, and unify the notation for the insertion position m.","section":"Sec. 4.1, Table 3"},{"comment":"The ManyHarm dataset is released \"upon request, subject to eligibility review,\" which limits reproducibility relative to a standard dataset release. Consider also providing a detailed metadata description and a seed-based sampling protocol so that researchers without access to the exact dataset can still replicate the experimental conditions approximately.","section":"Sec. 4.2"},{"comment":"The transferability results are based on only 20 prompts per source-target pair. The text describes effects like \"nearly 100%\" and asymmetry, but with n=20 the confidence intervals are wide; please report exact counts or bootstrap intervals for these comparisons.","section":"Appendix H, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the evaluation protocol. The paper's headline claim rests on numbers obtained after selecting the PA/ND configuration and the adaptive sampling distribution directly on the evaluation benchmarks. I would ask for a re-estimation with a clean split (for instance, tune on AdvBench50 and report held-out AdvBench and HarmBench, or use leave-one-topic-out for HarmBench), and for reporting of variance. The direction of the effect is plausible, especially for PA+ND, which has no free distribution to fit; the AS component is the least supported and should be treated cautiously. The attention analysis error in Sec. 4.5 is independent of the main ASR claim but should be fixed. This is a safety-critical empirical paper with a contributable dataset; the manuscript is within scope, but the evaluation protocol needs correction before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: PANDAS is a real, incremental extension of many-shot jailbreaking, and the paper is worth engaging with, but the headline numbers in Table 1 are not trustworthy as reported. The authors chose the PA/ND insertion positions as the best of four configurations on AdvBench50 at 256 shots, then report PANDAS on the same AdvBench50 split. The adaptive sampling distribution is fitted via Bayesian optimization on the same datasets used for evaluation. So the AdvBench50 columns (and to a lesser degree, the AdvBench and HarmBench columns) are selected maxima, not independent predictions. The direction of the effect is probably real—the gains are consistent across five models and three datasets, and the component ablations in Table 3 show PA and ND help even outside the tuned configuration—but the magnitudes are likely over-stated.\n\nWhat's actually new: the combination of positive affirmations, refusal-then-correction negative demonstrations, and topic-adaptive sampling is not in Anil et al. (2024) or Zheng et al. (2024). The ManyHarm dataset (2,400 curated Q-A pairs) is a reusable resource, and the attention analysis, while correlational, is a reasonable first look at why these prompt modifications matter. The evaluation is broad by the standards of this subfield: five open 8B models, three datasets, several defenses, and an ablation. That breadth is the paper's best evidence.\n\nThe soft spots, in order of how much they matter. First, the tuning-on-evaluation problem. This is not a fatal flaw—it's a re-estimation problem—but it means the 94 vs 80 gaps on AdvBench50 should be re-measured with the configuration fixed on a separate split. Second, no error bars or significance tests anywhere. With three restarts aggregated as success-if-any, the variance is hidden. Third, the AS component is the most novel and the least supported: the BO procedure is described in a paragraph and an appendix, and the only evidence that the optimized distribution beats uniform sampling is a table with no confidence intervals. Fourth, i-MSJ is only run on AdvBench50, so the stronger baseline is absent on the other two datasets.\n\nWho is this for? Anyone working on LLM safety evaluation or red-teaming. The dataset and the PA+ND trick are immediately usable, and the transferability results are useful. It deserves a serious referee—not a desk reject—but the referee should ask for a held-out evaluation, variance reporting, and a proper comparison with i-MSJ across all datasets. My recommendation: engage with it, but treat the Table 1 numbers as upper bounds until re-estimated.","headline":"PANDAS is a useful, incremental extension of many-shot jailbreaking, but the headline gains are inflated because the adaptive sampling and PA/ND positions are tuned on the same benchmark used to report final numbers.","tokens_in":22614,"tokens_out":2954,"would_cite":true,"duration_ms":28152,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that many-shot jailbreaking can be made more effective by three prompt-level edits: positive affirmations inserted into the fabricated dialogue, a refusal-then-correction negative demonstration, and an optimized…","keywords":["many-shot jailbreaking","long-context attacks","positive affirmation","negative demonstration","adaptive sampling","Bayesian optimization","LLM safety alignment","attention analysis"],"falsifier":"Run PANDAS with the sampling distribution and PA/ND positions fixed exactly as selected during tuning, then evaluate on a disjoint holdout split of AdvBench and HarmBench (prompts never used in the Bayesian optimization or placement search) using the same judge; if the ASR-L gap over MSJ on that holdout is small or negative, the paper's central claim of a robust improvement over baselines would be refuted.","tokens_in":21559,"feed_emoji":"🔓","tokens_out":11352,"duration_ms":90105,"temperature":0.7,"pith_summary":"This paper aims to show that many-shot jailbreaking, which tricks a safety-aligned language model into answering harmful prompts by prefacing them with hundreds of fabricated user–model exchanges, can be made more effective through three inexpensive prompt-level edits. It introduces PANDAS, which inserts positive affirmation phrases into the fabricated dialogue, embeds a refusal-then-correction 'negative demonstration' into one exchange, and samples the demonstration topics from a distribution optimized by Bayesian optimization for the target prompt's topic. Across five open-source models and the AdvBench and HarmBench benchmarks, PANDAS reports higher attack success rates than standard many-shot jailbreaking and an improved variant (i-MSJ) at the same number of demonstrations. If the reported gains hold, the work suggests that long-context safety alignment can be bypassed with simple template modifications, and it provides a new benchmark dataset and attention-based evidence for how the model's attention shifts toward earlier demonstrations.","feed_headline":"PANDAS beats long-context jailbreak baselines on five models","feed_subtitle":"Affirmations, refusal-correction demos, and tuned topic sampling beat MSJ and i-MSJ on two benchmarks.","key_machinery":"The technical machinery is three modifications to the fabricated dialogue. Positive affirmation (PA) inserts a short appreciative phrase before a malicious question; negative demonstration (ND) rewrites one demonstration as question–refusal–correction–harmful answer; adaptive sampling (AS) replaces uniform topic sampling with a distribution over twelve topics that is optimized by Bayesian optimization separately for each target model. The paper also introduces the ManyHarm dataset of 2,400 malicious question–answer pairs under a controlled structure to support the attack. For analysis, it defines a segment-level reference score $R_i = 1 - S_{i,i}$, the fraction of attention segment $i$ directs toward previous segments, and shows PA and ND raise this score, tying the attack improvements to increased attention on the demonstrated compliant pattern.","core_discovery":"The central discovery is that the instruction-following pattern created by many-shot jailbreaking can be strengthened without adding more demonstrations. PA phrases ('Exactly the detail I needed! Thanks!') placed before subsequent malicious questions raise the cost of deviating from the established compliant pattern; ND sequences (a refusal, a user correction, then the original harmful answer) demonstrate how to override refusals; and an adaptive sampling distribution over twelve harmful-demonstration topics, learned via Bayesian optimization, yields higher attack success than uniform sampling. The paper reports that PANDAS consistently outperforms both MSJ and i-MSJ across Llama-3.1-8B, Qwen-2.5-7B, GLM-4-9B, OpenChat-3.6-8B, and OLMo-2-7B on AdvBench50, AdvBench, and HarmBench, with ASR-L gains of up to roughly 30 percentage points on the harder Qwen-2.5-7B case. Attention analysis shows that PA and ND increase the 'reference score'—the fraction of attention each demonstration segment pays to earlier segments—which the paper interprets as reinforcement of the instruction-following pattern.","pith_inferences":["If the gains survive evaluation on held-out prompts, safety teams may need to treat topical context engineering, not just token-level or instruction-level attacks, as a primary long-context threat in open 8B models.","The Bayesian-optimized distributions favoring 'regulated-content' and 'sexual' topics (Figure 2) suggest that certain topic categories act as stronger context primes; testing this topic-transfer hypothesis on other model families is a direct extension.","The attention reference score $R_i$ could be repurposed as an interpretability diagnostic: measuring whether a defense reduces reference-score growth might predict whether it will hold under long-context attacks.","Because PANDAS's gains did not require i-MSJ's costly random search, the comparison hints that rich context structure may matter more than token-level probability tuning in the many-shot regime."],"forward_implications":["PA and ND can be added to existing many-shot jailbreaking prompts with no extra inference cost and still raise attack success, lowering the barrier to mounting stronger long-context attacks.","Because jailbreak effectiveness does not reliably increase with shot count on 8B models, evaluating long-context attacks at a single shot count can miss the peak, which appeared as early as 32 shots on GLM-4-9B.","Input-perturbation defenses such as Retokenization and SmoothLLM lose their protective effect at high shot counts because the model begins mimicking the perturbations, so defense evaluations should include long-context settings.","Successful MSJ and PANDAS prompts transfer across models, with transferability improving at higher shot counts and showing asymmetry between GLM-4-9B and Qwen-2.5-7B, which supports using open models to generate transferable attacks against costly proprietary models.","Perplexity-based defenses fail against both MSJ and PANDAS because the attacks use fluent, in-distribution text rather than special strings."],"supporting_citations":[{"why":"Establishes the many-shot jailbreaking (MSJ) template and reports that broadly distributed demonstration sampling outperforms narrow sampling.","marker":"Anil et al. (2024)"},{"why":"Provides the competing-objectives explanation of jailbreaking failure that motivates the positive affirmation and negative demonstration designs.","marker":"Wei et al. (2023a)"},{"why":"Introduces the improved few-shot jailbreaking (i-MSJ) baseline, with special tokens and random search, that PANDAS is compared against.","marker":"Zheng et al. (2024)"},{"why":"Supplies the AdvBench dataset and the refusal-phrase list used to compute the rule-based attack success rate.","marker":"Zou et al. (2023)"},{"why":"Supplies the HarmBench dataset and the evidence that attack effectiveness is stable within model families but varies across families.","marker":"Mazeika et al. (2024)"},{"why":"Introduced the AdvBench50 subset on which PANDAS tunes the placement of affirmations/negative demonstrations and the sampling distribution.","marker":"Chao et al. (2023)"},{"why":"Supplies the Bayesian optimization methodology used to learn the adaptive sampling distribution.","marker":"Shahriari et al. (2015)"},{"why":"Provides the open-source Bayesian optimization toolbox used in the implementation.","marker":"Nogueira (2014)"}],"fun_headline_variants":["PANDAS strengthens many-shot jailbreaks without more demos","Affirmations and adaptive sampling boost jailbreak attacks","PANDAS improves long-context jailbreaking across five models","No extra demos: PANDAS refines many-shot jailbreak patterns","Positive affirmation and adaptive sampling make jailbreaks stronger"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measured advantage may not survive on prompts that were not used to tune the method, because the adaptive sampling distribution and the placement of the affirmations and negative demonstrations were chosen using the same evaluation prompts that appear in the headline results.","fun_headline_variants_meta":{"raw":{"variants":["PANDAS strengthens many-shot jailbreaks without more demos","Affirmations and adaptive sampling boost jailbreak attacks","PANDAS improves long-context jailbreaking across five models","No extra demos: PANDAS refines many-shot jailbreak patterns","Positive affirmation and adaptive sampling make jailbreaks stronger"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1335,"prompt_tokens":957,"completion_tokens":378,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":294}},"tokens_in":573,"tokens_out":378,"duration_ms":3807,"temperature":1.0,"reasoning_tokens":294,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T13:57:33.070617+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PANDAS with the sampling distribution and PA/ND positions fixed exactly as selected during tuning, then evaluate on a disjoint holdout split of AdvBench and HarmBench (prompts never used in the Bayesian optimization or placement search) using the same judge; if the ASR-L gap over MSJ on that holdout is small or negative, the paper's central claim of a robust improvement over baselines would be refuted.","supporting_citations":[{"cited_title":"J., et al","cited_arxiv_id":null,"evidence_quote":"Establishes the many-shot jailbreaking (MSJ) template and reports that broadly distributed demonstration sampling outperforms narrow sampling."},{"cited_title":"Improved few-shot jailbreaking can circumvent aligned language models and their defenses","cited_arxiv_id":null,"evidence_quote":"Introduces the improved few-shot jailbreaking (i-MSJ) baseline, with special tokens and random search, that PANDAS is compared against."},{"cited_title":"Z., and Fredrikson, M","cited_arxiv_id":null,"evidence_quote":"Supplies the AdvBench dataset and the refusal-phrase list used to compute the rule-based attack success rate."},{"cited_title":"Harmbench: A standardized evaluation framework for automated red teaming and robust refusal","cited_arxiv_id":null,"evidence_quote":"Supplies the HarmBench dataset and the evidence that attack effectiveness is stable within model families but varies across families."},{"cited_title":"J., and Wong, E","cited_arxiv_id":null,"evidence_quote":"Introduced the AdvBench50 subset on which PANDAS tunes the placement of affirmations/negative demonstrations and the sampling distribution."},{"cited_title":"P., and De Freitas, N","cited_arxiv_id":null,"evidence_quote":"Supplies the Bayesian optimization methodology used to learn the adaptive sampling distribution."},{"cited_title":"Bayesian Optimization : Open source constrained global optimization tool for Python , 2014","cited_arxiv_id":null,"evidence_quote":"Provides the open-source Bayesian optimization toolbox used in the implementation."}],"review_version":1}