Pith. sign in

REVIEW 4 major objections 5 minor 37 references

No Free Lunch for Defending Against Prefilling Attack by In-Context Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that in-context learning with adversative demonstrations — 'Sure, here is how to do it.

desk verdict A real empirical effect—ICL with affirmative lead-ins followed by 'However, I cannot assist' sharply cuts prefilling jailbreak success on most open-weight models—but the paper's 'adversative structure' mechanism is confounded with the lead-in itself, and the missing over-defense table needs to reappear. read the letter →

arxiv 2412.12192 v1 pith:ET7ZDUNR submitted 2024-12-13 cs.CR cs.AI

Zhiyu Xue , Guangliang Liu , Bocheng Chen , Kristen Marie Johnson , Ramtin Pedarsani This is my paper · ORCID
classification cs.CRcs.AI
keywords jailbreakattacksprefillingattackin-contextlearningLLMsafetyadversativedemonstrationssuccessrateover-defensivenessalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a simple, compute-free defense works against a stubborn class of jailbreak: when the attacker pre-fills the model's response with affirmative tokens like 'Sure, here is how to do it,' the model tends to continue along that path. The authors' claim is that including demonstrations with adversative structures — 'Sure... However, I cannot assist' — in the prompt teaches the model to refuse despite the affirmative prefix, across most tested open-weight LLMs and benchmarks. A sympathetic reader should care because prefilling attacks currently defeat standard safety alignment, and ICL is a cheap alternative that requires no fine-tuning. The paper also shows the defense has a cost: it makes models over-refuse benign queries, and the over-refusal appears independent of model size.

What carries the argument

The mechanism is the adversative sentence structure in the ICL demonstrations: a template response that starts with affirmative continuation ('Sure, here is how to do it...') and then pivots with a contrastive conjunction ('However') into a refusal. The paper operationalizes this as 'Adv' (a single adversative response) and 'Adv-mul' (a pool of multiple versions with the 'However' placed at different positions, randomly sampled per query). This structure does the argument's work by making the model's continuation after the prefilled affirmative tokens switch into refusal, whereas a plain refusal demonstration only models the 'no' answer and fails to override the affirmative prefix.

What would settle it

Run the same evaluation but build the demonstration pool by sampling harmful queries directly from the test benchmarks (e.g., use AdvBench queries as demonstrations for JailBench tests). If ASR remains near zero, the defense is robust to demonstration novelty; if ASR rises sharply, the reported defense depends on the demonstration pool being disjoint from the test queries. A cheaper check is to compute the maximum n-gram overlap between the generated demonstrations and the three benchmarks; high overlap confirms the leakage risk.

Watch

Extended reading notes

Core claim

The central claim is that ICL with adversative demonstrations is an effective defense against prefilling jailbreak attacks. Concretely, on AdvBench, JailBench, and SorryBench, replacing plain refusal demonstrations (e.g., 'No, I cannot answer') with adversative ones (e.g., 'Sure, here is how to do it. However, I reconsider and cannot assist with this question.') drops attack success rates from high baselines to near zero on most of the nine evaluated open-weight models, including Falcon-7b/11b, Llama-2-7b/13b, Llama-3.1-8b, Llama-3.2-3b, and Vicuna-7b/13b; Mistral-7B-v01 is the main exception. The paper further shows that safety alignment (comparing Wizard-Vicuna-13B with Vicuna-13B) does not mitigate the prefilling attack, that more demonstrations and larger models strengthen the defense, and that combined attacks (prefill plus AIM, Evil Confidant, or Refusal Suppression) degrade it but still leave adversative ICL relatively more effective than refusal ICL. The 'no free lunch' is the over-defensiveness: the same demonstrations cause LLMs to refuse benign queries, and this behavior appears independent of model size.

Load-bearing premise

The demonstrations used for ICL were generated separately from the test queries, and the paper assumes they are semantically disjoint from the harmful queries in AdvBench, JailBench, and SorryBench; if the demonstration pool overlaps with or retrievably resembles the test set, the measured defense could be test-set memorization rather than a general learned refusal pattern.

Editorial extensions

If this is right

  • Safety alignment alone does not stop prefilling attacks; adding adversative ICL demonstrations does, on most tested open-weight LLMs and across three benchmarks.
  • Increasing the number of adversative demonstrations (up to about eight) and using larger model sizes further reduces attack success, while plain refusal demonstrations remain ineffective.
  • Combining prefilling with other jailbreak techniques (AIM, Evil Confidant, Refusal Suppression) weakens but does not erase the defense; adversative ICL still beats refusal ICL in those combined settings.
  • The same demonstrations cause over-defensiveness: LLMs refuse benign queries more often, and this trade-off holds across model sizes, so deployment needs a selectivity mechanism.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors hint that the defense works because 'However' pivots are rare in pretraining data; this could be tested by varying the pivot word (e.g., 'Yet', 'But') and by moving the conjunction to different positions within the demonstration.
  • Because the demonstration pool was generated independently of the evaluation benchmarks, a test-set-overlap check is the natural next experiment: if generated demonstrations are semantically near any AdvBench, JailBench, or SorryBench query, the near-zero ASR may partly reflect retrieval rather than a general refusal skill.
  • The over-defensiveness result implies a calibration design: adversative ICL could be switched on only for inputs resembling the harmful-question distribution, preserving benign utility while blocking prefilled completions.
  • Extending the same prompt-level method to closed commercial LLMs (the paper evaluates only open-weight models) would show whether the adversative template transfers, and would give system-prompt writers a zero-training defence to try.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes using in-context learning (ICL) demonstrations that contain adversative sentence structures (e.g., "Sure, here is how to do it. However, I cannot assist...") to defend open-weight LLMs against prefilling jailbreak attacks, where the attacker pre-fills affirmative tokens. Across three benchmarks (AdvBench, JailBench, SorryBench) and nine models, the authors report large reductions in attack success rate (ASR) for their Adv and Adv-mul variants relative to a no-defense baseline and a refusal-demonstration baseline, for both rule-based and model-based ASR. They also study safety alignment, combined attacks, number of demonstrations, and over-defense. The paper concludes that ICL with adversative demonstrations is effective but causes over-defense, and that effectiveness is sensitive to textual similarity between demonstrations and queries.

Significance. If the result holds, the empirical finding that simple prompt-only demonstrations can neutralize prefilling attacks without fine-tuning is practically valuable, as prefilling attacks remain a known weakness of open-weight models. The study's breadth is a strength: it evaluates nine models, three benchmarks, two evaluation metrics, multiple prefilling token counts, and combined jailbreak attacks, with no fitted parameters. The paper also honestly documents over-defense and the sensitivity to textual similarity. However, the central mechanistic claim that the adversative conjunction is the active ingredient is not isolated from the affirmative lead-in, and the "robust across model sizes and complex attacks" claim is stronger than the data show. With additional ablations and toned-down claims, this could be a solid empirical contribution.

major comments (4)
  1. [§5.1, Appendix B] The comparison that supports the paper's central claim is confounded. In Table 1 and Appendix B, the Adv and Adv-mul demonstrations begin with an affirmative lead-in ("Sure, here is how to do it...") followed by "However" and a refusal, while the Refusal baseline is only "No, I cannot answer." These two demonstration families differ in both the lead-in and the adversative conjunction. Because the prefilling attack itself injects an affirmative prefix such as "Sure, here is...", the Adv demonstrations may simply teach the model to continue after an affirmative prefix with a refusal; the "However" may be incidental. The paper's conclusion that "adversative sentence structures" are the active ingredient is therefore not established. An ablation is needed, for instance comparing "Sure, here is how to do it. I cannot assist with this question." (lead-in without adversative conjunction) and "However, I cannot assist with this question." (adversative without affirmative lead-in) against the current Adv template. Section 6's admission that effectiveness is sensitive to textual similarity is consistent with the lead-in/content explanation, not uniquely with the adversative explanation.
  2. [§5.5] Section 5.5 refers to "Table 4" that summarizes over-defense results, but the manuscript as provided contains no Table 4; only Figure 4 is present. The over-defense claim is an explicit contribution of the paper (and the basis for the "no free lunch" conclusion), so the missing table with the benign-query refusal rates and harmful-query ASRs is load-bearing evidence that must be included or explicitly referenced to an appendix.
  3. [Abstract, §5.1, §5.3] The abstract and Section 5.1 claim that adversative demonstrations provide "robust defense across various model sizes and complex jailbreak attacks." This is contradicted by the paper's own data. In Table 1, Mistral-7B-v01 remains mostly vulnerable (Adv rule-based ASR 83.7, Adv-mul 60.8; model-based 84.8/63.1). In Table 2, combining prefilling with AIM, Evil Confidant, or Refusal Suppression raises ASR substantially; for example, Llama3.1-8b Adv-mul goes from 17 (prefill only) to 44 (Prefill+AIM), 79 (Prefill+EC), and 76 (Prefill+RS) on rule-based ASR, and Vicuna-7b reaches 68/75 on Prefill+EC. The paper's more careful statement that adversative demonstrations are "relatively more effective" than refusal demonstrations is supported, but the "robust" characterization is an overclaim that should be revised.
  4. [Appendix A, §4] Appendix A describes generating harmful demonstration questions with Wizard-13b and filtering with GPT-4o-mini, using ten categories and examples that resemble JailBench taxonomy items. The paper does not report any overlap or near-duplicate analysis between these generated demonstrations and the test queries in AdvBench, JailBench, and SorryBench used in Section 4. Since Section 6 attributes effectiveness partly to textual similarity between demonstrations and queries, the lack of a leakage analysis leaves open the possibility that the measured ASR reductions for some models reflect memorization or retrieval of near-identical test content rather than a general ICL capability. The authors should either demonstrate disjointness or quantify how many test queries are near-duplicates of demonstration questions.
minor comments (5)
  1. [Table 1 caption] The caption states "The utilized benchmarks are AdvBench and Jailbench" even though SorryBench results appear in the same table; the caption should list all three benchmarks.
  2. [Figure 2 caption] The caption contains a typo: "safty" should be "safety."
  3. [Appendix A] The example for "Harassment/Discrimination" is duplicated in the list of category examples; one duplicate should be removed.
  4. [References] The reference for "Anil et al., Many-shot jailbreaking" lacks a year and venue, and the entry for "Onel AJ" is inconsistently formatted; these should be brought into citation style.
  5. [Table 3 and Table 1] Table 3 lists Llama2-70b and Llama3.1-70b among evaluated LLMs, but these models do not appear in the main results tables; the authors should either report their results or clarify why they were excluded.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the ICL defense is evaluated on external benchmarks with independently generated demonstrations; self-citations are background, not load-bearing.

full rationale

The paper's central claim is an empirical finding, not a derivation: adversatively structured ICL demonstrations reduce prefilling-attack success rates across multiple external benchmarks (JailBench, AdvBench, SorryBench) and several open-weight LLMs on an independent attack success rate metric. The demonstrations are generated with Wizard-13b and filtered by GPT-4o-mini and rule-based filters, i.e., they are not fitted to or derived from the evaluation set. No parameter is fit to the target ASR, and no equation defines the defense in terms of the outcome it predicts. The self-citations (Liu et al. 2024, Mao et al. 2024, Qi et al. 2024b, Qi et al. 2024a) are contextual claims about ICL and alignment superficiality, not the evidence for the measured defense effect. The reviewer-identified confound that Adv and Refusal demonstrations differ in both the affirmative lead-in and the adversative conjunction is an experimental-control issue, not circularity, because the comparison is not true by construction and could in principle be resolved with additional ablations. The paper's own limitations about textual similarity and the absence of a fine-tuning comparison are honest caveats, not circular reductions. Thus the circularity score is low; the main risks are external-validity and confound risks, not self-referential reasoning.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted parameters exist; the paper is an empirical evaluation. The main hidden assumptions are the validity of the ASR metrics, the absence of test-set leakage in demonstration generation, and the representativeness of the small model family evaluated.

assumptions (3)
  • domain assumption Rule-based ASR (refusal keyword matching) and model-based ASR (sorry-bench finetuned judge) correctly measure jailbreak success.
    All conclusions in Section 5 rely on these metrics; the paper itself notes the two metrics diverge for Llama3.1-8b (Appendix C), so the choice of metric affects reported effectiveness.
  • domain assumption The adversarial demonstrations and benchmark test queries are disjoint, so the defense gains are not due to memorization of test-set answers.
    Demonstrations are generated from Wizard-13b and GPT-4o-mini (Appendix A), while tests use AdvBench, JailBench, and SorryBench (Section 4); no overlap check or leakage analysis is reported.
  • domain assumption The behavior of the evaluated open-source models (3B-13B) generalizes to other model sizes and families.
    Claims about 'various model sizes' are based on models up to 13B; 70B models listed in Table 3 are not evaluated in the main results. Mistral-7B-v01 is an outlier that the claim must accommodate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of No Free Lunch for Defending Against Prefilling Attack by In-Context Learning." pith.science (2026). https://pith.science/paper/ET7ZDUNR

@misc{pith2026241212192,
  author       = {Pith},
  title        = {Pith review of: No Free Lunch for Defending Against Prefilling Attack by In-Context Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ET7ZDUNR}},
  note         = {Machine review of arXiv:2412.12192}
}
read the original abstract

The security of Large Language Models (LLMs) has become an important research topic since the emergence of ChatGPT. Though there have been various effective methods to defend against jailbreak attacks, prefilling attacks remain an unsolved and popular threat against open-sourced LLMs. In-Context Learning (ICL) offers a computationally efficient defense against various jailbreak attacks, yet no effective ICL methods have been developed to counter prefilling attacks. In this paper, we: (1) show that ICL can effectively defend against prefilling jailbreak attacks by employing adversative sentence structures within demonstrations; (2) characterize the effectiveness of this defense through the lens of model size, number of demonstrations, over-defense, integration with other jailbreak attacks, and the presence of safety alignment. Given the experimental results and our analysis, we conclude that there is no free lunch for defending against prefilling jailbreak attacks with ICL. On the one hand, current safety alignment methods fail to mitigate prefilling jailbreak attacks, but adversative structures within ICL demonstrations provide robust defense across various model sizes and complex jailbreak attacks. On the other hand, LLMs exhibit similar over-defensiveness when utilizing ICL demonstrations with adversative structures, and this behavior appears to be independent of model size.

Figures

Figures reproduced from arXiv: 2412.12192 by the authors.

Figure 1
Figure 1. The example of defending against prefilling jail￾break attacks through in-context learning and adversative struc￾tures. By injecting adversative structure, a.k.a. however, in demonstrations, LLMs can defend prefilling jailbreak (back￾ground color of blue). prompt LLMs into providing responses that are harmful to society. Jailbreaking attacks aim to in￾ject a sequence of jailbreaking tokens into a harm￾ful query to e… view at source ↗
Figure 2
Figure 2. The effectiveness of safty alignment for defending against prefilling attack. Rule-based (left) and Model-based ASR (right) of aligned and unaligned LLMs on AdvBench. We utilized Wizard-Vicuna-13B and Vicuna-13B as the un￾aligned/aligned models, respectively. In this paper, aligned indicates that the LLM has been fine-tuned with safety align￾ment [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Over-defense performance examined through the handling rate trade-off between benign and harmful queries. The closer to the top-right, the better performance trade-off. We take the negative values of attack successful rate on harm￾ful queries and refusal rate on benign queries to indicate the harmful and benign performance, respectively. Note that the blue/red marker indicates the performance of Baseline/Adv￾mul. ac… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Our Pipeline for Harmful Question Generation [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 9 canonical work pages

  1. [1]

    ONeal AJ. 2023. https://gist.github.com/coolaj86/6f4f7b30129b0251f61fa7baaa881516 Chat gpt "dan"

    work page 2023

  2. [2]

    Alex Albert. 2023. https://github.com/Nikhil-Makwana1/ChatGPT-JailbreakChat Jailbreak Chat

    work page 2023

  3. [3]

    Gabriel Alon and Michael Kamfonas. 2023. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132

    arXiv 2023

  4. [4]

    Many-shot jailbreaking

    Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, et al. Many-shot jailbreaking

  5. [5]

    Simran Arora, Avanika Narayan, Mayee F Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher R \'e . 2022. Ask me anything: A simple strategy for prompting language models. arXiv preprint arXiv:2210.02441

    arXiv 2022

  6. [6]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

    arXiv 2022

  7. [7]

    Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. arXiv preprint arXiv:2404.01318

    arXiv 2024

  8. [8]

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. In R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models

    work page 2023

Show all 37 references
  1. [9]

    Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2024. Struq: Defending against prompt injection with structured queries. arXiv preprint arXiv:2402.06363

    2024 arXiv

  2. [10]

    Xiang Cheng, Yuxin Chen, and Suvrit Sra. 2024. Transformers implement functional gradient descent to learn non-linear functions in context. In Forty-first International Conference on Machine Learning

    2024

  3. [11]

    Chuan Guo, Alexandre Sablayrolles, Herv \'e J \'e gou, and Douwe Kiela. 2021. Gradient-based adversarial attacks against text transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5747--5757

    2021

  4. [12]

    Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614

    2023 arXiv

  5. [13]

    Piyush Jha, Arnav Arora, and Vijay Ganesh. 2024. Llmstinger: Jailbreaking llms using rl fine-tuned llms. arXiv preprint arXiv:2411.08862

    2024

  6. [14]

    Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. 2023. Automatically auditing large language models via discrete optimization. In International Conference on Machine Learning, pages 15307--15329. PMLR

    2023

  7. [15]

    Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. 2023. The unlocking spell on base llms: Rethinking alignment via in-context learning. In The Twelfth International Conference on Learning Representations

    2023

  8. [16]

    Feng Liu, Ke-Sheng Liu, Chao Chang, and Yan Wang. 2016. Research on the technology of ios jailbreak. In 2016 Sixth International Conference on Instrumentation & Measurement, Computer, Communication and Control (IMCCC), pages 644--647. IEEE

    2016

  9. [17]

    Guangliang Liu, Haitao Mao, Jiliang Tang, and Kristen Johnson. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.918 Intrinsic self-correction for enhanced morality: An analysis of internal mechanisms and the superficial hypothesis . In Proceedings of the 2024 Conference on Em...

    2024 doi

  10. [18]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning ...

    2022

  11. [19]

    Haitao Mao, Guangliang Liu, Yao Ma, Rongrong Wang, Kristen Johnson, and Jiliang Tang. 2024. A data generation perspective to the mechanism of in-context learning. arXiv preprint arXiv:2402.02212

    2024 arXiv

  12. [20]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing...

    2022

  13. [21]

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2024 a . Safety alignment should be made more than just a few tokens deep. arXiv preprint arXiv:2406.05946

    2024 arXiv

  14. [22]

    Zimo Qi, Guangliang Liu, Kristen Marie Johnson, and Lu Chen. 2024 b . Is moral self-correction an innate capability of large language models? a mechanistic analysis to self-correction. arXiv preprint arXiv:2410.20513

    2024

  15. [23]

    Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended abstracts of the 2021 CHI conference on human factors in computing systems, pages 1--7

    2021

  16. [24]

    Neeraj Varshney, Pavel Dolin, Agastya Seth, and Chitta Baral. 2023. The art of defending: A systematic evaluation and analysis of llm defense strategies on safety and over-defensiveness. arXiv preprint arXiv:2401.00287

    2023 arXiv

  17. [25]

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2024. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36

    2024

  18. [26]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. Transactions on Machine Learning Research

    2022

  19. [27]

    Zeming Wei, Yifei Wang, Ang Li, Yichuan Mo, and Yisen Wang. 2023. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387

    2023 arXiv

  20. [28]

    Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. 2023. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ...

    2023

  21. [29]

    Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, et al. 2024. Sorry-bench: Systematically evaluating large language model safety refusal behaviors. arXiv preprint arXiv:2406.14598

    2024 arXiv

  22. [30]

    An llm can fool itself: A prompt-based adversarial attack

    Xilie Xu, Keyi Kong, Ning Liu, Lizhen Cui, Di Wang, Jingfeng Zhang, and Mohan Kankanhalli. An llm can fool itself: A prompt-based adversarial attack. In The Twelfth International Conference on Learning Representations

  23. [31]

    Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. In International Conference on Machine Learning, pages 39818--39833. PMLR

    2023

  24. [32]

    Jiahao Yu, Xingwei Lin, and Xinyu Xing. 2023. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253

    2023 arXiv

  25. [33]

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36

    2024

  26. [34]

    Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. 2023. Autodan: Automatic and interpretable adversarial attacks on large language models. arXiv preprint arXiv:2310.15140

    2023 arXiv

  27. [35]

    Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043

    2023 arXiv

  28. [36]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  29. [37]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.