Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A preference-optimized model can automatically generate jailbreak prompts that work across multiple aligned black-box LLMs with very few queries, replacing handcrafted templates and white-box optimization.

desk verdict New preference-optimized jailbreak attackers are promising, but the circular reward/metric choice makes the headline ASR untrustworthy. read the letter →

arxiv 2412.15623 v1 pith:GGPJ3P3B submitted 2024-12-20 cs.CR cs.AI

Hongyi Li , Jiawei Ye , Jie Wu , Tianjie Yan , Chu Wang , Zhixin Li This is my paper · ORCID
classification cs.CRcs.AI
keywords jailbreakattackpreferenceoptimizationblack-boxLLMalignmentSimPOuniversaladversarialpromptssafetyevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes JailPO, a framework that trains small language models to produce jailbreak prompts automatically. It claims that preference optimization—the same family of techniques used to align LLMs—can teach an attack model to rewrite harmful questions into covert forms and to build complex templates that make aligned black-box LLMs answer instead of refuse. The central promise is scale and portability: one training run yields a generator that works across Llama2, Mistral, Vicuna, and GPT-3.5 (and later Llama3-70B, GPT-4, Command-R) with as few as one query per question, replacing handcrafted templates and costly white-box token optimization. If true, this turns LLM safety evaluation into a cheap, automated probe, while also lowering the bar for real attackers.

What carries the argument

The load-bearing mechanism is a two-stage training pipeline: supervised imitation of GPT-3.5 rewrites, followed by preference optimization with SimPO. The preference signal comes from ClassJudge, a binary classifier that scores whether a target model's response to a candidate prompt is an affirmative answer; prompts with higher scores are treated as preferred completions. Because the same classifier is also the primary evaluation metric, the whole method's reported effectiveness rests on that detector's judgments.

What would settle it

Take the set of prompts that JailPO generated and were scored as successful by ClassJudge, present the target LLM's responses to independent human annotators (or a separately trained harmfulness classifier), and measure the agreement rate; if agreement is low, or if the prompts' 'success' depends on phrasing that exploits the classifier, the central claim of genuine jailbreak effectiveness is not supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that jailbreak prompting can be learned end-to-end. Starting from a small aligned model (Llama2-7B), it first supervises fine-tuning on GPT-3.5-rewritten versions of 522 harmful questions and 78 existing templates, then applies Simple Preference Optimization (SimPO) on preference pairs where the reward is whether a target LLM's response is judged a positive answer by the RoBERTa-based ClassJudge detector. The resulting Question-Enhanced and Template-Enhanced models generate covert prompts that, according to the paper's measurements, outperform handcrafted and token-optimization baselines in attack success rate, question success rate, and robustness against perplexity and toxicity defenses.

Load-bearing premise

The pipeline assumes that the RoBERTa-based ClassJudge detector reliably identifies whether a response is a harmful answer; since ClassJudge is both the reward signal for preference optimization and the primary evaluation metric, a poorly calibrated or gameable detector would inflate both the training signal and the reported attack success rates.

Editorial extensions

If this is right

  • A single preference-optimized generator can probe multiple aligned LLMs automatically, removing the need for handcrafted templates or per-target white-box optimization.
  • Attack success improves with query iterations: MixAsking, which falls back to a template prompt when the target refuses, reaches the highest question success rate at acceptable query cost.
  • Different prompt families have different failure modes: complex templates achieve higher attack success, while covert question rewrites elicit riskier content and evade perplexity and toxicity defenses more often.
  • Safety alignment itself may be leveraged against the model: the same preference-optimization machinery used to align LLMs can be repurposed to break that alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because ClassJudge is both the training reward and the reported metric, the method's measured gains might reflect optimization against this specific detector; testing with independent human annotations or a different harmfulness classifier would clarify whether the attacks elicit genuinely harmful content.
  • The transferability of a Llama2-trained generator to GPT-4 and Command-R suggests that jailbreak prompt styles generalize across model families, which implies that defenses will need to target prompt transformations rather than specific templates.
  • A natural extension is to use the same two-stage recipe with a stronger (or ensemble) reward model, or to adapt the generator to a specific target with a few preference queries, turning jailbreak generation into an on-demand service.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces JailPO, a black-box jailbreak framework that trains two attack models—QEM and TEM—using supervised fine-tuning followed by Simple Preference Optimization (SimPO). Preference pairs are constructed by scoring responses of a fixed aligned LLM (Llama2-7B) with a RoBERTa-based binary detector, ClassJudge. The resulting attack models generate covert questions (QEPrompt), templates (TemplatePrompt), and a combined pattern (MixAsking). The method is evaluated on four aligned LLMs (Llama2, Mistral, Vicuna, GPT-3.5) plus three larger models, using ClassJudge and a second RoBERTa-large detector (Rob-lg) as automatic evaluators. The paper claims superior attack success rate, question success rate, efficiency, universality, and robustness against perplexity and LLM-Guard defenses, and includes ablations, query-iteration analyses, and tests on advanced models.

Significance. If the reported results are trustworthy, JailPO constitutes a meaningful advance in automated red-teaming: it generates jailbreak prompts with very few queries, transfers across target models, and is far more query-efficient than token-level optimization baselines such as GCG. The experimental coverage is extensive, including ablations, defense evaluations, and tests on current strong models (GPT-4, Llama3-70B, Command-R). However, the main effectiveness metric is the same detector used to score the preference data, so the validity of the headline attack-success numbers rests on an independent demonstration that JailPO's prompts elicit genuinely harmful content rather than classifier-positive outputs. The secondary RoBERTa evaluator and the Llama Guard high-risk analysis provide partial but incomplete mitigation, since they are also automatic classifiers and are not validated against human judgment in this paper.

major comments (3)
  1. [Problem Formulation, Eq. (1), Table 1] The circularity of using ClassJudge as both the reward signal for preference optimization and the primary attack-success metric is a load-bearing concern. Eq. (1) constructs preference pairs by scoring target-model responses with ClassJudge, and Table 1 reports ASR with the same detector. Consequently, the reported success rates may reflect JailPO's ability to exploit classifier blind spots rather than to elicit genuinely harmful content. The second evaluator, Rob-lg, is also a RoBERTa-based classifier and may share the same learned biases; no human evaluation is provided. The Llama Guard high-risk analysis (Figure 4) is reported separately and is not integrated into the main success-rate claim. Please add a human-annotation spot-check or an independent, non-classifier-based benchmark to validate the ASR numbers.
  2. [Table 1 and Algorithm 1 (Appendix A.5)] The query count for MixAsking appears inconsistent. Table 1 lists QN=520 for all JailPO patterns, but Algorithm 1 shows that MixAsking can issue up to two queries per question: first QEPrompt, and then TemplatePrompt if PatternJudge flags non-compliance. Thus the actual number of LLM calls for MixAsking is at most 1040, not 520. Please clarify whether QN counts questions or LLM queries; if it counts questions, the efficiency comparison against baselines should use the worst-case query count. This affects the claimed 'two orders of magnitude fewer queries' relative to TemplateJailbreak.
  3. [Experiments Settings (Settings paragraph)] The paper states that each experiment was repeated five times, yet no standard deviation, confidence interval, or significance test is reported anywhere. Without variance information, the observed differences in ASR/QSR—such as the 3.26% vs. 6.21% difference between QEPrompt and TemplatePrompt on Llama2 in Table 1—cannot be judged for statistical significance. Since many of the headline comparisons involve small percentage-point gaps, please report means with error bars or a significance analysis, at least for the main results in Tables 1 and 2 and Figures 4 and 5.
minor comments (6)
  1. [Introduction] The reference to SimPO is incorrectly attributed to (Wei, Haghtalab, and Steinhardt 2023); the correct citation is Meng, Xia, and Chen (2024).
  2. [Experiments Settings (Metrics)] 'Defense Rassing Rate' appears to be a typo; the metric should likely be 'Defense Passing Rate' or 'Defense Bypass Rate.'
  3. [Eq. (1)] The variable l in Eq. (1) is not defined in the main text. Please state explicitly that l is the number of queries to the target LLM used to evaluate each prompt.
  4. [Prompts setup] The PatternJudge used in MixAsking is only briefly described as a refusal-pattern detector from Zou et al. (2023). Please provide the exact set of refusal patterns or a precise description so the pattern is reproducible.
  5. [Enhanced Model Construction] The construction of the 6000 preference pairs for QEM is unexplained. With 522 questions and 10 responses each, the number of possible preference pairs is much larger; please specify how the pairs were selected or sampled.
  6. [Preliminary Experiments] The preliminary experiments use 50 questions from Deng et al. (2023), while the main training set uses 522 questions from multiple sources. Please clarify whether the test questions (AdvBench) are guaranteed disjoint from the training questions, since the paper states they are distinct but does not show the overlap analysis.

Circularity Check

1 steps flagged · score 6.0 of 10

JailPO's reward signal and its primary ASR metric are the same ClassJudge detector, so the reported attack success partially reduces to the training objective.

  1. fitted input called prediction [Methodology, 'Optimize Model against Preference', Eq. (1); Experiments, 'Evaluation' and Table 1]
    "By querying aligned LLM Md (we use Llama2-7B in our experiments) with yi ∈ yf , we employ detector to assess the jailbreak prompt quality of yi: Score(yi) = Σ_l S(yi, query(Md, yi)) (1) ... we assign preference labels based on the jailbreak success scores: yw = yi and yl = yj if Score(yi)>Score(yj); otherwise, assign yw = yj and yl = yi. ... ClassJudge is the primary evaluator discussed in the preceding section. ... The Attack Success Rate (ASR = o/t) is the main metric to evaluate jailbreak effectiveness."

    The same ClassJudge function S is used twice: Eq. (1) computes the reward Score(yi) that determines which jailbreak prompts are preferred/dispreferred for SimPO training, and the 'Evaluation' paragraph defines ASR using ClassJudge as the primary detector. A response is a 'success' in Table 1 exactly when S(p, y)=1, the same oracle that produced the preference labels. The attack model is therefore optimized to maximize the precise quantity later reported as attack success; on Llama2 the reported ASR is essentially the training reward, and on other target LLMs it inherits the same detector's biases.

full rationale

The central circularity is real and localized: JailPO's preference-optimization reward (Eq. 1) is the ClassJudge detector score, and the primary evaluation metric (ASR in Table 1) is the same detector's positive classification. Thus the claim of 'superior attack effectiveness' is in part a direct consequence of optimizing against the evaluator. However, this is not a complete identity: training feedback comes only from Llama2 responses, while results are reported on Mistral, Vicuna, and GPT-3.5; the paper also reports Rob-lg results and Llama Guard high-risk analysis, which are not part of the training reward and provide some independent evidence. The RoBERTa-based Rob-lg may share biases with ClassJudge, but it is still a separate fitted model not used in Eq. (1). No load-bearing self-citation chain or imported uniqueness theorem appears; SimPO, ClassJudge, and the baseline attacks are external prior work. Because one of the paper's main predictive claims—attack success rate—reduces by construction to the same classifier used to construct the preference data, a moderate circularity score of 6 is appropriate rather than a higher score that would require the entire framework to be vacuous.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework depends on the reliability of the automatic jailbreak judge, the transferability of behavior-specific feedback from Llama-2, and the validity of SimPO as a preference optimizer. No code or data is released, so these dependencies cannot currently be independently checked.

free parameters (3)
  • reward scaling constant alpha (Eq. 2) = default from SimPO (Meng et al. 2024)
    Controls the scaling of the reward difference; not tuned for jailbreak, adopted from SimPO paper.
  • target reward margin beta (Eq. 3) = default from SimPO (Meng et al. 2024)
    Helps differentiate preferred vs dispreferred responses; inherited from SimPO hyperparameters.
  • preference dataset sizes (6000 question pairs, 2580 template pairs) = 6000 / 2580
    The size of the preference datasets is chosen by the authors based on available training queries, and directly affects the quality of the attack models.
assumptions (4)
  • domain assumption ClassJudge is a valid judge of jailbreak success.
    Used as reward in Eq. 1 and as primary metric in Table 1; if detector is biased, results are biased.
  • domain assumption Llama2-7B responses to candidate prompts provide a reliable preference signal for attacks against other LLMs.
    Preference pairs are constructed by querying Llama2 only; universality claims assume transfer.
  • standard math SimPO (reference-free preference optimization) is an effective optimizer for this setting.
    The paper relies on SimPO from Meng et al. 2024 without modification.
  • domain assumption AdvBench test questions are disjoint from the 522 training questions.
    The paper states the test questions are distinct from the training set; if overlap exists, results are inflated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs." pith.science (2026). https://pith.science/paper/GGPJ3P3B

@misc{pith2026241215623,
  author       = {Pith},
  title        = {Pith review of: JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GGPJ3P3B}},
  note         = {Machine review of arXiv:2412.15623}
}
read the original abstract

Large Language Models (LLMs) aligned with human feedback have recently garnered significant attention. However, it remains vulnerable to jailbreak attacks, where adversaries manipulate prompts to induce harmful outputs. Exploring jailbreak attacks enables us to investigate the vulnerabilities of LLMs and further guides us in enhancing their security. Unfortunately, existing techniques mainly rely on handcrafted templates or generated-based optimization, posing challenges in scalability, efficiency and universality. To address these issues, we present JailPO, a novel black-box jailbreak framework to examine LLM alignment. For scalability and universality, JailPO meticulously trains attack models to automatically generate covert jailbreak prompts. Furthermore, we introduce a preference optimization-based attack method to enhance the jailbreak effectiveness, thereby improving efficiency. To analyze model vulnerabilities, we provide three flexible jailbreak patterns. Extensive experiments demonstrate that JailPO not only automates the attack process while maintaining effectiveness but also exhibits superior performance in efficiency, universality, and robustness against defenses compared to baselines. Additionally, our analysis of the three JailPO patterns reveals that attacks based on complex templates exhibit higher attack strength, whereas covert question transformations elicit riskier responses and are more likely to bypass defense mechanisms.

Figures

Figures reproduced from arXiv: 2412.15623 by the authors.

Figure 1
Figure 1. (a) Comparison of attack success rates between [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Method overview. BaseM/FM/EM represents base model, fine-tuned model, and enhanced model, respectively. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Three patterns in JailPO. Question Enhanced Model (QEM). Our goal is to fine￾tune LLM to generate covert jailbreak questions that by￾pass model alignment. We collect 522 questions (Deng et al. 2023; Yu, Lin, and Xing 2023; Yuan et al. 2023; Deng et al. 2024) as the question origin query set Dq. Due to the limited number of questions, GPT-3.5 is employed to rephrase the query set, creating more complex expressions to… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: High-Risk response results of attacks. A higher [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: Effects of the query iteration w.r.t. QSR. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: (a) QEPrompt attack effectiveness across different [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 10
Figure 10. Figure 10: Effects of the query iteration w.r.t. QSR on GPT [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Safety Alignment of LMs via Non-cooperative Games

    cs.AI 2025-12 conditional novelty 7.0 of 10

    Jointly training an Attacker and Defender LLM in a non-zero-sum game with pairwise preference judges produces a defender with much lower jailbreak success while preserving general utility.

  2. LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems

    cs.CR 2025-09 conditional novelty 6.0 of 10

    A systematic review that categorizes LLM threats, severity scores, and mitigations across development and operation life cycles and multiple deployment scenarios.

Reference graph

Works this paper leans on

42 extracted references · 11 canonical work pages · cited by 2 Pith papers

  1. [1]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Alon, G.; and Kamfonas, M. 2023. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132

    arXiv 2023

  3. [3]

    G.; Guo, Z

    Azar, M. G.; Guo, Z. D.; Piot, B.; Munos, R.; Rowland, M.; Valko, M.; and Calandriello, D. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, 4447--4455. PMLR

    2024

  4. [4]

    Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  5. [5]

    A.; Adeli, E.; Altman, R.; Arora, S.; von Arx, S.; Bernstein, M

    Bommasani, R.; Hudson, D. A.; Adeli, E.; Altman, R.; Arora, S.; von Arx, S.; Bernstein, M. S.; Bohg, J.; Bosselut, A.; Brunskill, E.; et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258

    arXiv 2021

  6. [6]

    A.; and Terry, M

    Bradley, R. A.; and Terry, M. E. 1952. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39: 324

    work page 1952

  7. [7]

    J.; and Wong, E

    Chao, P.; Robey, A.; Dobriban, E.; Hassani, H.; Pappas, G. J.; and Wong, E. 2023. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419

    arXiv 2023

  8. [8]

    Deng, G.; Liu, Y.; Li, Y.; Wang, K.; Zhang, Y.; Li, Z.; Wang, H.; Zhang, T.; and Liu, Y. 2024. Masterkey: Automated jailbreaking of large language model chatbots. In Proc. ISOC NDSS

    work page 2024

Show all 42 references
  1. [9]

    J.; and Bing, L

    Deng, Y.; Zhang, W.; Pan, S. J.; and Bing, L. 2023. Multilingual jailbreak challenges in large language models. arXiv preprint arXiv:2310.06474

    2023 arXiv

  2. [10]

    Ethayarajh, K.; Xu, W.; Muennighoff, N.; Jurafsky, D.; and Kiela, D. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306

    2024 arXiv

  3. [11]

    A.; Matar, N.; Sowan, B.; Al Khaldy, M.; and Barham, H

    Fasha, M.; Rub, F. A.; Matar, N.; Sowan, B.; Al Khaldy, M.; and Barham, H. 2024. Mitigating the OWASP Top 10 For Large Language Models Applications using Intelligent Agents. In 2024 2nd International Conference on Cyber Resilience (ICCR), 1--9. IEEE

    2024

  4. [12]

    Ge, Y.; Hua, W.; Mei, K.; Tan, J.; Xu, S.; Li, Z.; Zhang, Y.; et al. 2024. Openagi: When llm meets domain experts. Advances in Neural Information Processing Systems, 36

    2024

  5. [13]

    Inan, H.; Upasani, K.; Chi, J.; Rungta, R.; Iyer, K.; Mao, Y.; Tontchev, M.; Hu, Q.; Fuller, B.; Testuggine, D.; et al. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674

    2023 arXiv

  6. [14]

    Jain, N.; Schwarzschild, A.; Wen, Y.; Somepalli, G.; Kirchenbauer, J.; Chiang, P.-y.; Goldblum, M.; Saha, A.; Geiping, J.; and Goldstein, T. 2023. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614

    2023 arXiv

  7. [15]

    Ji, J.; Qiu, T.; Chen, B.; Zhang, B.; Lou, H.; Wang, K.; Duan, Y.; He, Z.; Zhou, J.; Zhang, Z.; et al. 2023. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852

    2023 arXiv

  8. [16]

    Jones, E.; Dragan, A.; Raghunathan, A.; and Steinhardt, J. 2023. Automatically auditing large language models via discrete optimization. In International Conference on Machine Learning, 15307--15329. PMLR

    2023

  9. [17]

    Li, H.; Guo, D.; Fan, W.; Xu, M.; Huang, J.; Meng, F.; and Song, Y. 2023 a . Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197

    2023 arXiv

  10. [18]

    Li, X.; Zhou, Z.; Zhu, J.; Yao, J.; Liu, T.; and Han, B. 2023 b . Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191

    2023 arXiv

  11. [19]

    Liu, X.; Xu, N.; Chen, M.; and Xiao, C. 2024. Auto DAN : Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. In The Twelfth International Conference on Learning Representations

    2024

  12. [20]

    Liu, Y.; Deng, G.; Xu, Z.; Li, Y.; Zheng, Y.; Zhang, Y.; Zhao, L.; Zhang, T.; Wang, K.; and Liu, Y. 2023. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860

    2023 arXiv

  13. [21]

    Meng, Y.; Xia, M.; and Chen, D. 2024. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734

    2024 arXiv

  14. [22]

    Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 27730--27744

    2022

  15. [23]

    Perez, E.; Huang, S.; Song, F.; Cai, T.; Ring, R.; Aslanides, J.; Glaese, A.; McAleese, N.; and Irving, G. 2022. Red teaming language models with language models. arXiv preprint arXiv:2202.03286

    2022 arXiv

  16. [24]

    ProtectAI. 2023. Llm-guard. Accessed: 2024-07-21

    2023

  17. [25]

    D.; Ermon, S.; and Finn, C

    Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Ermon, S.; and Finn, C. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  18. [26]

    Schwinn, L.; Dobre, D.; G \"u nnemann, S.; and Gidel, G. 2023. Adversarial attacks and defenses in large language models: Old and new threats. In Proceedings on, 103--117. PMLR

    2023

  19. [27]

    Shen, T.; Jin, R.; Huang, Y.; Liu, C.; Dong, W.; Guo, Z.; Wu, X.; Liu, Y.; and Xiong, D. 2023. Large language model alignment: A survey. arXiv preprint arXiv:2309.15025

    2023 arXiv

  20. [28]

    Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  21. [29]

    Wang, Y.; Zhong, W.; Li, L.; Mi, F.; Zeng, X.; Huang, W.; Shang, L.; Jiang, X.; and Liu, Q. 2023. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966

    2023 arXiv

  22. [30]

    Wei, A.; Haghtalab, N.; and Steinhardt, J. 2023. Jailbroken: How Does LLM Safety Training Fail? In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  23. [31]

    Wen, J.; Ke, P.; Sun, H.; Zhang, Z.; Li, C.; Bai, J.; and Huang, M. 2023. Unveiling the Implicit Toxicity in Large Language Models. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 1322--1338....

    2023

  24. [32]

    Z.; Xiao, C.; and Chen, M

    Xu, N.; Wang, F.; Zhou, B.; Li, B. Z.; Xiao, C.; and Chen, M. 2023. Cognitive overload: Jailbreaking large language models with overloaded logical thinking. arXiv preprint arXiv:2311.09827

    2023 arXiv

  25. [33]

    Xu, Z.; Liu, Y.; Deng, G.; Li, Y.; and Picek, S. 2024. LLM Jailbreak Attack versus Defense Techniques--A Comprehensive Study. arXiv preprint arXiv:2402.13457

    2024 arXiv

  26. [34]

    Yang, J.; Jin, H.; Tang, R.; Han, X.; Feng, Q.; Jiang, H.; Zhong, S.; Yin, B.; and Hu, X. 2024. Harnessing the power of llms in practice: A survey on chatgpt and beyond. ACM Transactions on Knowledge Discovery from Data, 18(6): 1--32

    2024

  27. [35]

    Yi, S.; Liu, Y.; Sun, Z.; Cong, T.; He, X.; Song, J.; Xu, K.; and Li, Q. 2024. Jailbreak Attacks and Defenses Against Large Language Models: A Survey. arXiv e-prints, arXiv--2407

    2024

  28. [36]

    Yong, Z.-X.; Menghini, C.; and Bach, S. H. 2023. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446

    2023 arXiv

  29. [37]

    Yu, J.; Lin, X.; and Xing, X. 2023. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253

    2023 arXiv

  30. [38]

    Yuan, Y.; Jiao, W.; Wang, W.; Huang, J.-t.; He, P.; Shi, S.; and Tu, Z. 2023. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463

    2023 arXiv

  31. [39]

    Zhang, M.; Pan, X.; and Yang, M. 2023. Jade: A linguistics-based safety evaluation platform for llm. arXiv preprint arXiv:2311.00286

    2023 arXiv

  32. [40]

    Z.; and Fredrikson, M

    Zou, A.; Wang, Z.; Kolter, J. Z.; and Fredrikson, M. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043

    2023 arXiv

  33. [41]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  34. [42]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.