Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Direct GCG attacks succeed against a 20B-parameter LLM, but success rates drop sharply as models grow.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

The paper reports that GCG attack success drops with model size, prefix-based metrics overestimate harm, and coding prompts are more vulnerable, but these findings rest on a small model set and a GPT-4o pipeline that generates and judges prompts.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A straightforward empirical study with one useful scaling datapoint, but the coding-vs-safety claim rests on an unvalidated judge that also generated the coding prompts. the 4 major comments →

arxiv 2509.00391 v1 pith:CVFSLXIW submitted 2025-08-30 cs.CL cs.AIcs.CRcs.LG

The Resurgence of GCG Adversarial Attacks on Large Language Models

Yuting Tan , Xuying Li , Zhuo Li , Huizhen Shu , Peikang Hu This is my paper
classification cs.CL cs.AIcs.CRcs.LG
keywords adversarial attackjailbreakGreedy Coordinate GradientLLM safetyattack success ratesemantic evaluationsimulated annealingcode generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that gradient-based jailbreak attacks, specifically the Greedy Coordinate Gradient (GCG) algorithm, still work on very large language models, including a 20B-parameter model, but that attack success rates decline with model size. It also tries to show that common refusal-prefix heuristics overstate how often attacks succeed, and that coding- and reasoning-focused prompts are more vulnerable than standard safety benchmarks. If true, this would mean reasoning tasks are an overlooked attack surface, and that safety evaluations relying on string-matching heuristics consistently overestimate alignment quality. The paper further proposes an annealing-augmented variant, T-GCG, which diversifies adversarial search but does not yet deliver large gains under semantic judgment.

Core claim

On the paper's own terms, it demonstrates that GCG can elicit harmful outputs from GPT-OSS-20B, the largest model tested, with a 4.0% attack success rate on AdvBench prompt judged by GPT-4o and 12.3% on harmful coding prompts. ASR decreases monotonically with model size: on AdvBench, prefix-based ASR drops from 93.3% for Qwen2.5-0.5B to 68.7% for LLaMA-3.2-1B, and GPT-4o-judged ASR drops from 62.6% to 39.9% to 4.0% across the three models. Across all models, prefix-based heuristics yield much higher ASR than GPT-4o semantic judgments, and coding prompts consistently show higher ASR than AdvBench. The paper argues these patterns indicate that larger models have more complex loss landscapes, t

What carries the argument

The central object is the GCG suffix-optimization loop, which treats the adversarial suffix as coordinates and greedily replaces tokens according to gradient magnitude to maximize the likelihood of a harmful completion. The paper's contribution, T-GCG, adds temperature-weighted sampling at both the token-selection and suffix-acceptance stages, converting the deterministic greedy updates into a probabilistic search that can escape local minima. The dual evaluation protocol—prefix-based refusal detection versus GPT-4o semantic judgment—is the measurement machinery that produces the paper's discrepancy and vulnerability findings.

Load-bearing premise

The whole comparison rests on GPT-4o being a fair and accurate judge of harmfulness, yet the coding prompts it judges were generated by GPT-4o itself, which could bias the reported gap between coding and safety success rates.

What would settle it

Annotate the same attack outputs with human raters or with a second independent judge model and recompute the attack success rate gap between coding prompts and AdvBench; if the gap disappears, the paper's central claim that reasoning tasks are an especially exploitable attack surface fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Safety claims based on refusal-string heuristics overstate jailbreak resistance; evaluations should use semantic judges or human review.
  • Scale alone is not a defense: 20B models remain attackable, so large-model deployment still requires adversarial testing.
  • Coding assistants and reasoning-centric products should be treated as a separate, higher-risk surface for gradient-based attacks.
  • Annealing-style exploration can diversify adversarial suffixes but does not yet translate to higher semantic attack success, indicating that more tuning or alternative search strategies are needed.
  • Defenses based on reasoning or chain-of-thought safety training may be insufficient for code-generation domains, since reasoning can be co-opted as an attack vector.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If GPT-4o's judge output is influenced by its authorship of the coding prompts, the reported gap between coding and AdvBench ASR could shrink under an independent judge; a human-annotation head-to-head would settle it.
  • The decreasing-ASR-with-scale result hints at a qualitative change in loss landscape complexity between roughly 1B and 20B parameters; testing intermediate sizes (e.g., 3B, 7B) could reveal whether the decline is smooth or discontinuous.
  • The temperature-sampling trick could transfer to other discrete optimizers, such as genetic search or black-box coordinate methods, since it only changes candidate selection and acceptance.
  • For defenders, the coding-prompt vulnerability suggests that benchmark suites should include domain-specific malicious coding tasks rather than only generic harmful requests.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper reports an empirical study of the Greedy Coordinate Gradient (GCG) attack and a proposed temperature-based variant, T-GCG, on three open-weight models: Qwen2.5-0.5B-Instruct, LLaMA-3.2-1B-Instruct, and GPT-OSS-20B. The authors claim four main findings: (1) non-zero ASR can be achieved on a 20B model and ASR decreases with model size; (2) prefix-based heuristics overestimate attack success relative to GPT-4o semantic judgments; (3) coding-related prompts are more vulnerable than AdvBench safety prompts; and (4) T-GCG, with a tuned annealing schedule, improves prefix-based ASR over vanilla GCG, though semantic ASR is largely unchanged. The evaluation uses 10 random seeds per prompt and reports standard deviations.

Significance. If the findings hold, the paper would provide useful red-teaming evidence: it extends white-box GCG attacks to a 20B model, quantifies the gap between heuristic and semantic evaluation, and identifies code-generation as a potentially under-tested attack surface. The use of two evaluation protocols and reporting of means with standard deviations are strengths. However, the central claims currently rest on a single, unvalidated GPT-4o judge that also generated the coding prompts, on a three-model comparison that does not isolate model scale, and on a T-GCG prefix improvement that may be partly built into the optimization objective. These issues make the evidence suggestive rather than conclusive.

major comments (4)
  1. [Section 4.1, Appendix A, Table 1] The coding-prompt dataset is generated by GPT-4o, and GPT-4o is also the sole semantic judge. No human-annotated subset or second judge is provided. The reported coding-vs-AdvBench gaps (LLaMA-3.2-1B: 63.4% vs 39.9%; GPT-OSS-20B: 12.32% vs 4.0%) could therefore partly reflect the judge's bias toward outputs resembling its own generation style, or toward code responses that are longer and more actionable, rather than a true vulnerability of reasoning tasks. Please validate the judge against human labels, add an independent judge model, and generate the coding prompts from a public benchmark or with a different model.
  2. [Section 4.2, Table 1] The claim that 'attack effectiveness decreases as model size increases' is not supported as a size effect because the three models differ in family, training data, alignment method, and safety budget; no two models share a family, so parameter count is not isolated. The additional explanation that larger models have 'more complex and highly non-convex loss landscapes' is an unsupported hypothesis. Please reframe this as a descriptive observation over the specific models tested, or add same-family checkpoints (e.g., Qwen-0.5B/1.5B/7B) and/or direct loss-landscape diagnostics.
  3. [Section 4.2, Table 1] The prefix ASR for GPT-OSS-20B is reported as '—', and the text says prefix evaluation 'fails almost completely, since the model rarely produces explicit refusal markers.' Under the manuscript's own prefix rule (success = no refusal marker), the absence of refusal markers would imply high prefix ASR, not failure. If the model's output format makes the prefix rule inapplicable, this needs to be stated clearly. Without this number, the claim that prefix heuristics overestimate semantic ASR is untested on the largest model, and the 4.0% GPT-4o ASR cannot be compared against a heuristic baseline.
  4. [Section 3, Algorithm 1, Section 4.3, Table 2] T-GCG's suffix-acceptance rule samples candidates with probability proportional to exp(-Δℓ/T2), where Δℓ is computed from the same adversarial loss (cross-entropy against a targeted 'Sure,...' completion) that the prefix-based heuristic approximates. The reported prefix ASR improvement from 68.7 (Table 1) to 73.6 (α=0.005, Table 2) may therefore be an artifact of optimizing the evaluation metric itself, while GPT-4o ASR remains unchanged. Please clarify this circularity and test T-GCG with a semantic or judge-independent acceptance criterion, or state explicitly that the prefix gain is not evidence of stronger attack quality.
minor comments (4)
  1. [Section 3] Algorithm 1 uses 'T' both as the number of iterations and as temperature; the notation is confusing. The temperature schedule '0.01*(0.96^epoch)' appears only in the text, not in the algorithm or a formal definition.
  2. [Section 4.3] There are multiple typos and inconsistent symbols: 'Nevetherless', 'significalty', 'predix', 'responeces', and inconsistent use of 'LLama3.2' vs 'LLaMA-3.2'. Please copyedit.
  3. [Figures and Appendix] Fig. 4 appears only as a caption after the references and is not referenced in the body. Fig. 2 shows '<suffix>' placeholders rather than the actual adversarial suffixes, so the examples cannot be reproduced.
  4. [Section 4.1 / Reproducibility] No code, data, or model-checkpoint details are provided; the number of AdvBench prompts is not stated, the judge template is only an image, and suffix length / optimization hyperparameters are not fully specified. Releasing these would strengthen reproducibility.

Circularity Check

0 steps flagged

No circularity found; the empirical pipeline is self-contained, though the GPT-4o judge/generator overlap is a validity concern rather than a circularity.

full rationale

The paper's central claims are empirical measurements: ASR on AdvBench versus coding prompts, evaluated with prefix heuristics and a GPT-4o semantic judge. The optimization loss for GCG/T-GCG (cross-entropy toward a harmful completion) and the prefix-based success metric (absence of refusal tokens) are not definitionally identical; T-GCG's temperature-based sampling modifies the search procedure but does not define the evaluation outcome. No parameter is fitted to the reported ASR and then re-used as a prediction; alpha values in T-GCG are manually chosen and reported. The coding-prompt dataset is generated by GPT-4o and judged by GPT-4o, which may introduce a confound in the comparison between coding and safety prompts, but this is not a circular derivation—the paper does not define coding-prompt vulnerability in terms of GPT-4o's judgments such that the result is true by construction. No load-bearing self-citations or borrowed uniqueness theorems appear in the argument. The derivation chain is therefore free of circularity, though the evaluation protocol deserves independent validation.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central claims rest on a handful of hand-picked hyperparameters (T1 schedule, α) and several unverified assumptions about the evaluation pipeline and the interpretation of model-size trends. No new physical or conceptual entities are introduced.

free parameters (2)
  • temperature scale T1_0 and decay rate = T1 = 0.01 * 0.96^epoch
    Chosen by hand to define the temperature schedule for token sampling; not derived from theory.
  • alpha (α) in T2 = α·ℓ = 0.005 and 0.01
    Tuned to data; the paper admits fine tuning can improve results.
axioms (4)
  • domain assumption GPT-4o provides an accurate and unbiased semantic judgment of harmfulness.
    The paper uses GPT-4o as the ground truth for ASR, but does not validate it against human labels or other judges.
  • domain assumption The three models differ primarily in size, so ASR differences can be attributed to model scale.
    Qwen2.5-0.5B, LLaMA-3.2-1B, and GPT-OSS-20B have different training data, alignment methods, and families; size is not an isolated variable.
  • ad hoc to paper Larger models have more complex and highly non-convex loss landscapes, explaining lower ASR.
    Asserted in Section 4.2 without evidence or citation.
  • domain assumption The GPT-4o-generated coding prompts are representative of 'reasoning-intensive' tasks.
    No validation that these 100 synthetic prompts capture a general class of reasoning or coding vulnerability.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of The Resurgence of GCG Adversarial Attacks on Large Language Models." pith.science (2026). https://pith.science/paper/CVFSLXIW

@misc{pith2026250900391,
  author       = {Pith},
  title        = {Pith review of: The Resurgence of GCG Adversarial Attacks on Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CVFSLXIW}},
  note         = {Machine review of arXiv:2509.00391}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Gradient-based adversarial prompting, such as the Greedy Coordinate Gradient (GCG) algorithm, has emerged as a powerful method for jailbreaking large language models (LLMs). In this paper, we present a systematic appraisal of GCG and its annealing-augmented variant, T-GCG, across open-source LLMs of varying scales. Using Qwen2.5-0.5B, LLaMA-3.2-1B, and GPT-OSS-20B, we evaluate attack effectiveness on both safety-oriented prompts (AdvBench) and reasoning-intensive coding prompts. Our study reveals three key findings: (1) attack success rates (ASR) decrease with model size, reflecting the increasing complexity and non-convexity of larger models' loss landscapes; (2) prefix-based heuristics substantially overestimate attack effectiveness compared to GPT-4o semantic judgments, which provide a stricter and more realistic evaluation; and (3) coding-related prompts are significantly more vulnerable than adversarial safety prompts, suggesting that reasoning itself can be exploited as an attack vector. In addition, preliminary results with T-GCG show that simulated annealing can diversify adversarial search and achieve competitive ASR under prefix evaluation, though its benefits under semantic judgment remain limited. Together, these findings highlight the scalability limits of GCG, expose overlooked vulnerabilities in reasoning tasks, and motivate further development of annealing-inspired strategies for more robust adversarial evaluation.

Figures

Figures reproduced from arXiv: 2509.00391 by Huizhen Shu, Peikang Hu, Xuying Li, Yuting Tan, Zhuo Li.

Figure 1
Figure 1. Figure 1: Flow diagram for automatic generation of the most effective adversarial suffixes. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Examples of harmful behaviors elicited by our attack on GPT-OSS-20B, shown for general prompts [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ten individual prompt+suffix attacks against GPT-OSS-20B, evaluated on AdvBench and coding [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Examples of harmful coding-generation prompts [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The judge template for GPT-4o. , Vol. 1, No. 1, Article . Publication date: September 2025 [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. RouteHijack: Routing-Aware Attack on Mixture-of-Experts LLMs

    cs.LG 2026-05 unverdicted novelty 7.0

    RouteHijack is a routing-aware jailbreak that identifies safety-critical experts via activation contrast and optimizes suffixes to suppress them, reaching 69.3% average attack success rate on seven MoE LLMs with stron...

Reference graph

Works this paper leans on

39 extracted references · 10 canonical work pages · cited by 1 Pith paper · 4 internal anchors

  1. [1]

    Mohamed Ahmed, Mohamed Abdelmouty, Mingyu Kim, Gunvanth Kandula, Alex Park, and James C. Davis. 2025. Advancing Jailbreak Strategies: A Hybrid Approach to Exploiting LLM Vulnerabilities and Bypassing Modern Defenses. arXiv:2506.21972 [cs.CL] https://arxiv.org/abs/2506.21972 , Vol. 1, No. 1, Article . Publication date: September 2025. The Resurgence of GCG...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2025. Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks. In The Thirteenth International Conference on Learning Representations . https: //openreview.net/forum?id=hXA8wqRdyV

    work page 2025

  3. [3]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    Pith/arXiv arXiv 2023

  4. [4]

    Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

    Pith/arXiv arXiv 2022

  5. [5]

    Matan Ben-Tov, Mor Geva, and Mahmood Sharif. 2025. Universal Jailbreak Suffixes Are Strong Attention Hijackers. arXiv:2506.12880 [cs.CR] https://arxiv.org/abs/2506.12880

    arXiv 2025

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    Pith/arXiv arXiv 2021

  8. [8]

    Yifan Chen, Yixuan Wang, Hao Huang, Yu Tang, and Lei Zhou. 2024. AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulation

    work page 2024

  9. [9]

    Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soňa Mokrá, Nich...

    Pith/arXiv arXiv 2022

  10. [10]

    Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, Yossi Matias, Andrew Carroll, Kavita Kulkarni, Nenad Tomasev, Yuan Guan, Vi...

    Pith/arXiv arXiv 2025

  11. [12]

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv:2302.12173 [cs.CR] https://arxiv.org/abs/2302.12173

    Pith/arXiv arXiv 2023

  12. [13]

    Enkelejda Kasneci, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stephan Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia , Vol. 1, No. 1, Article . Publication date: September 2025. 10 Yuting Tan, Xuying Li, Zhuo Li, huizhen Shu, and Peikang Hu Nerdel, Jür...

    arXiv 2025

  13. [14]

    Taeyoun Kim, Fahim Tajwar, Aditi Raghunathan, and Aviral Kumar. 2025. Reasoning as an Adaptive Defense for Safety. (2025). arXiv:2507.00971 [cs.LG] https://arxiv.org/abs/2507.00971

    arXiv 2025

  14. [15]

    Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. 2023. Multi-step Jailbreaking Privacy Attacks on ChatGPT. arXiv:2304.05197 [cs.CL] https://arxiv.org/abs/2304.05197

    Pith/arXiv arXiv 2023

  15. [16]

    Jiahui Li, Yongchang Hao, Haoyu Xu, Xing Wang, and Yu Hong. 2024. Exploiting the Index Gradients for Optimization- Based Jailbreaking on Large Language Models. (2024). arXiv:2412.08615 [cs.CL] https://arxiv.org/abs/2412.08615

    Pith/arXiv arXiv 2024

  16. [17]

    Qizhang Li, Yiwen Guo, Wangmeng Zuo, and Hao Chen. 2024. Improved Generation of Adversarial Examples Against Safety-aligned LLMs. arXiv:2405.20778 [cs.CR] https://arxiv.org/abs/2405.20778

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  17. [18]

    Zeyi Liao and Huan Sun. 2024. AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs. arXiv:2404.07921 [cs.CL] https://arxiv.org/abs/2404.07921

    Pith/arXiv arXiv 2024

  18. [19]

    Narek Maloyan, Bislan Ashinov, and Dmitry Namiot. 2025. Investigating the Vulnerability of LLM-as-a-Judge Archi- tectures to Prompt-Injection Attacks. (2025). arXiv:2505.13348 [cs.CL] https://arxiv.org/abs/2505.13348

    Pith/arXiv arXiv 2025

  19. [20]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human fee...

    Pith/arXiv arXiv 2022

  20. [21]

    O’Brien, Carrie J

    Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. arXiv:2304.03442 [cs.HC] https://arxiv.org/abs/2304. 03442

    Pith/arXiv arXiv 2023

  21. [22]

    Logan IV, Eric Wallace, and Sameer Singh

    Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . 4222–4235

    work page 2020

  22. [23]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman, Blaise Agüera y Arcas, Dale Webster, Greg S. Corrado,...

    2023

  23. [24]

    Chawin Sitawarin, Norman Mu, David Wagner, and Alexandre Araujo. 2024. PAL: Proxy-Guided Black-Box Attack on Large Language Models. (2024). arXiv:2402.09674 [cs.CL] https://arxiv.org/abs/2402.09674

    Pith/arXiv arXiv 2024

  24. [25]

    Bulaong, John E

    Kyle Swanson, Wesley Wu, Nash L. Bulaong, John E. Pak, and James Zou. 2025. The Virtual Lab of AI agents designs new SARS-CoV-2 nanobodies. Nature (July 2025). doi:10.1038/s41586-025-09442-9

    doi:10.1038/s41586-025-09442-9 2025

  25. [26]

    Ben Thompson and Michael Sklar

    T. Ben Thompson and Michael Sklar. 2024. FLRT: Fluent Student-Teacher Redteaming. arXiv:2407.17447 [cs.CL] https://arxiv.org/abs/2407.17447

    Pith/arXiv arXiv 2024

  26. [27]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 [cs.CL] https://arxiv.org/ abs/2302.13971

    Pith/arXiv arXiv 2023

  27. [28]

    Victor Valbuena. 2024. WHITE PAPER: A Brief Exploration of Data Exfiltration using GCG Suffixes. arXiv:2408.00925 [cs.CR] https://arxiv.org/abs/2408.00925

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  28. [29]

    Haowei Wang, Rupeng Zhang, Junjie Wang, Mingyang Li, Yuekai Huang, Dandan Wang, and Qing Wang. 2025. Joint- GCG: Unified Gradient-Based Poisoning Attacks on Retrieval-Augmented Generation Systems. arXiv:2506.06151 [cs.CR] https://arxiv.org/abs/2506.06151

    work page arXiv 2025

  29. [30]

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: how does LLM safety training fail?. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 3508, 32 pages

    work page 2023

  30. [31]

    Prompt Recovery for Image Generation Models: A Comparative Study of Discrete Optimizers

    Joshua Nathaniel Williams, Avi Schwarzschild, Yutong He, and J. Zico Kolter. 2025. Prompt Recovery for Image Generation Models: A Comparative Study of Discrete Optimizers. arXiv:2408.06502 [cs.CV] https://arxiv.org/abs/2408. 06502

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  31. [32]

    Zhen Xu, Xiang Li, Hao Zhang, and Fan Wu. 2024. Faster GCG: Efficient Gradient-Based Adversarial Suffix Search. arXiv preprint arXiv:2410.15362 (2024)

    Pith/arXiv arXiv 2024

  32. [33]

    Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. 2023. FinGPT: Open-Source Financial Large Language Models. arXiv:2306.06031 [q-fin.ST] https://arxiv.org/abs/2306.06031 , Vol. 1, No. 1, Article . Publication date: September 2025. The Resurgence of GCG Adversarial Attacks on Large Language Models 11 Fig. 4. Examples of harmful coding-generation prompts

    arXiv 2023

  33. [34]

    Xiaoxue Yang, Bozhidar Stevanoski, Matthieu Meeus, and Yves-Alexandre de Montjoye. 2025. Alignment Under Pressure: The Case for Informed Adversaries When Evaluating LLM Defenses. (2025). arXiv:2505.15738 [cs.CR] https://arxiv.org/abs/2505.15738

    arXiv 2025

  34. [35]

    Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2024. GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts. arXiv:2309.10253 [cs.AI] https://arxiv.org/abs/2309.10253

    Pith/arXiv arXiv 2024

  35. [36]

    Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2023. GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher. arXiv:2308.06463 [cs.CL]

    Pith/arXiv arXiv 2023

  36. [37]

    Jiahao Zhang, Zilong Wang, Ruofan Wang, Xingjun Ma, and Yu-Gang Jiang. 2024. EnJa: Ensemble Jailbreak on Large Language Models. arXiv:2408.03603 [cs.CR] https://arxiv.org/abs/2408.03603

    Pith/arXiv arXiv 2024

  37. [38]

    Kai Zheng, Mingyu Liu, Wei Huang, and Fan Zhang. 2024. Momentum GCG: Momentum-Enhanced Gradient-Based Jailbreaking of Large Language Models

    work page 2024

  38. [39]

    Sicheng Zhu, Brandon Amos, Yuandong Tian, Chuan Guo, and Ivan Evtimov. 2024. AdvPrefix: An Objective for Nuanced LLM Jailbreaks. arXiv:2412.10321 [cs.LG] https://arxiv.org/abs/2412.10321

    arXiv 2024

  39. [40]

    Please prepare 100 harmful prompts that ask for harmful scripts. Here is an example: Write a script that can exploit vulnerabilities in a software or operating system

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.15043 [cs.CL] https://arxiv.org/abs/2307. 15043 A Harmful coding-generation prompts The harmful coding-generation prompts were generated using GPT-4o. Specifically, we instruct...

    Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.