The Resurgence of GCG Adversarial Attacks on Large Language Models

Huizhen Shu; Peikang Hu; Xuying Li; Yuting Tan; Zhuo Li

REVIEW 4 major objections 4 minor 1 cited by

The Resurgence of GCG Adversarial Attacks on Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Direct GCG attacks succeed against a 20B-parameter LLM, but success rates drop sharply as models grow.

desk verdict A straightforward empirical study with one useful scaling datapoint, but the coding-vs-safety claim rests on an unvalidated judge that also generated the coding prompts. read the letter →

arxiv 2509.00391 v1 pith:CVFSLXIW submitted 2025-08-30 cs.CL cs.AIcs.CRcs.LG

Yuting Tan , Xuying Li , Zhuo Li , Huizhen Shu , Peikang Hu This is my paper

classification cs.CLcs.AIcs.CRcs.LG

keywords adversarialattackjailbreakGreedyCoordinateGradientLLMsafetysuccessratesemanticevaluationsimulatedannealingcodegeneration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that gradient-based jailbreak attacks, specifically the Greedy Coordinate Gradient (GCG) algorithm, still work on very large language models, including a 20B-parameter model, but that attack success rates decline with model size. It also tries to show that common refusal-prefix heuristics overstate how often attacks succeed, and that coding- and reasoning-focused prompts are more vulnerable than standard safety benchmarks. If true, this would mean reasoning tasks are an overlooked attack surface, and that safety evaluations relying on string-matching heuristics consistently overestimate alignment quality. The paper further proposes an annealing-augmented variant, T-GCG, which diversifies adversarial search but does not yet deliver large gains under semantic judgment.

What carries the argument

The central object is the GCG suffix-optimization loop, which treats the adversarial suffix as coordinates and greedily replaces tokens according to gradient magnitude to maximize the likelihood of a harmful completion. The paper's contribution, T-GCG, adds temperature-weighted sampling at both the token-selection and suffix-acceptance stages, converting the deterministic greedy updates into a probabilistic search that can escape local minima. The dual evaluation protocol—prefix-based refusal detection versus GPT-4o semantic judgment—is the measurement machinery that produces the paper's discrepancy and vulnerability findings.

What would settle it

Annotate the same attack outputs with human raters or with a second independent judge model and recompute the attack success rate gap between coding prompts and AdvBench; if the gap disappears, the paper's central claim that reasoning tasks are an especially exploitable attack surface fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, it demonstrates that GCG can elicit harmful outputs from GPT-OSS-20B, the largest model tested, with a 4.0% attack success rate on AdvBench prompt judged by GPT-4o and 12.3% on harmful coding prompts. ASR decreases monotonically with model size: on AdvBench, prefix-based ASR drops from 93.3% for Qwen2.5-0.5B to 68.7% for LLaMA-3.2-1B, and GPT-4o-judged ASR drops from 62.6% to 39.9% to 4.0% across the three models. Across all models, prefix-based heuristics yield much higher ASR than GPT-4o semantic judgments, and coding prompts consistently show higher ASR than AdvBench. The paper argues these patterns indicate that larger models have more complex loss landscapes, t

Load-bearing premise

The whole comparison rests on GPT-4o being a fair and accurate judge of harmfulness, yet the coding prompts it judges were generated by GPT-4o itself, which could bias the reported gap between coding and safety success rates.

Editorial extensions

If this is right

Safety claims based on refusal-string heuristics overstate jailbreak resistance; evaluations should use semantic judges or human review.
Scale alone is not a defense: 20B models remain attackable, so large-model deployment still requires adversarial testing.
Coding assistants and reasoning-centric products should be treated as a separate, higher-risk surface for gradient-based attacks.
Annealing-style exploration can diversify adversarial suffixes but does not yet translate to higher semantic attack success, indicating that more tuning or alternative search strategies are needed.
Defenses based on reasoning or chain-of-thought safety training may be insufficient for code-generation domains, since reasoning can be co-opted as an attack vector.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If GPT-4o's judge output is influenced by its authorship of the coding prompts, the reported gap between coding and AdvBench ASR could shrink under an independent judge; a human-annotation head-to-head would settle it.
The decreasing-ASR-with-scale result hints at a qualitative change in loss landscape complexity between roughly 1B and 20B parameters; testing intermediate sizes (e.g., 3B, 7B) could reveal whether the decline is smooth or discontinuous.
The temperature-sampling trick could transfer to other discrete optimizers, such as genetic search or black-box coordinate methods, since it only changes candidate selection and acceptance.
For defenders, the coding-prompt vulnerability suggests that benchmark suites should include domain-specific malicious coding tasks rather than only generic harmful requests.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick read: this is an incremental but honest empirical paper. The genuinely new thing is that direct GCG gets a non-zero ASR on a 20B open model (GPT-OSS-20B) and that the ASR on coding prompts is higher than on AdvBench for that model. The dual prefix-vs-GPT-4o evaluation is also worth having, since it quantifies how much the usual refusal-string heuristic overcounts. They ran 10 seeds, report std, and the T-GCG variant is clearly labeled preliminary. The writing is rough in places, but the table and figures tell a coherent story.

Now the soft spots. The coding prompts were generated by GPT-4o and judged by GPT-4o with no human validation or second judge. That is a real circularity: the judge may systematically favor outputs that match its own prompt style, and the 12.32% vs 4.0% gap on GPT-OSS-20B is the main evidence for the "reasoning can be co-opted" claim. The missing prefix ASR for GPT-OSS-20B (they write "—") means the overestimation claim isn't actually tested on the largest model, which undercuts one of their three findings. The "first time" scaling claim is overreach; there are earlier results showing GCG succeeding on models in that size range, and the citation list doesn't support the way they phrase it. T-GCG is competitive at best, and the one significant-looking improvement (73.6 vs 68.7 prefix) is within the noise given the std. The judge template in Appendix B has no calibration or inter-annotator agreement, so the severity/breadth/novelty scores in Figure 3 are illustrative, not evidence.

None of this is fatal to the basic empirical trend. The scale effect is plausible and consistent with prior work, and the coding prompts likely are somewhat more vulnerable—but the paper doesn't prove that, because the judge confound is untreated. It's a desk-review decision that should go to referees, not a reject: the scaling result and the evaluation-protocol comparison deserve scrutiny, and a round of revision with external validation, artifacts, and a toned-down novelty claim could make it a solid workshop or short-conference contribution. I'd send it out, with the judge issue as the first question to the authors.

Referee Report

4 major / 4 minor

Summary. The paper reports an empirical study of the Greedy Coordinate Gradient (GCG) attack and a proposed temperature-based variant, T-GCG, on three open-weight models: Qwen2.5-0.5B-Instruct, LLaMA-3.2-1B-Instruct, and GPT-OSS-20B. The authors claim four main findings: (1) non-zero ASR can be achieved on a 20B model and ASR decreases with model size; (2) prefix-based heuristics overestimate attack success relative to GPT-4o semantic judgments; (3) coding-related prompts are more vulnerable than AdvBench safety prompts; and (4) T-GCG, with a tuned annealing schedule, improves prefix-based ASR over vanilla GCG, though semantic ASR is largely unchanged. The evaluation uses 10 random seeds per prompt and reports standard deviations.

Significance. If the findings hold, the paper would provide useful red-teaming evidence: it extends white-box GCG attacks to a 20B model, quantifies the gap between heuristic and semantic evaluation, and identifies code-generation as a potentially under-tested attack surface. The use of two evaluation protocols and reporting of means with standard deviations are strengths. However, the central claims currently rest on a single, unvalidated GPT-4o judge that also generated the coding prompts, on a three-model comparison that does not isolate model scale, and on a T-GCG prefix improvement that may be partly built into the optimization objective. These issues make the evidence suggestive rather than conclusive.

major comments (4)

[Section 4.1, Appendix A, Table 1] The coding-prompt dataset is generated by GPT-4o, and GPT-4o is also the sole semantic judge. No human-annotated subset or second judge is provided. The reported coding-vs-AdvBench gaps (LLaMA-3.2-1B: 63.4% vs 39.9%; GPT-OSS-20B: 12.32% vs 4.0%) could therefore partly reflect the judge's bias toward outputs resembling its own generation style, or toward code responses that are longer and more actionable, rather than a true vulnerability of reasoning tasks. Please validate the judge against human labels, add an independent judge model, and generate the coding prompts from a public benchmark or with a different model.
[Section 4.2, Table 1] The claim that 'attack effectiveness decreases as model size increases' is not supported as a size effect because the three models differ in family, training data, alignment method, and safety budget; no two models share a family, so parameter count is not isolated. The additional explanation that larger models have 'more complex and highly non-convex loss landscapes' is an unsupported hypothesis. Please reframe this as a descriptive observation over the specific models tested, or add same-family checkpoints (e.g., Qwen-0.5B/1.5B/7B) and/or direct loss-landscape diagnostics.
[Section 4.2, Table 1] The prefix ASR for GPT-OSS-20B is reported as '—', and the text says prefix evaluation 'fails almost completely, since the model rarely produces explicit refusal markers.' Under the manuscript's own prefix rule (success = no refusal marker), the absence of refusal markers would imply high prefix ASR, not failure. If the model's output format makes the prefix rule inapplicable, this needs to be stated clearly. Without this number, the claim that prefix heuristics overestimate semantic ASR is untested on the largest model, and the 4.0% GPT-4o ASR cannot be compared against a heuristic baseline.
[Section 3, Algorithm 1, Section 4.3, Table 2] T-GCG's suffix-acceptance rule samples candidates with probability proportional to exp(-Δℓ/T2), where Δℓ is computed from the same adversarial loss (cross-entropy against a targeted 'Sure,...' completion) that the prefix-based heuristic approximates. The reported prefix ASR improvement from 68.7 (Table 1) to 73.6 (α=0.005, Table 2) may therefore be an artifact of optimizing the evaluation metric itself, while GPT-4o ASR remains unchanged. Please clarify this circularity and test T-GCG with a semantic or judge-independent acceptance criterion, or state explicitly that the prefix gain is not evidence of stronger attack quality.

minor comments (4)

[Section 3] Algorithm 1 uses 'T' both as the number of iterations and as temperature; the notation is confusing. The temperature schedule '0.01*(0.96^epoch)' appears only in the text, not in the algorithm or a formal definition.
[Section 4.3] There are multiple typos and inconsistent symbols: 'Nevetherless', 'significalty', 'predix', 'responeces', and inconsistent use of 'LLama3.2' vs 'LLaMA-3.2'. Please copyedit.
[Figures and Appendix] Fig. 4 appears only as a caption after the references and is not referenced in the body. Fig. 2 shows '<suffix>' placeholders rather than the actual adversarial suffixes, so the examples cannot be reproduced.
[Section 4.1 / Reproducibility] No code, data, or model-checkpoint details are provided; the number of AdvBench prompts is not stated, the judge template is only an image, and suffix length / optimization hyperparameters are not fully specified. Releasing these would strengthen reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the empirical pipeline is self-contained, though the GPT-4o judge/generator overlap is a validity concern rather than a circularity.

full rationale

The paper's central claims are empirical measurements: ASR on AdvBench versus coding prompts, evaluated with prefix heuristics and a GPT-4o semantic judge. The optimization loss for GCG/T-GCG (cross-entropy toward a harmful completion) and the prefix-based success metric (absence of refusal tokens) are not definitionally identical; T-GCG's temperature-based sampling modifies the search procedure but does not define the evaluation outcome. No parameter is fitted to the reported ASR and then re-used as a prediction; alpha values in T-GCG are manually chosen and reported. The coding-prompt dataset is generated by GPT-4o and judged by GPT-4o, which may introduce a confound in the comparison between coding and safety prompts, but this is not a circular derivation—the paper does not define coding-prompt vulnerability in terms of GPT-4o's judgments such that the result is true by construction. No load-bearing self-citations or borrowed uniqueness theorems appear in the argument. The derivation chain is therefore free of circularity, though the evaluation protocol deserves independent validation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on a handful of hand-picked hyperparameters (T1 schedule, α) and several unverified assumptions about the evaluation pipeline and the interpretation of model-size trends. No new physical or conceptual entities are introduced.

free parameters (2)

temperature scale T1_0 and decay rate = T1 = 0.01 * 0.96^epoch
Chosen by hand to define the temperature schedule for token sampling; not derived from theory.
alpha (α) in T2 = α·ℓ = 0.005 and 0.01
Tuned to data; the paper admits fine tuning can improve results.

assumptions (4)

domain assumption GPT-4o provides an accurate and unbiased semantic judgment of harmfulness.
The paper uses GPT-4o as the ground truth for ASR, but does not validate it against human labels or other judges.
domain assumption The three models differ primarily in size, so ASR differences can be attributed to model scale.
Qwen2.5-0.5B, LLaMA-3.2-1B, and GPT-OSS-20B have different training data, alignment methods, and families; size is not an isolated variable.
ad hoc to paper Larger models have more complex and highly non-convex loss landscapes, explaining lower ASR.
Asserted in Section 4.2 without evidence or citation.
domain assumption The GPT-4o-generated coding prompts are representative of 'reasoning-intensive' tasks.
No validation that these 100 synthetic prompts capture a general class of reasoning or coding vulnerability.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Resurgence of GCG Adversarial Attacks on Large Language Models." pith.science (2026). https://pith.science/paper/CVFSLXIW

@misc{pith2026250900391,
  author       = {Pith},
  title        = {Pith review of: The Resurgence of GCG Adversarial Attacks on Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CVFSLXIW}},
  note         = {Machine review of arXiv:2509.00391}
}

read the original abstract

Gradient-based adversarial prompting, such as the Greedy Coordinate Gradient (GCG) algorithm, has emerged as a powerful method for jailbreaking large language models (LLMs). In this paper, we present a systematic appraisal of GCG and its annealing-augmented variant, T-GCG, across open-source LLMs of varying scales. Using Qwen2.5-0.5B, LLaMA-3.2-1B, and GPT-OSS-20B, we evaluate attack effectiveness on both safety-oriented prompts (AdvBench) and reasoning-intensive coding prompts. Our study reveals three key findings: (1) attack success rates (ASR) decrease with model size, reflecting the increasing complexity and non-convexity of larger models' loss landscapes; (2) prefix-based heuristics substantially overestimate attack effectiveness compared to GPT-4o semantic judgments, which provide a stricter and more realistic evaluation; and (3) coding-related prompts are significantly more vulnerable than adversarial safety prompts, suggesting that reasoning itself can be exploited as an attack vector. In addition, preliminary results with T-GCG show that simulated annealing can diversify adversarial search and achieve competitive ASR under prefix evaluation, though its benefits under semantic judgment remain limited. Together, these findings highlight the scalability limits of GCG, expose overlooked vulnerabilities in reasoning tasks, and motivate further development of annealing-inspired strategies for more robust adversarial evaluation.

Figures

Figures reproduced from arXiv: 2509.00391 by the authors.

**Figure 2.** Examples of harmful behaviors elicited by our attack on GPT-OSS-20B, shown for general prompts [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Ten individual prompt+suffix attacks against GPT-OSS-20B, evaluated on AdvBench and coding [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Examples of harmful coding-generation prompts [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

**Figure 5.** Figure 5: The judge template for GPT-4o. , Vol. 1, No. 1, Article . Publication date: September 2025 [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

RouteHijack: Routing-Aware Attack on Mixture-of-Experts LLMs
cs.LG 2026-05 unverdicted novelty 7.0 of 10

RouteHijack is a routing-aware jailbreak that identifies safety-critical experts via activation contrast and optimizes suffixes to suppress them, reaching 69.3% average attack success rate on seven MoE LLMs with stron...

Reference graph

Works this paper leans on

39 extracted references · 10 canonical work pages · cited by 1 Pith paper

[1]

Mohamed Ahmed, Mohamed Abdelmouty, Mingyu Kim, Gunvanth Kandula, Alex Park, and James C. Davis. 2025. Advancing Jailbreak Strategies: A Hybrid Approach to Exploiting LLM Vulnerabilities and Bypassing Modern Defenses. arXiv:2506.21972 [cs.CL] https://arxiv.org/abs/2506.21972 , Vol. 1, No. 1, Article . Publication date: September 2025. The Resurgence of GCG...

work page Pith review arXiv 2025
[2]

Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2025. Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks. In The Thirteenth International Conference on Learning Representations . https: //openreview.net/forum?id=hXA8wqRdyV

work page 2025
[3]

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

arXiv 2023
[4]

Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

arXiv 2022
[5]

Matan Ben-Tov, Mor Geva, and Mahmood Sharif. 2025. Universal Jailbreak Suffixes Are Strong Attention Hijackers. arXiv:2506.12880 [cs.CR] https://arxiv.org/abs/2506.12880

arXiv 2025
[6]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

2020
[7]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

arXiv 2021
[8]

Yifan Chen, Yixuan Wang, Hao Huang, Yu Tang, and Lei Zhou. 2024. AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulation

work page 2024

Show all 39 references

[9]

Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig...

2022 arXiv
[10]

Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Ko...

2025 arXiv
[12]

Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv:2302.12173 [cs.CR] https://arxiv.org/abs/2302.12173

2023 arXiv
[13]

Enkelejda Kasneci, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stephan Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia , Vol. 1, No. 1, Article . Publication date: September ...

2025
[14]

Taeyoun Kim, Fahim Tajwar, Aditi Raghunathan, and Aviral Kumar. 2025. Reasoning as an Adaptive Defense for Safety. (2025). arXiv:2507.00971 [cs.LG] https://arxiv.org/abs/2507.00971

2025
[15]

Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. 2023. Multi-step Jailbreaking Privacy Attacks on ChatGPT. arXiv:2304.05197 [cs.CL] https://arxiv.org/abs/2304.05197

2023 arXiv
[16]

Jiahui Li, Yongchang Hao, Haoyu Xu, Xing Wang, and Yu Hong. 2024. Exploiting the Index Gradients for Optimization- Based Jailbreaking on Large Language Models. (2024). arXiv:2412.08615 [cs.CL] https://arxiv.org/abs/2412.08615

2024 arXiv
[17]

Qizhang Li, Yiwen Guo, Wangmeng Zuo, and Hao Chen. 2024. Improved Generation of Adversarial Examples Against Safety-aligned LLMs. arXiv:2405.20778 [cs.CR] https://arxiv.org/abs/2405.20778

2024 arXiv
[18]

Zeyi Liao and Huan Sun. 2024. AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs. arXiv:2404.07921 [cs.CL] https://arxiv.org/abs/2404.07921

2024 arXiv
[19]

Narek Maloyan, Bislan Ashinov, and Dmitry Namiot. 2025. Investigating the Vulnerability of LLM-as-a-Judge Archi- tectures to Prompt-Injection Attacks. (2025). arXiv:2505.13348 [cs.CL] https://arxiv.org/abs/2505.13348

2025 arXiv
[20]

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

2022 arXiv
[21]

O’Brien, Carrie J

Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. arXiv:2304.03442 [cs.HC] https://arxiv.org/abs/2304. 03442

2023 arXiv
[22]

Logan IV, Eric Wallace, and Sameer Singh

Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) ...

2020
[23]

Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansf...

2023
[24]

Chawin Sitawarin, Norman Mu, David Wagner, and Alexandre Araujo. 2024. PAL: Proxy-Guided Black-Box Attack on Large Language Models. (2024). arXiv:2402.09674 [cs.CL] https://arxiv.org/abs/2402.09674

2024 arXiv
[25]

Bulaong, John E

Kyle Swanson, Wesley Wu, Nash L. Bulaong, John E. Pak, and James Zou. 2025. The Virtual Lab of AI agents designs new SARS-CoV-2 nanobodies. Nature (July 2025). doi:10.1038/s41586-025-09442-9

2025 doi
[26]

Ben Thompson and Michael Sklar

T. Ben Thompson and Michael Sklar. 2024. FLRT: Fluent Student-Teacher Redteaming. arXiv:2407.17447 [cs.CL] https://arxiv.org/abs/2407.17447

2024 arXiv
[27]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation La...

2023 arXiv
[28]

Victor Valbuena. 2024. WHITE PAPER: A Brief Exploration of Data Exfiltration using GCG Suffixes. arXiv:2408.00925 [cs.CR] https://arxiv.org/abs/2408.00925

2024 arXiv
[29]

Haowei Wang, Rupeng Zhang, Junjie Wang, Mingyang Li, Yuekai Huang, Dandan Wang, and Qing Wang. 2025. Joint- GCG: Unified Gradient-Based Poisoning Attacks on Retrieval-Augmented Generation Systems. arXiv:2506.06151 [cs.CR] https://arxiv.org/abs/2506.06151

2025
[30]

Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: how does LLM safety training fail?. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA,...

2023
[31]

Zico Kolter

Joshua Nathaniel Williams, Avi Schwarzschild, Yutong He, and J. Zico Kolter. 2025. Prompt Recovery for Image Generation Models: A Comparative Study of Discrete Optimizers. arXiv:2408.06502 [cs.CV] https://arxiv.org/abs/2408. 06502

2025 arXiv
[32]

Zhen Xu, Xiang Li, Hao Zhang, and Fan Wu. 2024. Faster GCG: Efficient Gradient-Based Adversarial Suffix Search. arXiv preprint arXiv:2410.15362 (2024)

2024 arXiv
[33]

Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. 2023. FinGPT: Open-Source Financial Large Language Models. arXiv:2306.06031 [q-fin.ST] https://arxiv.org/abs/2306.06031 , Vol. 1, No. 1, Article . Publication date: September 2025. The Resurgence of GCG Adversarial Attacks ...

2023
[34]

Xiaoxue Yang, Bozhidar Stevanoski, Matthieu Meeus, and Yves-Alexandre de Montjoye. 2025. Alignment Under Pressure: The Case for Informed Adversaries When Evaluating LLM Defenses. (2025). arXiv:2505.15738 [cs.CR] https://arxiv.org/abs/2505.15738

2025
[35]

Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2024. GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts. arXiv:2309.10253 [cs.AI] https://arxiv.org/abs/2309.10253

2024 arXiv
[36]

Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2023. GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher. arXiv:2308.06463 [cs.CL]

2023 arXiv
[37]

Jiahao Zhang, Zilong Wang, Ruofan Wang, Xingjun Ma, and Yu-Gang Jiang. 2024. EnJa: Ensemble Jailbreak on Large Language Models. arXiv:2408.03603 [cs.CR] https://arxiv.org/abs/2408.03603

2024 arXiv
[38]

Kai Zheng, Mingyu Liu, Wei Huang, and Fan Zhang. 2024. Momentum GCG: Momentum-Enhanced Gradient-Based Jailbreaking of Large Language Models

2024
[39]

Sicheng Zhu, Brandon Amos, Yuandong Tian, Chuan Guo, and Ivan Evtimov. 2024. AdvPrefix: An Objective for Nuanced LLM Jailbreaks. arXiv:2412.10321 [cs.LG] https://arxiv.org/abs/2412.10321

2024
[40]

Please prepare 100 harmful prompts that ask for harmful scripts. Here is an example: Write a script that can exploit vulnerabilities in a software or operating system

Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.15043 [cs.CL] https://arxiv.org/abs/2307. 15043 A Harmful coding-generation prompts The harmful ...

2023 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Mohamed Ahmed, Mohamed Abdelmouty, Mingyu Kim, Gunvanth Kandula, Alex Park, and James C. Davis. 2025. Advancing Jailbreak Strategies: A Hybrid Approach to Exploiting LLM Vulnerabilities and Bypassing Modern Defenses. arXiv:2506.21972 [cs.CL] https://arxiv.org/abs/2506.21972 , Vol. 1, No. 1, Article . Publication date: September 2025. The Resurgence of GCG...

work page Pith review arXiv 2025

[2] [2]

Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2025. Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks. In The Thirteenth International Conference on Learning Representations . https: //openreview.net/forum?id=hXA8wqRdyV

work page 2025

[3] [3]

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

arXiv 2023

[4] [4]

Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

arXiv 2022

[5] [5]

Matan Ben-Tov, Mor Geva, and Mahmood Sharif. 2025. Universal Jailbreak Suffixes Are Strong Attention Hijackers. arXiv:2506.12880 [cs.CR] https://arxiv.org/abs/2506.12880

arXiv 2025

[6] [6]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

2020

[7] [7]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

arXiv 2021

[8] [8]

Yifan Chen, Yixuan Wang, Hao Huang, Yu Tang, and Lei Zhou. 2024. AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulation

work page 2024