Pith. sign in

REVIEW 4 major objections 7 minor 2 cited by

Universal and Context-Independent Triggers for Precise Control of LLM Outputs

T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that one fixed pair of trigger tokens, optimized on general instruction data, can make Qwen-2 and Llama-3.1 emit an attacker-chosen output in a majority of novel contexts.

desk verdict A genuinely new attack capability with solid evidence within a narrow distribution; the 'universal' claim goes beyond what the experiments show. read the letter →

arxiv 2411.14738 v1 pith:BW75UQDK submitted 2024-11-22 cs.CL cs.AIcs.CR

Jiashuo Liang , Guancheng Li , Yang Yu This is my paper · ORCID
classification cs.CLcs.AIcs.CR
keywords promptinjectionuniversaladversarialtriggercontext-independentgradient-basedattackdiscreteoptimizationGCGLLMoutputcontrolwhite-box
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a fixed, model-specific trigger consisting of two short token sequences placed around a desired output text can make an instruction-tuned LLM ignore the surrounding prompt and reproduce the attacker-chosen text. The central finding is that this trigger can be found once with gradient-based discrete search over a diverse instruction dataset, rather than re-optimized for each task or prompt context. This matters because it turns prompt injection from a fragile, expert-crafted intervention into a reusable attack that any holder of the trigger can deploy. On held-out instructions, the trigger achieves exact-match success rates of 67.8 percent on Qwen-2 7B-Instruct and 54.1 percent on Llama-3.1 8B-Instruct, with approximate-prefix-match rates of 75.0 and 70.6 percent.

What carries the argument

The central object is a two-part adversarial trigger with the desired output text sandwiched between the two token sequences. The trigger is optimized with the Greedy Coordinate Gradient algorithm, which uses token-level gradients to propose replacement tokens, combined with a candidate queue, multi-coordinate updates, incremental search over output prefixes, and a loss that emphasizes the least-likely output token. The averaged loss over many instruction contexts is what teaches the trigger to act like a compact command, roughly equivalent in intent to “ignore other instructions and output the payload,” that works across varied system prompts and user inputs.

What would settle it

Train the trigger on one set of task families and then test it on held-out prompts from completely different domains, languages, or output schemas, for example prompts requiring strict XML or YAML output rather than JSON. If exact-match success on those unfamiliar inputs falls to the roughly 12 to 16 percent level of the handcrafted trigger, the universality and context-independence claims are refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that gradient-based discrete optimization can discover universal, context-independent adversarial triggers that give precise control over LLM outputs. Formally, the trigger is a pair of token sequences $(X_{\text{trigger1}}, X_{\text{trigger2}})$ trained to minimize the negative log-likelihood of a desired adversarial output $Y_{\text{adv}}$ when inserted as $X_{\text{trigger1}} \oplus \text{Encode}(Y_{\text{adv}}) \oplus X_{\text{trigger2}}$ into the user-controllable part of the prompt, averaged over an adversarial dataset built from public instruction data. Once trained, the same two trigger segments are wrapped around any target text at inference time. The paper reports that this recipe makes the model reproduce the target exactly in 67.8 percent of held-out cases on Qwen-2 and 54.1 percent on Llama-3.1, and approximately in 75.0 and 70.6 percent of cases. It also reports partial transfer to larger or newer models in the same family, and that success degrades only mildly with the injection position inside the user input.

Load-bearing premise

The load-bearing premise is that the OIG and Alpaca GPT-4 instruction distribution used to train the trigger is representative enough of the general prompt contexts and target texts over which the trigger is called universal, so a trigger that works on those examples will keep working on arbitrary, unseen prompts.

Editorial extensions

If this is right

  • A single captured trigger becomes a reusable attack primitive: an attacker who does not know the application's system prompt can still inject arbitrary content by wrapping it in the two trigger segments.
  • Because half of the training outputs were converted to JSON, the triggered outputs can be structured data, meaning downstream workflow or agent components that parse model output are in scope for the attack.
  • Transfer results show the same trigger can partially attack larger or newer models in the same family, reducing the need to retrain the trigger for each model version.
  • The method is white-box and model-specific, so the direct practical threat is concentrated on open-weight models or settings where gradient access is available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's universality is demonstrated on a single distribution of instruction-following prompts; a stronger version of the claim, not proven here, would be robustness to completely out-of-distribution contexts such as non-English instructions or specialized domain languages. We would not bet on that stronger claim from the present evidence.
  • The design of the attack suggests a testable ablation the paper does not run: removing the suffix trigger or the prefix trigger separately would reveal which half carries the attack, given that prefix-match failures show the model continuing to copy text after the payload.
  • The reported linear relationship between injection location and success rate implies a defensive lever: systems that constrain where user-supplied text can appear, or that isolate late-arriving content in long messages, could reduce the attack's effectiveness even without detecting the trigger.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes a gradient-based method for training fixed trigger tokens that, when placed around a payload in an injected prompt, aim to make an LLM output the payload exactly. The trigger is trained via GCG on an adversarial dataset built from OIG and Alpaca GPT-4 instruction data, with random split positions and adversarial outputs generated by three strategies. On held-out test sets, EM/PM/APM reach 67.8/71.6/75.0% for Qwen-2 7B-Instruct and 54.1/63.0/70.6% for Llama-3.1 8B-Instruct, with some transferability to same-series models. The paper claims universality, context independence, and precise output control.

Significance. If the claims were fully supported, this would be a notable contribution to LLM security, since it shows that fixed triggers can be trained to control outputs across a variety of instructions. The experimental design is clear and the handcrafted baseline is meaningful. The paper is honest in some places, notably the hedge to 'a degree of context independence' in Section 4.2. However, the evaluation does not establish the advertised 'universal and context-independent' properties: the test distribution shares the same dataset origin and data-generation pipeline as the training distribution, no out-of-distribution or cross-template evaluation is provided, and the main tables give only point estimates without uncertainty measures. The strong claims in the title and abstract exceed what the evidence supports.

major comments (4)
  1. [Sec. 3.2 and Sec. 4.2] The claim that the discovered triggers are 'universal and context-independent' is not supported by the evaluation design. The test set is built from the same two base datasets (OIG and Alpaca GPT-4) and the same AdvGen strategies as the training set, so the reported ASRs measure interpolation within that distribution rather than generalization to arbitrary target outputs or diverse prompt contexts. The context-independence analysis in Section 4.2 varies only the injection location within the user input (Figure 2); system prompts, templates, languages, and output formats are all drawn from the same pool. The paper itself only claims 'a degree of context independence' in Section 4.2, yet the abstract and conclusion assert the stronger property. An out-of-distribution evaluation (e.g., unseen task types, different prompt templates, non-English inputs, or other data-generation strategies) or a substantial softening of the claims is needed.
  2. [Table 2] The 'precise control' claim is undermined by the actual numbers. Exact Match is 67.8% for Qwen-2 and 54.1% for Llama-3.1, and the APM metric counts outputs with Rouge-L F1 > 0.9 as successful, which allows noticeable non-exact outputs. The introduction defines 'Precise Output' as 'capable of manipulating LLM inputs to yield any specified output with high accuracy', but the reported EM rates are far from that. The authors should either report a stricter metric as the primary success criterion or temper the 'precise' terminology in the title and abstract.
  3. [Sec. 4.1 and Tables 2-3] The main results are reported as point estimates without confidence intervals or significance tests for the key comparisons. For a test set of 800 cases, binomial standard errors are roughly 1–2 percentage points, but the paper does not provide them, so the reader cannot judge whether differences such as the JSON-vs-text gap for Llama-3.1 in Table 2 are meaningful. The transferability results in Table 3 similarly lack uncertainty measures. Including standard errors or confidence intervals would make the comparisons more reliable.
  4. [Sec. 4.2 and Sec. 6] The only baseline is a handcrafted trigger; there is no comparison with task-specific gradient-based attacks or with prior controllable-output attacks (e.g., Geiping et al. [8]). The paper motivates the work by the high cost of per-task optimization, but no experiments show that the universal trigger is more efficient than optimizing a trigger for each task. Without such comparisons, the claimed efficiency advantage is not demonstrated.
minor comments (7)
  1. [Sec. 4.1] The hyperparameters are not specified: K, B, C, Q, m1, m2, the number of epochs, the size of the initial trigger set, and the computational budget. These details are needed to reproduce the method.
  2. [Sec. 2.2, Eq. (2)] The second line of Equation (2) has an extra closing parenthesis: '... ⊕ y1 ⊕ · · · ⊕ yi−1))' should be '... ⊕ y1 ⊕ · · · ⊕ yi−1)'.
  3. [Sec. 4.2] The phrase 'significant better' should be 'significantly better'.
  4. [Abstract] The abstract contains a grammatical error: 'highlighting the potential for adversaries to taking over' should be 'to take over'.
  5. [Sec. 4.4 and Table 4] The text states that unmatched words are marked with a pink background, but this formatting is not visible in the text-only version; please describe the differences in words as well.
  6. [Sec. 4.2 and Figures 2-3] The description 'a group of nearby test cases' for the scatter points is vague; please specify the binning or smoothing procedure used for these figures.
  7. [Sec. 1 and Table 1] The actual trigger tokens are concealed for security reasons; the paper should state whether the trigger or the code will be made available (e.g., upon request) to enable independent verification.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the trigger is trained and evaluated on a held-out split of the same adversarial data-generation pipeline, which limits external generality but does not make the evaluation logically circular.

full rationale

The derivation chain is not circular. The trigger tokens are optimized by minimizing the negative log-likelihood loss (Eq. 2) summed over the training adversarial dataset (Eq. 5), with the adversarial outputs Yadv specified externally by AdvGen from OIG/Alpaca GPT-4; neither the loss nor the optimization procedure defines success in terms of the test statistic. The reported EM/PM/APM are measured on a separate 800-example test set not used for gradient updates, and the trigger is also transferred to unseen same-series models (Section 4.3), which is an out-of-distribution check beyond the training distribution. The paper contains no self-citations that carry a load-bearing assumption: the cited PLeak, GCG, and related works are independent prior methods. The main weakness is that the test set is constructed by the same pipeline (same base datasets, same splitting scheme, same AdvGen strategies, plus JSON conversion) as the training set, so the headline 'universal and context-independent' claim is supported only within this distribution; the paper itself hedges with 'a degree of context independence' and 'a degree of robustness' in Section 4.2. That is a breadth-of-evaluation limitation and a correctness-risk concern, not a logical circularity, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim requires no invented theoretical entities; the trigger and payload are string constructs within an existing optimization pipeline. The main design choices are unreported hyperparameters and a manual APM threshold, and the load-bearing assumption is that instruction-dataset diversity represents the real world.

free parameters (3)
  • Trigger segment lengths m1 and m2 = not reported
    The lengths of the two trigger segments are hand-chosen hyperparameters that bound how much adversarial context can be encoded, but the paper never states their values.
  • GCG training hyperparameters K, B, C, Q, epochs = not reported
    Top-K alternatives, candidate expansion, number of coordinates, queue size, and number of epochs all affect trigger quality; Algorithm 1 names them but gives no numerical settings.
  • Approximate Prefix Match Rouge-L threshold = 0.9
    The paper manually chooses 0.9 as the F1 threshold for APM; changing this threshold changes all reported attack success rates.
assumptions (3)
  • domain assumption The diversity of OIG and Alpaca GPT-4 instruction data is representative of real-world prompt contexts and target outputs.
    The universality claim rests on training on these datasets translating to unseen contexts and arbitrary payloads, introduced in Sections 3.2 and 4.1.
  • domain assumption White-box access to model gradients is available to the attacker.
    The method requires full gradient access; the threat analysis assumes attacker control of open-source model weights, as stated in Sections 2.2 and 6.
  • standard math Greedy Coordinate Gradient's linear embedding-space gradient approximation is adequate for optimizing trigger tokens.
    The optimization relies on the standard GCG approximation between embedding gradients and token replacements, as described in Section 3.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Universal and Context-Independent Triggers for Precise Control of LLM Outputs." pith.science (2026). https://pith.science/paper/BW75UQDK

@misc{pith2026241114738,
  author       = {Pith},
  title        = {Pith review of: Universal and Context-Independent Triggers for Precise Control of LLM Outputs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BW75UQDK}},
  note         = {Machine review of arXiv:2411.14738}
}
read the original abstract

Large language models (LLMs) have been widely adopted in applications such as automated content generation and even critical decision-making systems. However, the risk of prompt injection allows for potential manipulation of LLM outputs. While numerous attack methods have been documented, achieving full control over these outputs remains challenging, often requiring experienced attackers to make multiple attempts and depending heavily on the prompt context. Recent advancements in gradient-based white-box attack techniques have shown promise in tasks like jailbreaks and system prompt leaks. Our research generalizes gradient-based attacks to find a trigger that is (1) Universal: effective irrespective of the target output; (2) Context-Independent: robust across diverse prompt contexts; and (3) Precise Output: capable of manipulating LLM inputs to yield any specified output with high accuracy. We propose a novel method to efficiently discover such triggers and assess the effectiveness of the proposed attack. Furthermore, we discuss the substantial threats posed by such attacks to LLM-based applications, highlighting the potential for adversaries to taking over the decisions and actions made by AI agents.

Figures

Figures reproduced from arXiv: 2411.14738 by the authors.

Figure 1
Figure 1. Distribution of the Rouge-L F1 score. The left one shows the score over all test data. The right one excludes [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Distribution and linear regression of average attack success rate in respect to inject locations. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Distribution and linear regression of average attack success rate in respect to ratio of adversarial tokens. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Know Your Agent: Reconnaissance-Driven Pentesting of AI Agents

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Reconnaissance-driven black-box pentesting, built as KYA, reports large indirect-prompt-injection success gains on AgentDojo and InjecAgent over self-reproduced iterative baselines.

  2. Evaluating Language Models as Synthetic Data Generators

    cs.CL 2024-12 conditional novelty 6.0 of 10

    AgoraBench shows that an LM's ability to solve problems does not predict its ability to generate useful synthetic training data.

Reference graph

Works this paper leans on

40 extracted references · 11 canonical work pages · cited by 2 Pith papers

  1. [8]

    Coercing LLMs to do and reveal (almost) anything

    Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, and Tom Goldstein. Coercing LLMs to do and reveal (almost) anything. arXiv preprint arXiv:2402.14020, 2024

    arXiv 2024

  2. [1]

    GPT-4 Technical Report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [2]

    Kavosh Asadi and Michael L. Littman. An Alternative Softmax Operator for Reinforcement Learning. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research, pages 243–252. PMLR, 06–11 Aug 2017

    work page 2017

  4. [3]

    Jailbreaking Black Box Large Language Models in Twenty Queries

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking Black Box Large Language Models in Twenty Queries. arXiv preprint arXiv:2310.08419, 2023. 13

    arXiv 2023

  5. [4]

    The Llama 3 Herd of Models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  6. [5]

    HotFlip: White-Box Adversarial Examples for Text Classification

    Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. HotFlip: White-Box Adversarial Examples for Text Classification. arXiv preprint arXiv:1712.06751, 2017

    arXiv 2017

  7. [6]

    Imprompter: Tricking LLM Agents into Improper Tool Use

    Xiaohan Fu, Shuheng Li, Zihan Wang, Yihao Liu, Rajesh K Gupta, Taylor Berg-Kirkpatrick, and Earlence Fernandes. Imprompter: Tricking LLM Agents into Improper Tool Use. arXiv preprint arXiv:2410.14923, 2024

    arXiv 2024

  8. [7]

    Retrieval-Augmented Generation for Large Language Models: A Survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv preprint arXiv:2312.10997, 2023

    arXiv 2023

Show all 40 references
  1. [9]

    Gradient-based Adversarial Attacks against Text Transformers

    Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based Adversarial Attacks against Text Transformers. arXiv preprint arXiv:2104.13733, 2021

    2021 arXiv

  2. [10]

    Query-based Adversarial Prompt Generation

    Jonathan Hayase, Ema Borevkovic, Nicholas Carlini, Florian Tramèr, and Milad Nasr. Query-based Adversarial Prompt Generation. arXiv preprint arXiv:2402.12329, 2024

    2024 arXiv

  3. [11]

    PLeak: Prompt Leaking Attacks against Large Language Model Applications

    Bo Hui, Haolin Yuan, Neil Gong, Philippe Burlina, and Yinzhi Cao. PLeak: Prompt Leaking Attacks against Large Language Model Applications. arXiv preprint arXiv:2405.06823, 2024

    2024 arXiv

  4. [12]

    A Survey on Large Language Models for Code Generation

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A Survey on Large Language Models for Code Generation. arXiv preprint arXiv:2406.00515, 2024

    2024 arXiv

  5. [13]

    Automatically Auditing Large Language Models via Discrete Optimization

    Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. Automatically Auditing Large Language Models via Discrete Optimization. In International Conference on Machine Learning , pages 15307–15329. PMLR, 2023

    2023

  6. [14]

    Making a SOTA Adversarial Attack on LLMs 38x Faster.https://blog.haizelabs.com/posts/ acg/, 2024

    Haize Labs. Making a SOTA Adversarial Attack on LLMs 38x Faster.https://blog.haizelabs.com/posts/ acg/, 2024

    2024

  7. [15]

    Rouge: A Package for Automatic Evaluation of Summaries

    Chin-Yew Lin. Rouge: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74–81, 2004

    2004

  8. [16]

    Lost in the Middle: How Language Models Use Long Contexts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the Middle: How Language Models Use Long Contexts. arXiv preprint arXiv:2307.03172, 2023

    2023 arXiv

  9. [17]

    AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs

    Xiaogeng Liu, Peiran Li, Edward Suh, Yevgeniy V orobeychik, Zhuoqing Mao, Somesh Jha, Patrick McDaniel, Huan Sun, Bo Li, and Chaowei Xiao. AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs. arXiv preprint arXiv:2410.05295, 2024

    arXiv

  10. [18]

    Biases in Large Language Models: Origins, Inventory, and Discussion

    Roberto Navigli, Simone Conia, and Björn Ross. Biases in Large Language Models: Origins, Inventory, and Discussion. J. Data and Information Quality, 15(2), Jun 2023

    2023

  11. [19]

    The OIG Dataset

    Huu Nguyen, Sameer Suri, Ken Tsui, Shahules786, and Christoph Schuhmann. The OIG Dataset. https: //laion.ai/blog/oig-dataset/, 2023

    2023

  12. [20]

    AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs

    Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs. arXiv preprint arXiv:2404.16873, 2024

    2024 arXiv

  13. [21]

    Instruction Tuning with GPT-4

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction Tuning with GPT-4. arXiv preprint arXiv:2304.03277, 2023

    2023 arXiv

  14. [22]

    Toran Bruce Richards. AutoGPT. https://github.com/Significant-Gravitas/AutoGPT, 2023

    2023

  15. [23]

    Alexander Robey, Zachary Ravichandran, Vijay Kumar, Hamed Hassani, and George J. Pappas. Jailbreaking LLM-Controlled Robots. arXiv preprint arXiv:2410.13691, 2024

    2024 arXiv

  16. [24]

    Mathematical discoveries from program search with large language models

    Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995...

    2024

  17. [25]

    Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts

    Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, et al. Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts. arXiv preprint arXiv:2402.16822, 2024

    2024 arXiv

  18. [26]

    The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts

    Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi. The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts. arXiv preprint arXiv:2401.13136, 2024. 14

    2024 arXiv

  19. [27]

    AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts

    Taylor Shin, Yasaman Razeghi, Robert L Logan IV , Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. arXiv preprint arXiv:2010.15980, 2020

    2010 arXiv

  20. [28]

    StructuredRAG: JSON Response Formatting with Large Language Models

    Connor Shorten, Charles Pierse, Thomas Benjamin Smith, Erika Cardenas, Akanksha Sharma, John Trengrove, and Bob van Luijt. StructuredRAG: JSON Response Formatting with Large Language Models. arXiv preprint arXiv:2408.11061, 2024

    2024 arXiv

  21. [29]

    Ben Thompson, and Michael Sklar

    Zygimantas Straznickas, T. Ben Thompson, and Michael Sklar. Takeaways from the NeurIPS 2023 Trojan Detection Competition. https://confirmlabs.org/posts/TDC2023.html, 2024

    2023

  22. [30]

    Role- Break: Character Hallucination as a Jailbreak Attack in Role-Playing Systems

    Yihong Tang, Bo Wang, Xu Wang, Dongming Zhao, Jing Liu, Jijun Zhang, Ruifang He, and Yuexian Hou. Role- Break: Character Hallucination as a Jailbreak Attack in Role-Playing Systems. arXiv preprint arXiv:2409.16727, 2024

    2024 arXiv

  23. [31]

    Solving olympiad geometry without human demonstrations

    Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476–482, 2024

    2024

  24. [32]

    Adversarial Attacks on LLMs

    Lilian Weng. Adversarial Attacks on LLMs. https://lilianweng.github.io/posts/ 2023-10-25-adv-attack-llm/ , Oct 2023

    2023

  25. [33]

    AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv preprint arXiv:2308.08155, 2023

    2023 arXiv

  26. [34]

    An LLM can Fool Itself: A Prompt-Based Adversarial Attack

    Xilie Xu, Keyi Kong, Ning Liu, Lizhen Cui, Di Wang, Jingfeng Zhang, and Mohan Kankanhalli. An LLM can Fool Itself: A Prompt-Based Adversarial Attack. In The Twelfth International Conference on Learning Representations, 2024

    2024

  27. [35]

    Qwen2 Technical Report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 Technical Report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  28. [36]

    GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts

    Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts. arXiv preprint arXiv:2309.10253, 2023

    2023 arXiv

  29. [37]

    Uncertainty is Fragile: Manipulating Uncertainty in Large Language Models

    Qingcheng Zeng, Mingyu Jin, Qinkai Yu, Zhenting Wang, Wenyue Hua, Zihao Zhou, Guangyan Sun, Yanda Meng, Shiqing Ma, Qifan Wang, et al. Uncertainty is Fragile: Manipulating Uncertainty in Large Language Models. arXiv preprint arXiv:2407.11282, 2024

    2024 arXiv

  30. [38]

    How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs

    Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs. arXiv preprint arXiv:2401.06373, 2024

    2024 arXiv

  31. [39]

    AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models

    Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models. In First Conference on Language Modeling, 2024

    2024

  32. [40]

    Universal and Transferable Adversarial Attacks on Aligned Language Models

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv preprint arXiv:2307.15043, 2023. 15

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.