Pith. sign in

REVIEW 4 major objections 5 minor 39 references

Direct prompt injection attacks can be made transferable across LLMs without querying the target model, using an energy-based model trained on a surrogate model's activations to guide MCMC prompt optimization.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 21:56 UTC pith:DPIJIZRQ

load-bearing objection Novel activation-guided MCMC for prompt injection, but the energy-to-success link is under-supported and the gains over seeds are modest. the 4 major comments →

arxiv 2509.07617 v1 pith:DPIJIZRQ submitted 2025-09-09 cs.AI

Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling

Minghui Li , Hao Zhang , Yechao Zhang , Wei Wan , Shengshan Hu , pei Xiaobing , Jing Wang This is my paper
classification cs.AI
keywords direct prompt injectionblack-box attackenergy-based modelMCMC samplingtransferabilityLLM securityactivation interpretabilityadversarial prompts
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that direct prompt injection attacks—malicious instructions that override an application's system prompt—can be made transferable across different large language models without any query to the target model. It proposes training an energy-based model on the internal activations of a surrogate LLM to score how attack-like a candidate prompt is, then using token-level MCMC sampling with BERT proposals to search for lower-energy prompts. The reported result is a 49.6% average attack success rate across five open- and closed-source LLMs, a 34.6% improvement over human-crafted prompts, and 36.6% ASR on tasks never seen during training. If true, an attacker with access only to one surrogate model can produce natural-sounding prompts that carry over to other models, including a hardened commercial model where manual prompts scored 0%.

Core claim

The central claim is that activation patterns inside a surrogate LLM carry enough information about prompt-injection success to guide generation of transferable black-box attacks. The authors build a binary classifier (success/failure) on layer-25 activations of Qwen2.5-7B-Instruct and treat its logits as an energy function; lower energy means more attack-like. They then run a Metropolis-style MCMC sampler over tokens, proposing replacements with BERT and accepting or rejecting by the energy ratio corrected by the MLM's proposal probability. Across five victim models and seven tasks, the optimized prompts beat white-box (GCG), gray-box (AutoDAN), and black-box (PromptFuzz) baselines in trans

What carries the argument

Energy-based model derived from a binary activation classifier: for activations x, E(x) = -log(exp(f_0(x)) + exp(f_1(x))). Trained with cross-entropy on 16,000 labeled prompt/activation pairs from a surrogate model, it defines a landscape over prompts. The MCMC acceptance rule p(X'|X) = min(1, exp(-E(X')) p_MLM(X_i|X_{-i}) / (exp(-E(X)) p_MLM(X'_i|X_{-i})) uses this energy to steer token replacements toward lower-energy (more attack-like) prompts while BERT supplies natural proposals.

Load-bearing premise

The energy score, computed from how confidently a classifier labels a prompt's activations as an attack, is assumed to rank prompts by how likely they are to actually take over a victim model—rather than merely matching the training distribution of the surrogate.

What would settle it

On a held-out set of seed prompts, record the energy score and the attack success rate on several victim models, then check the Pearson correlation. If prompts with lower energy do not consistently produce higher ASR across victims, or if randomly generated BERT replacements without energy guidance match the success rate, the claim that activation-guided energy drives transferability is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A black-box attacker can craft transferable injection prompts using only one surrogate model and no queries to the target model.
  • Prompt injection can succeed on commercial models hardened against known manual attack patterns.
  • Generated attack prompts generalize to new tasks beyond those used to train the energy model.
  • Low perplexity in the generated prompts means perplexity-based filtering is not a sufficient defense.
  • The activation-energy link offers an interpretable signal for auditing or detecting injection attempts.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper does not isolate whether the energy landscape itself is the active ingredient; an ablation replacing the EBM with a random-walk or perplexity-only objective would clarify this, and that experiment is not reported.
  • The same activation-energy signal could be repurposed defensively: monitor whether a prompt's activations drift into low-energy regions associated with successful injections.
  • The template decomposition into prefix, infix, and suffix suggests attack components are modular; a testable extension is whether energy-guided recombination transfers to multimodal LLMs or tool-use agents.
  • Because the EBM is trained on a single surrogate, its guidance may become brittle when victim models use different alignment methods; testing on more diverse model families would map the limits of the transferability claim.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a black-box Direct Prompt Injection (DPI) attack framework. It builds a template dataset from Tensor Trust by decoupling attack prompts into prefix/infix/suffix, collects hidden-state activations from a surrogate LLM (Qwen2.5-7B-Instruct or Llama-3.1-8B-Instruct), trains a binary classifier as an energy-based model (EBM), and then uses token-level MCMC sampling with BERT proposals to optimize seed prompts. The acceptance rule combines EBM energies with MLM probabilities, and the lowest-energy sample in the chain is selected. The method is evaluated on five LLMs and seven CYBERSECEVAL tasks, reporting ASR 49.6% vs. 44.6% for Initial Prompts, ASR-T 44.1% vs. 41.3% for the Qwen2.5 surrogate, and 36.6% ASR on unseen tasks. The paper claims superior cross-model and cross-task transferability and provides an interpretability analysis linking activations to attack success.

Significance. If the central mechanism is sound, the paper makes a useful contribution: a black-box DPI attack that does not query the victim, uses surrogate activations rather than gradients, produces relatively natural prompts, and transfers to a closed model (GPT-4o-mini) where manual prompts fail. The use of an energy-based model over activations is a novel idea in this threat model. However, the key energy-to-attack-success link is not established, and the main results show only modest and uneven gains over the Initial Prompts baseline. The absence of ablations, confidence intervals, and a detailed evaluation protocol makes the central claim conditional on further evidence.

major comments (4)
  1. [Eq. (5), Sec. 4.2.3, Table 1] Eθ(x) = -log(exp(f0(x)) + exp(f1(x))) is minimized whenever the binary classifier is confident in either class, not only when the attack succeeds. The paper asserts that lower energy corresponds to more effective attack prompts (Sec. 4.2.3) and uses this in Eq. (8) and Algorithm 1. Table 1 contains a direct counterexample: for Ours(Llama3.1), the ASR on Llama3.1 is 36.80%, below the Initial Prompts ASR of 44.40% on the same model, even though the optimized prompts were selected by lowest energy from Llama3.1 activations. The authors must provide held-out, per-sample energy-ASR correlations and an energy-free MCMC baseline; otherwise the guidance mechanism is unsupported.
  2. [Figure 5 / Sec. 4.2.3] The claimed validation of the energy-ASR relationship (-0.979 Pearson) uses mean energy per ASR bin, is in-sample, has no error bars, and no held-out check. Binned correlations can be misleading. Since Algorithm 1 selects the lowest-energy sample on the same surrogate used to train the EBM, the apparent alignment is partly a selection artifact. Please report per-sample correlations on held-out tasks (e.g., Tasks 6-7) and on victims that are not surrogates (Qwen2, Llama3, GPT-4o-mini for the Qwen2.5 row), with confidence intervals.
  3. [Table 1 / Sec. 4.1 (Metrics)] The transfer improvements over Initial Prompts are modest and uneven. For Ours(Qwen2.5), ASR-T is 44.10 vs. 41.30; per-model gains vs. Initial are +4.2 (Qwen2), +5.0 (Llama3), 0.0 (GPT-4o-mini), and -0.4 (Llama3.1). No confidence intervals or significance tests are provided, and there is no ablation that removes energy guidance (e.g., random acceptance or BERT-only random search). Without such baselines, the reported gains cannot be attributed to activation-guided MCMC, and the central claim of superior cross-model transferability is not fully supported.
  4. [Appendix B / Sec. 4.1] The manuscript states that LLM-based evaluation implementation details are provided in Appendix B, but Appendix B only lists tasks and judge functions; it does not describe the LLM judge prompt, decoding parameters, or how judgments are aggregated. Per-task sample counts and standard deviations are also missing. This prevents assessment of whether the reported differences are statistically meaningful and hampers reproducibility.
minor comments (5)
  1. [Algorithm 1, line 4] Typo: 'Replacing the i-th token X(t)i of X(t)' should be 'Replace the i-th token'.
  2. [Eq. (7)] The term log Z(θ) appears inside the sum over i; if Z is the global partition over all x,y, it should be outside the sum. Clarify the normalization convention.
  3. [Eq. (8)] The notation p(X′|X) denotes the acceptance probability, not a transition probability. Consider using α(X′|X) to avoid confusion with the proposal distribution.
  4. [Table 1] Specify exactly which models are excluded from ASR-T for each row, since both Qwen2.5 and Llama3.1 are used as surrogates in different rows and the definition of 'white-box model' is row-dependent.
  5. [Figure 4] The energy score decreases over iterations, but without a corresponding ASR trace this does not demonstrate improved attack effectiveness. Add an ASR-vs-iteration plot or clarify that energy decrease alone is not evidence of success.

Circularity Check

0 steps flagged

No significant circularity: the EBM-guided MCMC pipeline is a surrogate-model optimization with independently measured transfer results.

full rationale

The paper's derivation chain is: collect activations and success/failure labels from a surrogate model; train a binary classifier and reinterpret its logits as an EBM energy (Eqs. 1-7); run MCMC whose acceptance ratio uses that energy (Eq. 8) to propose prompt edits; measure ASR on the surrogate and on other models. None of these steps defines the predicted quantity in terms of the fitted quantity by construction. The energy is a function of classifier logits, not of attack-success labels; the claim that lower energy tracks higher ASR is an empirical correlation checked in Fig. 5, not an identity. Final ASR values are obtained by actually executing attacks, not read off the EBM. The cross-model transfer claim is measured on victim models other than the surrogate (ASR-T) and on unseen tasks, so it is not forced by the training objective. The EBM and MCMC machinery are cited from independent prior work, and there are no load-bearing self-citations. Concerns about whether the energy landscape truly aligns with attack success are validity/robustness issues rather than circularity. Therefore no circular step can be exhibited.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The method introduces no new physical or conceptual entities; it combines existing models (classifier, BERT, MCMC) with a new objective. The main unproven ingredients are the transferability of activations and the validity of the energy as an attack-quality measure.

free parameters (5)
  • Activation layer index for EBM = 25
    Selected as the layer with lowest validation loss (Fig. 10); a design choice tuned on the training data.
  • Number of top infixes after filtering = 35
    Infixes ranked by ASR on the first training task (Sec 3.2); the 35 threshold is arbitrary and chosen based on surrogate performance.
  • EBM MLP hidden dimensions = 1024, 256
    Architecture hyperparameters in Appendix C; not theoretically motivated.
  • Number of training samples and tasks = 20,000 / 7 tasks
    4,000 templates x 5 training tasks; task count and template count are choices.
  • MCMC proposal model = BERT
    The MLM (BERT) is used for token replacement; the specific BERT checkpoint is not specified.
axioms (4)
  • standard math A binary classifier can be interpreted as an energy-based model (Grathwohl et al., 2020).
    Used in Section 3.3 to derive the EBM from classifier logits; this is a known result.
  • domain assumption Surrogate model activations transfer across different LLM architectures.
    The entire method relies on the EBM trained on surrogate activations to guide prompts that work on victim models; no theoretical guarantee.
  • domain assumption The MCMC sampler with BERT proposals will efficiently explore the space of high-threat prompts.
    The algorithm uses random token replacement; convergence to a useful distribution is assumed (Algorithm 1).
  • domain assumption Success/failure labels from the surrogate model are reliable proxies for victim model behavior.
    The EBM is trained on surrogate labels; transfer success depends on this proxy relationship.

pith-pipeline@v1.3.0-alltime-deepseek · 13212 in / 14595 out tokens · 146453 ms · 2026-08-04T21:56:11.660318+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling." pith.science (2026). https://pith.science/paper/DPIJIZRQ

@misc{pith2026250907617,
  author       = {Pith},
  title        = {Pith review of: Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DPIJIZRQ}},
  note         = {Machine review of arXiv:2509.07617}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Direct Prompt Injection (DPI) attacks pose a critical security threat to Large Language Models (LLMs) due to their low barrier of execution and high potential damage. To address the impracticality of existing white-box/gray-box methods and the poor transferability of black-box methods, we propose an activations-guided prompt injection attack framework. We first construct an Energy-based Model (EBM) using activations from a surrogate model to evaluate the quality of adversarial prompts. Guided by the trained EBM, we employ the token-level Markov Chain Monte Carlo (MCMC) sampling to adaptively optimize adversarial prompts, thereby enabling gradient-free black-box attacks. Experimental results demonstrate our superior cross-model transferability, achieving 49.6% attack success rate (ASR) across five mainstream LLMs and 34.6% improvement over human-crafted prompts, and maintaining 36.6% ASR on unseen task scenarios. Interpretability analysis reveals a correlation between activations and attack effectiveness, highlighting the critical role of semantic patterns in transferable vulnerability exploitation.

Figures

Figures reproduced from arXiv: 2509.07617 by Hao Zhang, Jing Wang, Minghui Li, Pei Xiaobing, Shengshan Hu, Wei Wan, Yechao Zhang.

Figure 1
Figure 1. Figure 1: Toy examples of direct prompt injection attack [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Pipeline of our activations-guided MCMC sampling. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Message structure. mary instruction, while the user input includes ad￾versarial prompt consisting of the prefix, infix, and suffix. The message structure is shown in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The energy scores over the iterative process. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The energy scores of samples. baselines. First, our framework significantly outper￾forms white-box GCG-Inject, gray-box AutoDAN￾GA-Inject and black-box PromptFuzz across all evaluated models. For instance, when transferring attacks to Llama3.1, traditional white-box meth￾ods like GCG-Inject suffer catastrophic ASR drops from 58.6% to 5.55%, whereas our approach main￾tains robust performance (71.6% → 44.4%)… view at source ↗
Figure 6
Figure 6. Figure 6: The visualization of activations. ness levels meet human acceptability thresholds (PPL=127.68), potential improvements could in￾volve fine-tuning the proposal model or introducing additional constraints to enhance attack strength at the expense of naturalness. Second, our ap￾proach does not address defense based on text clas￾sifiers (Li et al., 2024). Future research should investigate bypassing detection … view at source ↗
Figure 8
Figure 8. Figure 8: Judge function for task 3 I’m trying to take the prompt injection at￾tack template out of the data. These injected prompts are constructed in 3 parts: prefix for model escaping, middle for misleading model and including attack payload, and suffix for simulating the user inputs. I would like you to extract the three parts of the following injection attack and output them in JSON format. Text: {Manual Sample… view at source ↗
Figure 9
Figure 9. Figure 9: The decoupling prompts. C EBM Details We implement an EBM using a two-hidden-layer MLP with dimensions 1024 and 256, employing ReLU (Agarap, 2018) activation functions. The model is optimized using AdamW (Loshchilov and Hutter, 2019) with a learning rate of 0.0003, incor￾porating warmup and cosine annealing scheduling. Training utilizes a cross-entropy loss function with class-weighted adjustments based on… view at source ↗
Figure 7
Figure 7. Figure 7: Judge function for task 2 We present representative evaluation examples of the first task in [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 10
Figure 10. Figure 10: EBM training process [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 16 canonical work pages

  1. [1]

    Sahar Abdelnabi, Aideen Fay, Giovanni Cherubin, Ahmed Salem, and Mario Fritz. 2024. https://doi.org/10.48550/ARXIV.2406.00799 Are you still on track!? catching LLM task drift with activations . CoRR, abs/2406.00799

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.00799 2024

  2. [2]

    Abien Fred Agarap. 2018. https://arxiv.org/abs/1803.08375 Deep learning using rectified linear units (relu) . CoRR, abs/1803.08375

    Pith/arXiv arXiv 2018

  3. [3]

    AgentGPT. 2023. https://github.com/reworkd/AgentGPT [link]

    work page 2023

  4. [4]

    AutoGPT. 2023. https://github.com/Significant-Gravitas/AutoGPT [link]

    work page 2023

  5. [5]

    Sizhe Chen, Julien Piet, Chawin Sitawarin, and David A. Wagner. 2024. https://doi.org/10.48550/ARXIV.2402.06363 StruQ : Defending against prompt injection with structured queries . CoRR, abs/2402.06363

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.06363 2024

  6. [6]

    Coeeter. 2023. https://github.com/Coeeter/emailgpt [link]

    work page 2023

  7. [7]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/V1/N19-1423 BERT: pre-training of deep bidirectional transformers for language understanding . In Proceeding of the 17th Conference of the North American Chapter of the Association for Computational Linguistics (NAACL'19), pages 4171--4186. Association for C...

    doi:10.18653/v1/n19-1423 2019

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and Ahmad Al-Dahle . 2024. https://doi.org/10.48550/ARXIV.2407.21783 The llama 3 herd of models . CoRR, abs/2407.21783

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.21783 2024

  9. [9]

    Freysa. 2024. https://www.freysa.ai/act-i [link]

    work page 2024

  10. [10]

    Futurism. 2023. https://futurism.com/the-byte/car-dealership-ai [link]

    work page 2023

  11. [11]

    Leo Gao, Tom Dupr \' e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. https://doi.org/10.48550/ARXIV.2406.04093 Scaling and evaluating sparse autoencoders . CoRR, abs/2406.04093

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.04093 2024

  12. [12]

    GitHub. 2023. https://github.com/features/copilot [link]

    work page 2023

  13. [13]

    Will Grathwohl, Kuan-Chieh Wang, J \"o rn-Henrik Jacobsen, David Duvenaud, and Mohammad Norouzi. 2020. https://doi.org/10.48550/arXiv.1912.03263 Your classifier is secretly an energy based model and you should treat it like one . In Proceeding of the 8th International Conference on Learning Representations (ICLR'20). OpenReview.net

    Pith/arXiv arXiv doi:10.48550/arxiv.1912.03263 2020

  14. [14]

    Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. 2024. COLD-attack : Jailbreaking LLMs with stealthiness and controllability. In Proceeding of the 41st International Conference on Machine Learning (ICML'24), pages 16974--17002. PMLR

    work page 2024

  15. [15]

    Huggingface. 2023. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html [link]

    work page 2023

  16. [16]

    Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. 1977. Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1):S63--S63

    work page 1977

  17. [17]

    Aounon Kumar and Himabindu Lakkaraju. 2024. https://doi.org/10.48550/ARXIV.2404.07981 Manipulating large language models to increase product visibility . CoRR, abs/2404.07981

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.07981 2024

  18. [18]

    Hao Li, Xiaogeng Liu, and Chaowei Xiao. 2024. https://doi.org/10.48550/ARXIV.2410.22770 Injecguard: Benchmarking and mitigating over-defense in prompt injection guardrail models . CoRR, abs/2410.22770

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.22770 2024

  19. [19]

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024. https://doi.org/10.48550/arXiv.2310.04451 AutoDAN : Generating stealthy jailbreak prompts on aligned large language models . In Proceeding of the 12st International Conference on Learning Representations (ICLR'24). OpenReview.net

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.04451 2024

  20. [20]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In Proceeding of the 7th International Conference on Learning Representations (ICLR'19). OpenReview.net

    work page 2019

  21. [21]

    Fatemehsadat Mireshghallah, Kartik Goyal, and Taylor Berg-Kirkpatrick . 2022. https://doi.org/10.18653/v1/2022.acl-long.31 Mix and match: Learning-free controllable text generationusing energy language models . In Proceeding of the 60st Annual Meeting of the Association for Computational Linguistics (ACL'22), pages 401--415, Dublin, Ireland. Association f...

    work page doi:10.18653/v1/2022.acl-long.31 2022

  22. [22]

    OpenAI. 2023. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html Language models can explain neurons in language models

    work page 2023

  23. [23]

    OWASP. 2024. https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025 [link]

    work page 2024

  24. [24]

    Dario Pasquini, Martin Strohmeier, and Carmela Troncoso. 2024. https://doi.org/10.1145/3689932.3694764 Neural exec: Learning (and learning from) execution triggers for prompt injection attacks . In Proceeding of the 2024 Workshop on Artificial Intelligence and Security (AISec '24), pages 89--100, Salt Lake City UT USA. ACM

    work page arXiv 2024

  25. [25]

    Yao Qiang, Xiangyu Zhou, Saleh Zare Zade, Mohammad Amin Roshani, Douglas Zytko, and Dongxiao Zhu. 2024. https://doi.org/10.48550/ARXIV.2402.13459 Learning to poison large language models during instruction tuning . CoRR, abs/2402.13459

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.13459 2024

  26. [26]

    Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/3e25d1aff47964c8409fd5c8dc0438d7-Abstract-Conference.html COLD decoding: Energy-based constrained text generation with langevin dynamics . In Proceedings of the 35th International Conference on Neural Information Processing Systems (NeurIPS'22)

    work page 2022

  27. [27]

    Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, and Tom Goldstein. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/c2a8060fd22744b38177d9e428a052e0-Abstract-Conference.html On the exploitability of instruction tuning . In Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS'23)

    work page 2023

  28. [28]

    Yang Song and Diederik P. Kingma. 2021. https://arxiv.org/abs/2101.03288 How to train your energy-based models . CoRR, abs/2101.03288

    Pith/arXiv arXiv 2021

  29. [29]

    Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, and Luke Bailey. 2024. https://doi.org/10.48550/arXiv.2311.01011 Tensor trust: Interpretable prompt injection attacks from an online game . In Proceeding of the 21st International Conference on Learning Representations (ICLR'24). OpenReview.net

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.01011 2024

  30. [30]

    Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, and James Crnkovich. 2024. https://doi.org/10.48550/ARXIV.2408.01605 CYBERSECEVAL 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models . CoRR, abs/2408.01605

    Pith/arXiv arXiv doi:10.48550/arxiv.2408.01605 2024

  31. [31]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei...

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.10671 2024

  32. [32]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu X...

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.15115 2024

  33. [33]

    Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2023. https://doi.org/10.48550/ARXIV.2309.10253 GPTFUZZER : Red teaming large language models with auto-generated jailbreak prompts . CoRR, abs/2309.10253

    Pith/arXiv arXiv doi:10.48550/arxiv.2309.10253 2023

  34. [34]

    Jiahao Yu, Yangguang Shao, Hanwen Miao, Junzheng Shi, and Xinyu Xing. 2024. https://doi.org/10.48550/ARXIV.2409.14729 PROMPTFUZZ: harnessing fuzzing techniques for robust testing of prompt injection in llms . CoRR, abs/2409.14729

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.14729 2024

  35. [35]

    Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. https://doi.org/10.18653/V1/2024.FINDINGS-ACL.624 Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents . In Proceeding of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL'24), pages 10471--10506. Association for Computa...

    doi:10.18653/v1/2024.findings-acl.624 2024

  36. [36]

    Andy Zou, Long Phan, Sarah Chen, James Campbell, and Phillip Guo. 2023 a . https://doi.org/10.48550/ARXIV.2310.01405 Representation engineering: A top-down approach to AI transparency . CoRR, abs/2310.01405

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.01405 2023

  37. [37]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023 b . https://doi.org/10.48550/ARXIV.2307.15043 Universal and transferable adversarial attacks on aligned language models . CoRR, abs/2307.15043

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.15043 2023

  38. [38]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  39. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...