Pith. sign in

REVIEW 3 major objections 6 minor 60 references

JavelinGuard: Low-Cost Transformer Architectures for LLM Security

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Small encoder-only classifiers can match or beat gpt-4o at jailbreak and prompt-injection detection while running 20–40× faster, with the multi-task Raudra architecture delivering the strongest overall accuracy across nine benchmarks.

desk verdict A useful engineering survey and a new hard-negative benchmark, but the headline accuracy numbers do not establish generalization because the training corpus overlaps the evaluation sets and no decontamination is described. read the letter →

arxiv 2506.07330 v1 pith:HUS3LFLW submitted 2025-06-09 cs.LG cs.AIcs.CR

Yash Datta , Sharath Rajasekar This is my paper
classification cs.LGcs.AIcs.CR
keywords jailbreakdetectionpromptinjectionguardrailmodelsmulti-taskclassificationModernBERTfocallossJavelinBenchtransformerclassifiers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

JavelinGuard asks whether compact transformer encoders can provide production-grade detection of jailbreak and prompt-injection attempts without the cost of large decoder-only LLMs. The paper builds five progressively more complex encoder-based classifiers and claims that the multi-task architecture, Raudra, consistently delivers the strongest accuracy across nine adversarial benchmarks while running on CPU hardware in tens of milliseconds. On the paper's own JavelinBench, Raudra4.2 reaches 96.2 percent accuracy, above the 91.3 percent reported for gpt-4o, at roughly one-twentieth the latency. If these numbers hold, small classifiers could replace large-model moderation in high-throughput guardrail pipelines.

What carries the argument

The load-bearing mechanism is Raudra's multi-task head. It keeps one shared ModernBERT encoder but learns a separate attention-weighted token-pooling distribution for each label—'jailbreak' and 'prompt_injection'—using the [CLS] hidden state as a global query over the sequence. Each label is then routed through its own residual feed-forward classifier, and training uses focal loss with per-task weights to emphasize hard examples. This design is what lets a single small model output both attack classes at once while keeping false positives low on benign prompts.

What would settle it

Run every evaluation again after deleting from the training corpus all examples that also appear in garak, the InjecGuard/NotInject sets, or the other nine benchmarks, then compare Raudra4.2's five-set mean accuracy; a material drop below the reported ~92.8 percent would settle the generalization question.

Watch

Extended reading notes

Core claim

The paper's central claim is that a moderately sized encoder model can match or beat frontier decoder-only LLMs at jailbreak and prompt-injection detection when the classification head is engineered for multi-task output. In particular, the Raudra framework—a shared ModernBERT-large encoder with per-label attention-weighted pooling, per-task residual heads, and focal loss—is reported to achieve about 92.8 percent mean accuracy across the consolidated five-benchmark set, near-perfect F1 on the garak positives, and 96.2 percent accuracy on the newly introduced JavelinBench. The authors also claim that Raudra and Mahendra run 20 to 40 times faster than gpt-4o while outperforming it on JavelinBench, and that the suite differentiates jailbreak from prompt injection without labeling toxic content as an attack. The paper further claims that among modern BERT-family backbones, ModernBERT-large offers the best production balance of accuracy, latency, and parameter count.

Load-bearing premise

The central result assumes the nine benchmark sets were truly held out from training, since the paper trains on some of the same public sources it later evaluates on and never describes a decontamination or overlap-filtering step.

Editorial extensions

If this is right

  • If the reported accuracy holds, production guardrails can use 400M-parameter CPU-only classifiers instead of calling large LLMs for every moderation decision, saving cost and latency.
  • The multi-task output means one model can separately flag jailbreak and prompt-injection attempts, which existing guardrails typically merge into a single 'malicious' class.
  • JavelinBench provides a shared hard-negative test set that stresses false positives, so future guardrail models have a common benchmark for borderline cases.
  • The hybrid Vaishnava and Ashwina heads offer feature-importance explanations at some cost in recall, giving practitioners a documented accuracy-interpretability tradeoff.
  • The architecture sweep across ModernBERT, NeoBERT, and EuroBERT supports choosing ModernBERT-large as a default backbone for similar detection deployments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors leave implicit is that the per-label multi-task head would generalize to more than two classes, so one model could carry a third head for policy-violating toxic content instead of a separate classifier.
  • Because JavelinBench deliberately concentrates on hard negatives and borderline cases, it could serve as an adversarial training source for other guardrails, not just as an evaluation set.
  • The reported latency comparison uses short, unoptimized prompts on CPU; on GPU with batched inference the absolute gap would shrink, but the parameter-count and cost advantage should remain.
  • The paper evaluates several benchmarks whose source datasets also appear in its training corpus, so a decontaminated re-run would make the generalization claim directly testable rather than assumed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents JavelinGuard, a suite of five encoder-based classifier architectures (Sharanga, Mahendra, Vaishnava, Ashwina, Raudra) for detecting jailbreak and prompt-injection in LLM interactions. The authors benchmark these models on nine datasets, introduce a new JavelinBench dataset, and compare against gpt-4o, gpt-4.1-mini, and several open-source guardrails. They claim that the multi-task Raudra model “consistently achieves the strongest performance,” with up to about 92.8% mean accuracy on a five-set consolidated metric and 96.2% accuracy on JavelinBench, while running 20–40× faster than gpt-4o on CPU hardware.

Significance. If the claimed results were trustworthy, the paper would be a valuable practical contribution: compact, CPU-friendly classifiers that can replace large LLM-based guardrails at a fraction of the cost. The systematic exploration of five architectures, the latency measurements, and the introduction of JavelinBench are strengths, as is the direct comparison against gpt-4o and several open-source baselines. However, the central empirical claim is not supported as written because the training corpus appears to overlap with the evaluation benchmarks and no decontamination is described. The near-perfect F1 scores on Garak and the low false-positive rates on the NotInject series may reflect memorization rather than generalization. The headline results therefore cannot be taken as evidence of held-out performance.

major comments (3)
  1. [Section 4.3 / Section 4.2] The training corpus described in Section 4.3 is aggregated “from sources including InjecGuard ... Garak ...” and totals 120,021 samples. Section 4.2 evaluates on the NotInject series, which is explicitly stated to come “from the InjecGuard paper,” and on Garak. The paper describes no decontamination, overlap filtering, or deduplication between training and evaluation data; the only filtering mentioned in Section 4.3 removes trivial duplicates within synthetic data. If any benchmark examples appear in the training set, the near-perfect Garak F1 scores in Tables 4 and 5, the low FPRs on the NotInject sets, and the consolidated 5-set accuracy of 0.928 for Raudra4.2 are partly memorization scores rather than measurements of generalization. The authors must either demonstrate that no benchmark sample was in the training set or re-run the evaluation on decontaminated held-out data.
  2. [Section 4.2] All benchmarks are reduced to a binary malicious/benign label: “Because most benchmarks do not distinguish between these two attack types, we merge them into a single ‘malicious’ class for classification metrics.” However, the paper’s central claim and the Raudra architecture (Section 3.5) emphasize differentiating jailbreak from prompt injection. The reported metrics never evaluate that differentiation; any claim about multi-task capability is therefore unsupported. The authors should provide per-class (jailbreak vs. prompt_injection) metrics on benchmarks that carry fine-grained labels, or explicitly restrict the claims.
  3. [Section 5 (Tables 4 and 5)] All results are reported as single numbers without error bars, multiple seeds, or significance tests. Given that several score differences between the proposed models are small (e.g., Raudra4.2 vs. Mahendra1.1 on JavelinBench: 0.962 vs. 0.945; 5-set accuracy 0.928 vs. 0.905), it is not possible to know whether these differences are stable or within run-to-run noise. The claim that Raudra “consistently achieves the strongest performance” requires variance information and, ideally, a paired test across seeds.
minor comments (6)
  1. [Section 1] The introduction states that “Section 7 concludes with future research directions,” but in the manuscript Section 6 is the conclusion and Section 7 is Future Work; this cross-reference is off by one.
  2. [Table 4] Prompt-Guard-86M is reported with an accuracy of 0.041 on ToxicChat, which is far worse than random and inconsistent with its performance on other benchmarks; this value should be checked for a typo or evaluation error.
  3. [Abstract / Table 2] The abstract says the models are “as few as approximately 400M parameters,” but Sharanga8 uses NeoBERT with 250M parameters (Table 2); the phrasing should be adjusted to reflect the parameter range actually explored.
  4. [References] The reference list contains duplicate entries for SALAD-Bench: Li et al. 2024a and Li et al. 2024b refer to the same paper; one should be removed.
  5. [Section 3.2] Figure 2 is referenced in the text but does not appear in the manuscript; ensure the figure is included and clearly described in the final version.
  6. [Section 5] The discussion of the “lost in the middle” problem proposes segmentation as a mitigation but provides no experimental evaluation; this should be labeled as future work or supported with experiments.

Circularity Check

1 steps flagged · score 6.0 of 10

Training and evaluation sets overlap by source (InjecGuard and Garak), so the headline benchmark scores do not establish held-out generalization.

  1. fitted input called prediction [Section 4.2 Benchmarks; Section 4.3 Training Data Preparation; Tables 3-5]
    "Specifically, we aggregate samples from sources including InjecGuard [Li and Liu, 2024], jailbreak_llms [Shen et al., 2024], Garak [Derczynski et al., 2024], ReneLLM [Ding et al., 2023], PAIR [Chao et al., 2023], ALERT [Tedeschi et al., 2024], BoN [Hughes et al., 2024], SALAD [Li et al., 2024a] etc. ... NotInject series: Three sets from the InjecGuard paper [Li and Liu, 2024] to test for over-refusal and short explicit overrides. ... Garak: Prompts derived from TAP and DAN probes in the garak tool [Derczynski et al., 2024]."

    Section 4.3 says training data were aggregated 'from sources including InjecGuard ... Garak', while Section 4.2 evaluates on Garak and on NotInject sets 'from the InjecGuard paper'; no decontamination, overlap filtering, or held-out split is described, and the only stated filter removes 'trivial duplicates' from synthetic data. Therefore the near-perfect garak F1 (1.000), the NotInject FPR values, and the 0.928 five-set accuracy can reflect memorization of training examples rather than generalization. The conclusion that 'Raudra consistently achieves the strongest performance' and the comparison against gpt-4o rest on these contaminated benchmarks, so the reported predictions are partially fitted inputs rather than independent outcomes.

full rationale

The only load-bearing circular step I can pin to the paper's own text is the train/evaluation source overlap: Garak and InjecGuard/NotInject appear both in the training corpus (Section 4.3) and in the benchmark suite (Section 4.2), with no decontamination described. This makes the central empirical claim partially circular: the model is being scored on material from the same pools it was trained on. I do not see other circular patterns: the architectures are described as concrete engineering configurations (ModernBERT fine-tuning, attention-weighted pooling, hybrid RF/XGBoost heads, multi-task focal loss), the paper does not invoke any uniqueness theorem or prior self-citation as a load-bearing premise, and JavelinBench, while author-created, is not stated to be a training source. The issue is thus not a formal derivation that equals its inputs, but an evaluation loop that prevents the reported accuracy, F1, and FPR numbers from establishing held-out generalization.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

No new physical or theoretical entities are introduced. The free parameters are training hyperparameters chosen by hand or grid search, and the axioms are the implicit empirical assumptions behind the benchmark evaluations. The most consequential assumption is train/eval disjointness, which the paper neither states nor verifies.

free parameters (4)
  • Focal loss gamma = 2.0 (Mahendra), 3.0 (Raudra)
    Chosen via grid search for Raudra and fixed for Mahendra; it controls how strongly hard examples are penalized and affects all reported accuracy and F1 numbers.
  • Per-task class weights = {1.5, 1.0}
    Selected by validation grid search for Raudra; these weights balance the jailbreak and prompt-injection loss terms and directly influence the multi-task results.
  • Learning rate = 2e-5 to 3e-5 depending on model
    A standard fine-tuning hyperparameter, but the accuracy results depend on this choice and the paper does not report sensitivity.
  • Random Forest and XGBoost hyperparameters = RF: n_estimators=100, max_depth=20; XGB: n_estimators=100, max_depth=6, lr=0.1
    Chosen by hand for the hybrid models; the reported ensemble performance depends on these settings.
assumptions (3)
  • domain assumption Benchmark labels are correct
    All accuracy, FPR, and F1 numbers inherit the labels of the nine external datasets and JavelinBench. The paper itself argues that WildGuardTest mislabels benign prompts, so label noise is a known risk.
  • domain assumption Training and evaluation sets are disjoint
    Section 4.3 lists Garak and InjecGuard in the training corpus while Section 4.2 evaluates Garak and the NotInject series. No decontamination or overlap filtering is described, so this assumption is load-bearing and may be false.
  • domain assumption JavelinBench annotations are reliable and representative
    The new benchmark is constructed by the authors with manual and synthetic annotation, but no annotation protocol, inter-annotator agreement, or release is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of JavelinGuard: Low-Cost Transformer Architectures for LLM Security." pith.science (2026). https://pith.science/paper/HUS3LFLW

@misc{pith2026250607330,
  author       = {Pith},
  title        = {Pith review of: JavelinGuard: Low-Cost Transformer Architectures for LLM Security},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HUS3LFLW}},
  note         = {Machine review of arXiv:2506.07330}
}
read the original abstract

We present JavelinGuard, a suite of low-cost, high-performance model architectures designed for detecting malicious intent in Large Language Model (LLM) interactions, optimized specifically for production deployment. Recent advances in transformer architectures, including compact BERT(Devlin et al. 2019) variants (e.g., ModernBERT (Warner et al. 2024)), allow us to build highly accurate classifiers with as few as approximately 400M parameters that achieve rapid inference speeds even on standard CPU hardware. We systematically explore five progressively sophisticated transformer-based architectures: Sharanga (baseline transformer classifier), Mahendra (enhanced attention-weighted pooling with deeper heads), Vaishnava and Ashwina (hybrid neural ensemble architectures), and Raudra (an advanced multi-task framework with specialized loss functions). Our models are rigorously benchmarked across nine diverse adversarial datasets, including popular sets like the NotInject series, BIPIA, Garak, ImprovedLLM, ToxicChat, WildGuard, and our newly introduced JavelinBench, specifically crafted to test generalization on challenging borderline and hard-negative cases. Additionally, we compare our architectures against leading open-source guardrail models as well as large decoder-only LLMs such as gpt-4o, demonstrating superior cost-performance trade-offs in terms of accuracy, and latency. Our findings reveal that while Raudra's multi-task design offers the most robust performance overall, each architecture presents unique trade-offs in speed, interpretability, and resource requirements, guiding practitioners in selecting the optimal balance of complexity and efficiency for real-world LLM security applications.

Figures

Figures reproduced from arXiv: 2506.07330 by the authors.

Figure 1
Figure 1. An example academic prompt from the WildGuardTest dataset [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Diagram of our model architectures, illustrating layers, neural modules, and attention. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 10 canonical work pages

  1. [1]

    Are you still on track!? catching llm task drift with activations, 2024

    Sahar Abdelnabi, Aideen Fay, Giovanni Cherubin, Ahmed Salem, Mario Fritz, and Andrew Paverd. Are you still on track!? catching llm task drift with activations, 2024. URL https://arxiv.org/abs/2406.00799

    arXiv 2024

  2. [2]

    Jailbreaking leading safety-aligned llms with simple adaptive attacks, 2024

    Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety-aligned llms with simple adaptive attacks, 2024. URL https://arxiv.org/abs/2404.02151

    arXiv 2024

  3. [3]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. URL https://arxiv.org/abs/2004.05150

    arXiv 2020

  4. [4]

    Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M. Alves, André Martins, Ayoub Hammal, Caio Corro, Céline Hudelot, Emmanuel Malherbe, Etienne Malaboeuf, Fanny Jourdan, Gabriel Hautreux, João Alves, Kevin El-Haddad, Manuel Faysse, Maxime Peyrard, Nuno M. Guerreiro, Patrick Fernandes, Ricardo Rei, and Pierre Colombo. Eurobert: Scaling multilingual enco...

    arXiv 2025

  5. [5]

    Neobert: A next-generation bert, 2025

    Lola Le Breton, Quentin Fournier, Mariam El Mezouar, and Sarath Chandar. Neobert: A next-generation bert, 2025. URL https://arxiv.org/abs/2502.19587

    arXiv 2025

  6. [6]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries, 2023

    2023

  7. [7]

    Defense against prompt injection attack by leveraging attack techniques, 2025

    Yulin Chen, Haoran Li, Zihao Zheng, Yangqiu Song, Dekai Wu, and Bryan Hooi. Defense against prompt injection attack by leveraging attack techniques, 2025. URL https://arxiv.org/abs/2411.00459

    arXiv 2025

  8. [8]

    Rethinking attention with performers, 2022

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers, 2022. URL https://arxiv.org/abs/2009.14794

    arXiv 2022

Show all 60 references
  1. [9]

    Deepset prompt injection guardrail, 2024

    Deepset. Deepset prompt injection guardrail, 2024. URL https://huggingface.co/deepset/deberta-v3-base-injection

    2024

  2. [10]

    Duoguard: A two-player rl-driven framework for multilingual llm guardrails, 2025

    Yihe Deng, Yu Yang, Junkai Zhang, Wei Wang, and Bo Li. Duoguard: A two-player rl-driven framework for multilingual llm guardrails, 2025. URL https://arxiv.org/abs/2502.05163

    2025

  3. [11]

    garak: A framework for security probing large language models, 2024

    Leon Derczynski, Erick Galinkin, Jeffrey Martin, Subho Majumdar, and Nanna Inie. garak: A framework for security probing large language models, 2024. URL https://arxiv.org/abs/2406.11036

    2024 arXiv

  4. [12]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.org/abs/1810.04805

    2019 arXiv

  5. [13]

    A wolf in sheep's clothing: Generalized nested jailbreak prompts can fool large language models easily, 2023

    Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang. A wolf in sheep's clothing: Generalized nested jailbreak prompts can fool large language models easily, 2023

    2023

  6. [14]

    Moussa Koulako Bala Doumbouya, Ananjan Nandi, Gabriel Poesia, Davide Ghilardi, Anna Goldie, Federico Bianchi, Dan Jurafsky, and Christopher D. Manning. h4rm3l: A dynamic benchmark of composable jailbreak attacks for llm safety assessment, 2024. URL https://arxiv.org/abs/2408.04811

    2024 arXiv

  7. [15]

    Fmops prompt injection guardrail, 2024

    fmops. Fmops prompt injection guardrail, 2024. URL https://huggingface.co/fmops/distilbert-prompt-injection

    2024

  8. [16]

    Improved large language model jailbreak detection via pretrained embeddings, 2024

    Erick Galinkin and Martin Sablotny. Improved large language model jailbreak detection via pretrained embeddings, 2024. URL https://arxiv.org/abs/2412.01547

    2024 arXiv

  9. [17]

    Aegis: Online adaptive ai content safety moderation with ensemble of llm experts, 2024

    Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien. Aegis: Online adaptive ai content safety moderation with ensemble of llm experts, 2024. URL https://arxiv.org/abs/2404.05993

    2024 arXiv

  10. [18]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  11. [19]

    Mamba: Linear-time sequence modeling with selective state spaces, 2024

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL https://arxiv.org/abs/2312.00752

    2024 arXiv

  12. [20]

    Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms, 2024

    Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms, 2024. URL https://arxiv.org/abs/2406.18495

    2024 arXiv

  13. [21]

    Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, 2023

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, 2023. URL https://arxiv.org/abs/2111.09543

    2023 arXiv

  14. [22]

    Catastrophic jailbreak of open-source llms via exploiting generation, 2023

    Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source llms via exploiting generation, 2023. URL https://arxiv.org/abs/2310.06987

    2023 arXiv

  15. [23]

    Semantic-guided prompt organization for universal goal hijacking against llms, 2024

    Yihao Huang, Chong Wang, Xiaojun Jia, Qing Guo, Felix Juefei-Xu, Jian Zhang, Geguang Pu, and Yang Liu. Semantic-guided prompt organization for universal goal hijacking against llms, 2024. URL https://arxiv.org/abs/2405.14189

    2024 arXiv

  16. [24]

    Best-of-n jailbreaking, 2024

    John Hughes, Sara Price, Aengus Lynch, Rylan Schaeffer, Fazl Barez, Sanmi Koyejo, Henry Sleight, Erik Jones, Ethan Perez, and Mrinank Sharma. Best-of-n jailbreaking, 2024. URL https://arxiv.org/abs/2412.03556

    2024 arXiv

  17. [25]

    Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023. URL https://arxiv.org/abs/2312.06674

    2023 arXiv

  18. [26]

    Jinhwa Kim, Ali Derakhshan, and Ian G. Harris. Robust safety classifier for large language models: Adversarial prompt shield, 2023. URL https://arxiv.org/abs/2311.00172

    2023 arXiv

  19. [27]

    Lakera-guard, 2024

    lakera.ai. Lakera-guard, 2024. lakera.ai. 2024. Lakera-guard

    2024

  20. [28]

    Injecguard: Benchmarking and mitigating over-defense in prompt injection guardrail models, 2024

    Hao Li and Xiaogeng Liu. Injecguard: Benchmarking and mitigating over-defense in prompt injection guardrail models, 2024. URL https://arxiv.org/abs/2410.22770

    2024 arXiv

  21. [30]

    Salad-bench: A hierarchical and comprehensive safety benchmark for large language models, 2024 b

    Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models, 2024 b . URL https://arxiv.org/abs/2402.05044

    2024 arXiv

  22. [31]

    Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation, 2023

    Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation, 2023

    2023

  23. [32]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023. URL https://arxiv.org/abs/2307.03172

    2023 arXiv

  24. [33]

    Automatic and universal prompt injection attacks against large language models, 2024 a

    Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. Automatic and universal prompt injection attacks against large language models, 2024 a . URL https://arxiv.org/abs/2403.04957

    2024 arXiv

  25. [34]

    Formalizing and benchmarking prompt injection attacks and defenses, 2024 b

    Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. Formalizing and benchmarking prompt injection attacks and defenses, 2024 b . URL https://arxiv.org/abs/2310.12815

    2024

  26. [35]

    Promptguard prompt injection guardrail, 2024

    Meta. Promptguard prompt injection guardrail, 2024. URL https://www.llama.com/docs/model-cards-and-prompt-formats/prompt-guard

    2024

  27. [36]

    Daly, Michael Hind, Werner Geyer, Ambrish Rawat, Kush R

    Inkit Padhi, Manish Nagireddy, Giandomenico Cornacchia, Subhajit Chaudhury, Tejaswini Pedapati, Pierre Dognin, Keerthiram Murugesan, Erik Miehling, Martín Santillán Cooper, Kieran Fraser, Giulio Zizzo, Muhammad Zaid Hameed, Mark Purcell, Michael Desmond, Qian Pan, Zahra Ashkto...

    2024 arXiv

  28. [37]

    Ignore previous prompt: Attack techniques for language models, 2022

    Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models, 2022. URL https://arxiv.org/abs/2211.09527

    2022 arXiv

  29. [38]

    Fine-tuned deberta-v3-base for prompt injection detection, 2024 a

    ProtectAI.com. Fine-tuned deberta-v3-base for prompt injection detection, 2024 a . URL https://huggingface.co/ProtectAI/deberta-v3-base-prompt-injection-v2

    2024

  30. [39]

    Fine-tuned deberta-v3-base for prompt injection detection, 2024 b

    ProtectAI.com. Fine-tuned deberta-v3-base for prompt injection detection, 2024 b . URL https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2

    2024

  31. [40]

    Fine-tuning aligned language models compromises safety, even when users do not intend to!, 2023

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to!, 2023. URL https://arxiv.org/abs/2310.03693

    2023 arXiv

  32. [41]

    Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails, 2023

    Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails, 2023. URL https://arxiv.org/abs/2310.10501

    2023 arXiv

  33. [42]

    Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu

    Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu. Rainbow teaming: Open-ended generation of diverse adversarial prompts, 20...

    2024 arXiv

  34. [43]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020. URL https://arxiv.org/abs/1910.01108

    2020 arXiv

  35. [44]

    ``Do Anything Now'': Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. ``Do Anything Now'': Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models . In ACM SIGSAC Conference on Computer and Communications Security (CCS) . ACM, 2024

    2024

  36. [45]

    Optimization-based prompt injection attack to llm-as-a-judge, 2024

    Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. Optimization-based prompt injection attack to llm-as-a-judge, 2024. URL https://arxiv.org/abs/2403.17710

    2024 arXiv

  37. [46]

    A party of foundation models, 2024

    Qwen Team. A party of foundation models, 2024. URL https://qwenlm.github.io/blog/qwen2.5

    2024

  38. [47]

    Alert: A comprehensive benchmark for assessing large language models' safety through red teaming, 2024

    Simone Tedeschi, Felix Friedrich, Patrick Schramowski, Kristian Kersting, Roberto Navigli, Huu Nguyen, and Bo Li. Alert: A comprehensive benchmark for assessing large language models' safety through red teaming, 2024

    2024

  39. [48]

    Tensor trust: Interpretable prompt injection attacks from an online game, 2023

    Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, Alan Ritter, and Stuart Russell. Tensor trust: Interpretable prompt injection attacks from an online game, 2023. URL https:/...

    2023 arXiv

  40. [49]

    Safeguarding crowdsourcing surveys from chatgpt with prompt injection, 2023

    Chaofan Wang, Samuel Kernan Freire, Mo Zhang, Jing Wei, Jorge Goncalves, Vassilis Kostakos, Zhanna Sarsenbayeva, Christina Schneegass, Alessandro Bozzon, and Evangelos Niforatos. Safeguarding crowdsourcing surveys from chatgpt with prompt injection, 2023. URL https://arxiv.org...

    2023 arXiv

  41. [50]

    Defending llms against jailbreaking attacks via backtranslation, 2024

    Yihan Wang, Zhouxing Shi, Andrew Bai, and Cho-Jui Hsieh. Defending llms against jailbreaking attacks via backtranslation, 2024. URL https://arxiv.org/abs/2402.16459

    2024 arXiv

  42. [51]

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 2024

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional e...

    2024 arXiv

  43. [52]

    Jailbroken: How does llm safety training fail?, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail?, 2023. URL https://arxiv.org/abs/2307.02483

    2023 arXiv

  44. [53]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  45. [54]

    Fundamental limitations of alignment in large language models, 2024

    Yotam Wolf, Noam Wies, Oshri Avnery, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models, 2024. URL https://arxiv.org/abs/2304.11082

    2024 arXiv

  46. [55]

    Benchmarking and defending against indirect prompt injection attacks on large language models

    Jingwei Yi, Yueqi Xie, Bin Zhu, Keegan Hines, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Benchmarking and defending against indirect prompt injection attacks on large language models. arXiv preprint arXiv:2312.14197, 2023

    2023 arXiv

  47. [56]

    LLM-Fuzzer : Scaling assessment of large language model jailbreaks

    Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. LLM-Fuzzer : Scaling assessment of large language model jailbreaks. In 33rd USENIX Security Symposium (USENIX Security 24), pages 4657--4674, Philadelphia, PA, August 2024. USENIX Association. ISBN 978-1-939133-44-1. URL https:...

    2024

  48. [57]

    Shieldgemma: Generative ai content moderation based on gemma, 2024

    Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, Olivia Sturman, and Oscar Wahltinez. Shieldgemma: Generative ai content moderation based on gemma, 2024. URL https://arxiv.or...

    2024 arXiv

  49. [58]

    Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents, 2024

    Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents, 2024. URL https://arxiv.org/abs/2403.02691

    2024 arXiv

  50. [59]

    Defending jailbreak prompts via in-context adversarial game, 2025

    Yujun Zhou, Yufei Han, Haomin Zhuang, Kehan Guo, Zhenwen Liang, Hongyan Bao, and Xiangliang Zhang. Defending jailbreak prompts via in-context adversarial game, 2025. URL https://arxiv.org/abs/2402.13148

    2025 arXiv

  51. [60]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. URL https://arxiv.org/abs/2307.15043

    2023 arXiv

  52. [61]

    Improving alignment and robustness with circuit breakers, 2024

    Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers, 2024. URL https://arxiv.org/abs/2406.04313

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.