Evaluating the efficacy of LLM Safety Solutions : The Palit Benchmark Dataset

Daniel Woods; Sayon Palit

REVIEW 3 major objections 6 minor 83 references

Evaluating the efficacy of LLM Safety Solutions : The Palit Benchmark Dataset

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper builds a 500-prompt benchmark of malicious and benign prompts, evaluates seven LLM safety filters and a ChatGPT-3.5-Turbo baseline, and reports that the baseline's false-positive rate (0.438–0.859) makes it unusable as a safety…

desk verdict Useful first cut at benchmarking commercial LLM security tools, but the headline rankings are not load-bearing on their own: the benchmark isn't released and the results are prompt-source-dependent. read the letter →

arxiv 2505.13028 v2 pith:OC3IC6WI submitted 2025-05-19 cs.CR cs.AIcs.CL

Sayon Palit , Daniel Woods This is my paper · ORCID

classification cs.CRcs.AIcs.CL

keywords LLMsecuritypromptinjectionjailbreakbenchmarkdatasetfalsepositiveratesafetytoolevaluationLLM-integratedapplicationsclassification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish whether bolt-on LLM security tools—filters that screen prompts before they reach a model—actually work and whether they are usable in production. To answer this, the authors assemble the Palit benchmark: 500 prompts spanning four LLM-integrated application contexts and four sources of attack generation, and evaluate seven commercial or open-source filters plus a ChatGPT-3.5-Turbo baseline. The central finding is that the raw baseline model is unsuitable as a safety filter because its false-positive rate ranges from 0.438 to 0.859, while the two best tools, Lakera Guard and ProtectAI LLM Guard, achieve high precision with far fewer false positives. A sympathetic reader would take away that LLM safety is not yet solved by any single tool and that context matters: detection changes when system prompts are added, and performance differs sharply across attack-generation sources.

What carries the argument

The load-bearing object is the benchmark itself: a collection of 500 prompts built from hand-written attacks and the outputs of three automated attack generators, paired with a benign-prompt set and an existing public dataset for cross-checking. Prompts are grounded in four application scenarios—French translation, a finance chatbot, a code-to-SQL tool, and a general-purpose chat agent—and are tested with and without the system prompt to isolate context sensitivity. The argument runs through standard detection metrics, including precision, recall, accuracy, false-positive rate, F1 score, latency, and attack success rate, so the tool ranking is an empirical comparison rather than a feature checklist.

What would settle it

Collect a corpus of malicious prompts actually observed against deployed LLM-integrated applications, for example from production logs or bug-bounty reports, run the same seven tools plus the baseline on it, and check whether Lakera Guard still has the lowest false-positive rate and highest F1 score and whether the baseline still exceeds a 0.4 false-positive rate; a substantial reordering would show the benchmark's results are artifacts of its synthetic prompt mix.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that the emerging LLM safety-tool market has no obvious winner and that simple comparisons are misleading. The Palit benchmark evaluates seven filters—four open-source and three closed-source—plus ChatGPT-3.5-Turbo as a baseline. Lakera Guard posts the strongest combination across tests, with F1 scores of 0.659 without context, 0.809 with context, and 0.823 on an existing public prompt-injection dataset, along with sub-0.1-second latency; ProtectAI LLM Guard is close in precision but an order of magnitude slower. The baseline model often appears accurate because the dataset is imbalanced, yet its false-positive rates of 0.438–0.859 make it impractical. The paper also claims that the source of prompts matters: manually crafted attacks are the hardest to stop, while the behavior of generator-produced attacks changes sharply when context is added.

Load-bearing premise

The benchmark's malicious prompts are generated by researchers using manual techniques and automated attack tools, and the paper concedes it is unclear whether these resemble the attacks real adversaries would actually send to production LLM applications; if real-world attack distribution differs, the rankings may not transfer.

Editorial extensions

If this is right

Relying on a foundation model's built-in safety is not enough: at false-positive rates between 0.438 and 0.859, a production chatbot would flag or block a large share of legitimate requests.
If a deployer values both detection and low disruption, the results favor Lakera Guard and ProtectAI LLM Guard over the other five tested tools.
Context is a first-order variable: adding system prompts changed attack success rates sharply, including pushing manual prompts to 100% success against the baseline, so evaluations need application context to be meaningful.
Open-source tools in the study were hampered by outdated code and long latencies—one tool timed out on longer prompts and took seconds to minutes per prompt—so maintenance and speed are part of the safety story.
Accuracy is misleading on imbalanced attack/benign splits; false-positive rate and attack success rate should be reported alongside accuracy to judge a filter.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension would be to rerun the same seven tools on a corpus of attacks observed in real deployments, for example from production logs or bug-bounty reports; if the ranking changes, the synthetic prompt mix is not representative of field conditions.
The across-the-board improvement on the public prompt-injection dataset raises the possibility that several commercial filters have been trained or tuned on public benchmark data, which would make public-dataset results an upper bound rather than a field estimate.
The wide latency spread, from about 0.035 to over 29 seconds per prompt, implies an operational cost the paper leaves implicit: at production query volumes, latency alone may decide tool adoption even when a slower tool has marginally better detection.
The context ablation suggests that filter deployment should be tuned against the target application's actual prompt template, not validated in isolation.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. This is the first formal comparative evaluation of bolt-on LLM security tools, and it's worth reading for the market map alone. But the specific ranking—Lakera and ProtectAI as 'best overall'—should be treated as provisional, because the benchmark dataset isn't public and the paper itself shows the ranking shifts a lot depending on which prompt generator you use.

What's genuinely good: the authors found 13 tools, got 7 evaluated, and used a plausible mix of manual and automated attack generation. They measured precision, recall, FPR, latency, and ASR across three conditions (no context, with context, and the Deepset set), and they document a real market transparency problem—four closed-source vendors wouldn't give access. The baseline finding is the most robust result: ChatGPT-3.5-Turbo has FPR between 0.438 and 0.859 depending on condition, which makes it practically unusable as a filter. That conclusion holds up across all their conditions and is not sensitive to the dataset composition issues.

The soft spots are real but mostly disclosed. Section 5.3 admits the attacks come from research tools and manual templates, not threat intelligence, so representativeness for real-world attacks is unknown. Tables 6 and 7 show exactly why this matters: ASR varies sharply by prompt source. For example, with base prompts, Lakera's ASR is 0% on Garak prompts but 83% on PromptMap prompts, and the baseline is 100% on manual prompts and 0% on Garak and PromptMap. So the winning set is not invariant across prompt sources; 'best overall' is an artifact of the particular 75.8%-malicious mix unless representativeness is established. On top of that, the benchmark and code are not released, there are no confidence intervals, and the 'best overall' claim is not tied to any defined composite metric. The convenience sample—only four proprietary tools that happened to provide trials—further limits the generality.

That said, this is a legitimate empirical study, honestly reported. It's not a formal benchmark in the style of Garak or InjecAgent; it's a product evaluation. The right use is as a starting point, not as a verdict. I'd want the dataset and scoring criteria released, along with a sensitivity analysis across prompt sources, before treating the rankings as durable. But the authors clearly engaged with the literature, disclosed their limitations, and provided a useful snapshot of an emerging market.

Who it's for: practitioners choosing an LLM security tool, and researchers who want a baseline for future evaluations. It deserves a serious referee. I'd accept it with the expectation that the authors release the dataset and either define and justify a composite metric or temper the 'best overall' language.

Referee Report

3 major / 6 minor

Summary. This paper presents a comparative evaluation of seven LLM security tools (plus a ChatGPT-3.5-Turbo baseline) against a newly constructed benchmark called Palit, comprising 500 prompts (75.8% malicious) drawn from manual generation, Houyi, Garak, PromptMap, and the public Deepset prompt-injection dataset. The authors report precision, recall, accuracy, FPR, F1, latency, and attack success rates under three conditions: no context, with base-prompt context, and on the Deepset dataset. Their main conclusions are that the ChatGPT-3.5-Turbo baseline is unsuitable due to high false-positive rates (FPR 0.438-0.859), that Lakera Guard and ProtectAI LLM Guard are the best overall tools, and that the market of bolt-on LLM security solutions suffers from limited transparency and usability issues. The paper also surveys 13 tools and compares their claimed features.

Significance. If the results hold, this would be one of the first independent, published evaluations of commercial and open-source LLM security tools, and it would provide a useful template for future benchmarking in this area. The paper's strengths are its timely research question, the explicit attention to usability (latency, deployment, trial availability), the multi-source attack generation strategy, the ablation on system-prompt context, and the remarkably honest limitations section. The data tables support the qualitative claim that the baseline has a very high FPR and that several tools (especially Lakera, ProtectAI, and Azure Prompt Shield) achieve high precision with low FPR. However, the headline 'best overall' ranking is built on an undefined composite metric and on a benchmark whose representativeness of real-world attacks the authors themselves disclaim; the per-source ASR tables show that the ranking shifts substantially depending on prompt source. The paper is therefore a useful empirical contribution whose central claim needs substantial qualification before it can be accepted as stated.

major comments (3)

[Sections 4.3 and 5.2] The claim that 'Lakera Guard and ProtectAI LLM Guard emerged as the best overall tools' is not supported by any explicit definition of 'best overall.' The paper reports six separate metrics (accuracy, precision, recall, FPR, F1, latency) and no rule for combining them. Different tools win on different dimensions: e.g., Lakera has the highest accuracy on the benchmark with context (74.6%) and lowest latency (0.305s), but Azure Prompt Shield has higher precision (0.952) and lower FPR (0.074); on the Deepset dataset, Azure achieves precision 1.0 while Lakera achieves higher recall. Without a stated aggregation rule or a decision-theoretic justification (e.g., a cost model for false positives vs. false negatives vs. latency), the ordering is not reproducible and the 'best overall' wording should be replaced with a statement about specific metrics.
[Section 5.3, Tables 6 and 7] The external validity of the Palit benchmark is load-bearing for the ranking, and the manuscript itself disclaims it: 'it is unclear whether our attacks are representative of malicious prompts used by real-world attackers, given they were inspired by attacks created by researchers and not threat intelligence.' The internal results show that the ranking is not invariant across attack sources. For example, with context the baseline ASR is 100% on manual prompts but 0% on Garak and PromptMap prompts (Table 7), and Lakera's ASR on PromptMap prompts is 83.05% even though its overall ASR is low. This means the headline ranking could be an artifact of the particular 75.8%-malicious mix of manual/Houyi/Garak/PromptMap prompts. The authors should either (a) provide evidence that this mix approximates real-world malicious prompt distributions, or (b) restrict their conclusions to per-source results and characterize for which prompt families each tool is effective. Since the dataset is not released (no link or repository is given), readers cannot even check for overlaps with vendor training data, a risk the authors themselves raise for the Deepset portion in Section 5.2.
[Section 3.3 and Tables 3-7] All reported metrics are point estimates on a single dataset of 500 prompts, with no confidence intervals, bootstrap estimates, or significance tests. This is especially problematic for the per-source ASR tables, where the number of prompts per cell is small (e.g., the Garak-generated subset contains 13 DAN prompts in addition to encoding-based attacks, and the manual subset yields only 59-81 prompts per context condition). Differences that appear large, such as Vigil's drop in ASR from 93.22% to 70.37% on PromptMap prompts when context is added, may be within sampling noise. The authors should report uncertainties or at least explicitly discuss the statistical power limits before drawing conclusions about comparative performance.

minor comments (6)

[Abstract and Section 1] Grammar: 'evaluate these tools performance' should be 'evaluate these tools' performance'; 'addressed the following research questions' mixes tenses. The abstract would also benefit from stating the number of tools evaluated and the dataset size explicitly.
[Section 2.5] Typo: 'Similarlym, Greshake et al.' should be 'Similarly, Greshake et al.'
[Section 3.2] The benchmark dataset is not made available: no link, repository, or data availability statement is provided. For a paper whose main contribution is a 'benchmark dataset,' releasing it (or an anonymized subset) is essential for reproducibility and for verifying the authors' own concern about training-data overlap.
[Tables 6 and 7] The column header 'PAI Guard' should be 'ProtectAI LLM Guard' for consistency with Tables 3-5. Also, 'LK Similarity' and 'LK Canary' should be expanded or defined in the caption, since the abbreviations are not introduced in the table.
[Reference [79]] Reference [79] in Section 3.2 is the same paper as [43] (Liu et al., 'Prompt Injection attack against LLM-integrated Applications'); duplicate citations should be merged.
[Table 2] The table contains formatting issues, including a cut-off '/times✓' in the Output Scan. row and the header 'PAI Guard' (should be 'ProtectAI LLM Guard'). The legend says '-' for 'no information available,' but the cell for 'Jailbreak' in the ArthurAI column shows '-' which is inconsistent with the claim in the text that 'Arthur AI Firewall' provides jailbreak protection.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper is a direct empirical benchmark evaluation whose conclusions are computed from measured tool outputs, with no fitted inputs renamed as predictions and no load-bearing self-citation chain.

full rationale

This paper makes no derivation claims of the kind that can be circular. Its central results—relative tool rankings, false positive rates, attack success rates, and latency—are obtained by running seven security tools and a baseline model against a newly constructed benchmark dataset and reporting standard metrics (precision, recall, accuracy, FPR, F1, latency, ASR). The benchmark prompts were generated manually and with third-party generators (Houyi, Garak, PromptMap) plus the public Deepset dataset, independently of the tool outputs; no parameter was fitted to a subset of the evaluation data and then renamed as a prediction. The paper does not rest on a self-citation: the authors cite prior work for background, attack taxonomies, and tool descriptions, but the evaluative conclusions are supported by the measurements reported in Tables 3-7. The two validity concerns the authors themselves flag—whether research-generated prompts represent real-world attacks (Section 5.3) and the possibility that some tools were trained on the Deepset dataset (Section 5.2)—are external-validity and data-contamination risks, not circular reasoning. Acknowledging a limitation about representativeness does not make the benchmark's definition depend on the conclusion being drawn. Therefore, under the stated rules, the appropriate finding is no significant circularity with score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the validity of the benchmark as a measure of real-world attack detection, on reading each tool's API output as a true classification, on the fairness of the baseline prompt, and on the correctness of the external Deepset labels. No parameters were fitted and no new entities were postulated.

assumptions (4)

domain assumption The 500-prompt Palit benchmark, with 75.8% malicious prompts, is a valid instrument for measuring tool effectiveness.
The study's conclusions about tool ranking rest on this dataset; the paper acknowledges the imbalance inverts the real-world benign-majority distribution and that the attacks may not represent real attackers (Section 5.3).
domain assumption Binary classification by each tool (malicious or benign) via API responses is an accurate measure of the tool's security capability.
The evaluation reads tool outputs as labels; no independent verification of individual decisions is reported (Section 3.3).
domain assumption The sandwich defense prompt used for the ChatGPT-3.5-Turbo baseline ensures the baseline is classifying rather than following the user prompt.
Baseline results depend on this prompt design; if the sandwich defense fails, reported baseline false positive rates are not a fair measure (Section 3.3).
domain assumption The publicly available Deepset dataset labels are correct ground truth for maliciousness.
Deepset is used as an external check, but the study notes tools may have been trained on it, which would inflate their scores (Section 5.2).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating the efficacy of LLM Safety Solutions : The Palit Benchmark Dataset." pith.science (2026). https://pith.science/paper/OC3IC6WI

@misc{pith2026250513028,
  author       = {Pith},
  title        = {Pith review of: Evaluating the efficacy of LLM Safety Solutions : The Palit Benchmark Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OC3IC6WI}},
  note         = {Machine review of arXiv:2505.13028}
}

read the original abstract

Large Language Models (LLMs) are increasingly integrated into critical systems in industries like healthcare and finance. Users can often submit queries to LLM-enabled chatbots, some of which can enrich responses with information retrieved from internal databases storing sensitive data. This gives rise to a range of attacks in which a user submits a malicious query and the LLM-system outputs a response that creates harm to the owner, such as leaking internal data or creating legal liability by harming a third-party. While security tools are being developed to counter these threats, there is little formal evaluation of their effectiveness and usability. This study addresses this gap by conducting a thorough comparative analysis of LLM security tools. We identified 13 solutions (9 closed-source, 4 open-source), but only 7 were evaluated due to a lack of participation by proprietary model owners.To evaluate, we built a benchmark dataset of malicious prompts, and evaluate these tools performance against a baseline LLM model (ChatGPT-3.5-Turbo). Our results show that the baseline model has too many false positives to be used for this task. Lakera Guard and ProtectAI LLM Guard emerged as the best overall tools showcasing the tradeoff between usability and performance. The study concluded with recommendations for greater transparency among closed source providers, improved context-aware detections, enhanced open-source engagement, increased user awareness, and the adoption of more representative performance metrics.

Figures

Figures reproduced from arXiv: 2505.13028 by the authors.

**Figure 2.** Distribution of base prompts on the left & distribution of prompt sources on the right [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

**Figure 3.** Performance comparison of the tools when tested with the benchmark dataset with and without context and [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

83 extracted references · 58 canonical work pages

[1]

https://github.com

Github. https://github.com. Accessed: 2024-08-27. 3.1

2024
[2]

https://scholar.google.com

Google scholar. https://scholar.google.com. Accessed: 2024-08-27. 3.1

2024
[3]

https://simonwillison.net/ 2024/Mar/5/prompt-injection-jailbreaking/

Prompt injection and jailbreaking are not the same thing — simonwillison.net. https://simonwillison.net/ 2024/Mar/5/prompt-injection-jailbreaking/ . [Accessed 24-08-2024]. 2.3

2024
[4]

https://www.reddit.com

Reddit. https://www.reddit.com. Accessed: 2024-08-27. 3.1

work page 2024
[5]

https://www.pinecone.io/

The vector database to build knowledgeable AI | Pinecone — pinecone.io. https://www.pinecone.io/. [Accessed 25-08-2024]. 3.3

work page 2024
[6]

https://twitter.com

Twitter. https://twitter.com. Accessed: 2024-08-27. 3.1

work page 2024
[7]

https://learnprompting.org/docs/prompt_hacking/defensive_measures/ sandwich_defense, 2023

Sandwitch defense. https://learnprompting.org/docs/prompt_hacking/defensive_measures/ sandwich_defense, 2023. 2.4, 3.3

work page 2023
[8]

Conversational health agents: A personalized llm-powered agent framework

Mahyar Abbasian, Iman Azimi, Amir M Rahmani, and Ramesh Jain. Conversational health agents: A personalized llm-powered agent framework. arXiv preprint arXiv:2310.02374, 2023. 1

arXiv 2023

Show all 83 references

[9]

Gpt-4 technical report

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1

2023 arXiv
[10]

Vulnerabilities in personal firewalls caused by poor security usability

Bander Alfayyadh, James Ponting, Mohammed Alzomai, and Audun Jøsang. Vulnerabilities in personal firewalls caused by poor security usability. In 2010 IEEE International Conference on Information Theory and Information Security, pages 682–688. IEEE, 2010. 2.5

2010
[11]

Real Attackers Don’t Compute Gradients

Giovanni Apruzzese, Hyrum S. Anderson, Savino Dambra, David Freeman, Fabio Pierazzi, and Kevin Roundy. “Real Attackers Don’t Compute Gradients”: Bridging the Gap Between Adversarial ML Research and Practice. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (S...

2023
[12]

Protection — arthur.ai

ArthurAI. Protection — arthur.ai. https://www.arthur.ai/solution/firewall. [Accessed 26-08-2024]. 4.1

2024
[13]

LLM Hacking: Prompt Injection Techniques, July 2023

Austin Stubbs. LLM Hacking: Prompt Injection Techniques, July 2023. 2.2, 3.2

2023
[14]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Je...

2020
[15]

A LLM Assisted Exploitation of AI-Guardian, July 2023

Nicholas Carlini. A LLM Assisted Exploitation of AI-Guardian, July 2023. arXiv:2307.15008 [cs]. 2.1, 5.4

2023 arXiv
[16]

Vigil | Vigil: Documentation — vigil.deadbits.ai

DeadBitsAI. Vigil | Vigil: Documentation — vigil.deadbits.ai. https://vigil.deadbits.ai/. [Accessed 26-08-2024]. 4.1

2024
[17]

Gemini by google deepmind

Google DeepMind. Gemini by google deepmind. Accessed: 2024-08-26, 2023. https://www.deepmind.com/ gemini. 2.1

2024
[18]

deepset/prompt-injections · Datasets at Hugging Face — huggingface.co

deepset.ai. deepset/prompt-injections · Datasets at Hugging Face — huggingface.co. https://huggingface. co/datasets/deepset/prompt-injections. [Accessed 25-08-2024]. 3.2, 3.2, 4.3

2024
[19]

garak: A Framework for Security Probing Large Language Models, June 2024

Leon Derczynski, Erick Galinkin, Jeffrey Martin, Subho Majumdar, and Nanna Inie. garak: A Framework for Security Probing Large Language Models, June 2024. arXiv:2406.11036 [cs]. 2.5, 3.1, 3.2, 3.3

2024 arXiv
[20]

Machine learning models predicting returns: Why most popular performance metrics are misleading and proposal for an efficient metric

Jean Dessain. Machine learning models predicting returns: Why most popular performance metrics are misleading and proposal for an efficient metric. Expert Systems with Applications, 199:116970, 2022. 5.2

2022
[21]

Bert: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu...

2019
[22]

How should pre-trained language models be fine-tuned towards adversarial robustness? Advances in Neural Information Processing Systems , 34:4356–4369, 2021

Xinshuai Dong, Anh Tuan Luu, Min Lin, Shuicheng Yan, and Hanwang Zhang. How should pre-trained language models be fine-tuned towards adversarial robustness? Advances in Neural Information Processing Systems , 34:4356–4369, 2021. 2.4

2021
[23]

Comparing sql injection detection tools using attack injection: An experimental study

Ivano Alessandro Elia, Jose Fonseca, and Marco Vieira. Comparing sql injection detection tools using attack injection: An experimental study. In 2010 IEEE 21st International Symposium on Software Reliability Engineering, pages 289–298. IEEE, 2010. 2.5

2010
[24]

Comparing sql injection detection tools using attack injection: An experimental study

Ivano Alessandro Elia, Jose Fonseca, and Marco Vieira. Comparing sql injection detection tools using attack injection: An experimental study. In 2010 IEEE 21st International Symposium on Software Reliability Engineering, pages 289–298, 2010. 3.3

2010
[25]

Testing and comparing web vulnerability scanning tools for sql injection and xss attacks

Jose Fonseca, Marco Vieira, and Henrique Madeira. Testing and comparing web vulnerability scanning tools for sql injection and xss attacks. In 13th Pacific Rim international symposium on dependable computing (PRDC 2007), pages 365–372. IEEE, 2007. 2.5

2007
[26]

Testing and comparing web vulnerability scanning tools for sql injection and xss attacks

Jose Fonseca, Marco Vieira, and Henrique Madeira. Testing and comparing web vulnerability scanning tools for sql injection and xss attacks. In 13th Pacific Rim International Symposium on Dependable Computing (PRDC 2007), pages 365–372, 2007. 3.3

2007
[27]

Challenges in the real world use of classification accuracy metrics: From recall and precision to the matthews correlation coefficient

Giles M Foody. Challenges in the real world use of classification accuracy metrics: From recall and precision to the matthews correlation coefficient. Plos one, 18(10):e0291908, 2023. 5.2

2023
[28]

Github copilot

GitHub. Github copilot. Accessed: 2024-08-26, 2021. https://github.com/features/copilot. 2.1

2024
[29]

Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection, May 2023

Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection, May 2023. arXiv:2302.12173 [cs]. 2.5

2023 arXiv
[30]

Benchmarking approach to compare web applications static analysis tools detecting owasp top ten security vulnerabilities

Juan R Bermejo Higuera, Javier Bermejo Higuera, Juan A Sicilia Montalvo, Javier Cubo Villalba, and Juan José Nombela Pérez. Benchmarking approach to compare web applications static analysis tools detecting owasp top ten security vulnerabilities. Computers, Materials & Continua...

2020
[31]

Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming in the Wild, November 2023

Nanna Inie, Jonathan Stray, and Leon Derczynski. Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming in the Wild, November 2023. arXiv:2311.06237 [cs]. 2.5

2023 arXiv
[32]

Protect your AI applications in real time — Robust Intelligence — robustintelligence.com

Robust Intelligence. Protect your AI applications in real time — Robust Intelligence — robustintelligence.com. https://www.robustintelligence.com/platform/ai-firewall-guardrails . [Accessed 26-08-2024]. 4.1

2024
[33]

Chatgpt for good? on opportunities and challenges of large language models for education

Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and indi...

2023
[34]

Software updates as a security metric: Passive identification of update trends and effect on machine infection

Moazzam Khan, Zehui Bi, and John A Copeland. Software updates as a security metric: Passive identification of update trends and effect on machine infection. In MILCOM 2012-2012 IEEE Military Communications Conference, pages 1–6. IEEE, 2012. 5.1 15 arXiv Template A PREPRINT

2012
[35]

i have no idea what i’m doing

Katharina Krombholz, Wilfried Mayer, Martin Schmiedecker, and Edgar Weippl. " i have no idea what i’m doing"-on the usability of deploying{HTTPS}. In 26th USENIX Security Symposium (USENIX Security 17) , pages 1339–1356, 2017. 2.5

2017
[36]

Watch your language: large language models and content moderation

Deepak Kumar, Yousef AbuHashem, and Zakir Durumeric. Watch your language: large language models and content moderation. arXiv preprint arXiv:2309.14517, 2023. 1

2023 arXiv
[37]

Cummings, and Alexander Stimpson

Surender Suresh Kumar, M.L. Cummings, and Alexander Stimpson. Strengthening LLM Trust Boundaries: A Survey of Prompt Injection Attacks Surender Suresh Kumar Dr. M.L. Cummings Dr. Alexander Stimpson. In 2024 IEEE 4th International Conference on Human-Machine Systems (ICHMS), pa...

2024
[38]

Weight poisoning attacks on pre-trained models

Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pre-trained models. arXiv preprint arXiv:2004.06660, 2020. 2.1

2004 arXiv
[39]

12 Top LLM Security Tools: Paid & Free (Overview) | Lakera – Protecting AI teams that disrupt the world

Lakera. 12 Top LLM Security Tools: Paid & Free (Overview) | Lakera – Protecting AI teams that disrupt the world. 1, 3.1
[40]

End-to-End Security for the Generative AI Era — lasso.security

Lasso. End-to-End Security for the Generative AI Era — lasso.security. https://www.lasso.security/. [Accessed 26-08-2024]. 4.1

2024
[41]

Learn Prompting: Your Guide to Communicating with AI

Learn Prompting. Learn Prompting: Your Guide to Communicating with AI. 3.1
[42]

AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models, March 2024

Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models, March 2024. arXiv:2310.04451 [cs]. 2.5

2024 arXiv
[44]

Formalizing and benchmarking prompt injection attacks and defenses, 2024

Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. Formalizing and benchmarking prompt injection attacks and defenses, 2024. 2.2, 2.3, 2.5, 3.2

2024
[45]

Benchmarking of machine learning for anomaly based intrusion detection systems in the cicids2017 dataset

Ziadoon Kamil Maseer, Robiah Yusof, Nazrulazhar Bahaman, Salama A Mostafa, and Cik Feresa Mohd Foozy. Benchmarking of machine learning for anomaly based intrusion detection systems in the cicids2017 dataset. IEEE access, 9:22351–22370, 2021. 3.3

2021
[46]

PurpleLlama/Prompt-Guard/MODEL_card.md at main · meta-llama/PurpleLlama

Meta. PurpleLlama/Prompt-Guard/MODEL_card.md at main · meta-llama/PurpleLlama. 2.4, 3.1
[47]

Introducing ChatGPT

OpenAI. Introducing ChatGPT. https://openai.com/blog/chatgpt, 2023. 3.3

2023
[48]

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

2022
[49]

LLM Top 10 for LLMs v1.1

OW ASP. LLM Top 10 for LLMs v1.1. 2, 2.5, 3.1, 3.2
[50]

Comparative analysis of commercial and open source mobile device forensic tools

Radhika Padmanabhan, Karen Lobo, Mrunali Ghelani, Dhanika Sujan, and Mahesh Shirole. Comparative analysis of commercial and open source mobile device forensic tools. In 2016 Ninth International Conference on Contemporary Computing (IC3), pages 1–6. IEEE, 2016. 2.5

2016
[51]

Prompt Shields in Azure AI Content Safety - Azure AI services — learn.microsoft.com

PatrickFarley. Prompt Shields in Azure AI Content Safety - Azure AI services — learn.microsoft.com. https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/ jailbreak-detection. [Accessed 25-08-2024]. 3.3, 4.1

2024
[52]

Ignore Previous Prompt: Attack Techniques For Language Models, November 2022

Fábio Perez and Ian Ribeiro. Ignore Previous Prompt: Attack Techniques For Language Models, November 2022. arXiv:2211.09527 [cs]. 2.2, 2.5

2022 arXiv
[53]

Protect AI

Inc. Protect AI. Index - LLM Guard — llm-guard.com. https://llm-guard.com/. [Accessed 26-08-2024]. 4.1

2024
[54]

GitHub - protectai/rebuff: LLM Prompt Injection Detector — github.com

ProtectAI. GitHub - protectai/rebuff: LLM Prompt Injection Detector — github.com. https://github.com/ protectai/rebuff. [Accessed 25-08-2024]. 3.3, 4.1

2024
[55]

Fine-tuned deberta-v3 for prompt injection detection, 2023

ProtectAI.com. Fine-tuned deberta-v3 for prompt injection detection, 2023. 2.4, 3.1

2023
[56]

Software vulnerability detection using large language models

Moumita Das Purba, Arpita Ghosh, Benjamin J Radford, and Bill Chu. Software vulnerability detection using large language models. In 2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops (ISSREW), pages 112–119. IEEE, 2023. 1

2023
[57]

An Early Catego- rization of Prompt Injection Attacks on Large Language Models, January 2024

Sippo Rossi, Alisia Marianne Michel, Raghava Rao Mukkamala, and Jason Bennett Thatcher. An Early Catego- rization of Prompt Injection Attacks on Large Language Models, January 2024. arXiv:2402.00898 [cs]. 2.2, 2.5, 3.2 16 arXiv Template A PREPRINT

2024 arXiv
[58]

JasperLS/gelectra-base-injection· Hugging Face — huggingface.co

Jasper Schwenzow. JasperLS/gelectra-base-injection· Hugging Face — huggingface.co. https://huggingface. co/JasperLS/gelectra-base-injection . [Accessed 25-08-2024]. 4.1

2024
[59]

Aim | AI-FIREWALL — aim.security

AIM Security. Aim | AI-FIREWALL — aim.security. https://www.aim.security/solutions/ aim-ai-firewall. [Accessed 26-08-2024]. 4.1

2024
[60]

LLM Security — llmsecurity.net

LLM Security. LLM Security — llmsecurity.net. https://llmsecurity.net/. [Accessed 26-08-2024]. 3.1

2024
[61]

Prompt Security: The Platform for GenAI Security — prompt.security

Prompt Security. Prompt Security: The Platform for GenAI Security — prompt.security. https://www.prompt. security/. [Accessed 26-08-2024]. 4.1

2024
[62]

GitHub - utkusen/promptmap: automatically tests prompt injection attacks on ChatGPT instances — github.com

Utku Sen. GitHub - utkusen/promptmap: automatically tests prompt injection attacks on ChatGPT instances — github.com. https://github.com/utkusen/promptmap. [Accessed 25-08-2024]. 3.2

2024
[63]

Poison frogs! targeted clean-label poisoning attacks on neural networks.Advances in neural information processing systems, 31, 2018

Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks.Advances in neural information processing systems, 31, 2018. 2.1

2018
[64]

Large language model alignment: A survey

Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey. arXiv preprint arXiv:2309.15025, 2023. 2.4

2023 arXiv
[65]

Do Anything Now

Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models, May 2024. arXiv:2308.03825 [cs]. 2.3

2024 arXiv
[66]

Punctuation matters! stealthy backdoor attack for language models

Xuan Sheng, Zhicheng Li, Zhaoyang Han, Xiangmao Chang, and Piji Li. Punctuation matters! stealthy backdoor attack for language models. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 524–536. Springer, 2023. 1, 2.1

2023
[67]

Performance comparison of intrusion detection machine learning classifiers on benchmark and new datasets

Mohammed F Suleiman and Biju Issac. Performance comparison of intrusion detection machine learning classifiers on benchmark and new datasets. In 2018 28th International Conference on Computer Theory and Applications (ICCTA), pages 19–23. IEEE, 2018. 3.3

2018
[68]

Aligning large multimodal models with factually augmented rlhf

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023. 2.4

2023 arXiv
[69]

Adversarial machine learning : a taxonomy and terminology of attacks and mitigations

Apostol Vassilev, Alina Oprea, Alie Fordyce, and Hyrum Anderson. Adversarial machine learning : a taxonomy and terminology of attacks and mitigations. Technical Report NIST 100-2e2023, National Institute of Standards and Technology (U.S.), Gaithersburg, MD, January 2024. 1, 2,...

2024
[70]

Attention is all you need

Ashish Vaswani. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017. 1

2017 arXiv
[71]

Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36, 2024

Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36, 2024. 2.3

2024
[72]

Why johnny can’t encrypt: A usability evaluation of pgp 5.0

Alma Whitten and J Doug Tygar. Why johnny can’t encrypt: A usability evaluation of pgp 5.0. In USENIX security symposium, volume 348, pages 169–184, 1999. 2.5

1999
[73]

GitHub - whylabs/langkit: LangKit: An open-source toolkit for monitoring Large Language Models (LLMs)

WhyLabs. GitHub - whylabs/langkit: LangKit: An open-source toolkit for monitoring Large Language Models (LLMs). Extracts signals from prompts & responses, ensuring safety & security. Features include text quality, relevance metrics, & sentiment analysis. A comprehensive tool f...

2024
[74]

LLM Security Management — whylabs.ai

WhyLabs. LLM Security Management — whylabs.ai. https://whylabs.ai/llm-security. [Accessed 26-08-2024]. 4.1

2024
[75]

Bloomberggpt: A large language model for finance

Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023. 1, 2.1

2023 arXiv
[76]

Watch out for your agents! investigating backdoor threats to llm-based agents

Wenkai Yang, Xiaohan Bi, Yankai Lin, Sishuo Chen, Jie Zhou, and Xu Sun. Watch out for your agents! investigating backdoor threats to llm-based agents. arXiv preprint arXiv:2402.11208, 2024. 1

2024 arXiv
[77]

Poisonprompt: Backdoor attack on prompt-based large language models

Hongwei Yao, Jian Lou, and Zhan Qin. Poisonprompt: Backdoor attack on prompt-based large language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7745–7749. IEEE, 2024. 1, 2.1

2024
[78]

A survey on large language model (llm) security and privacy: The good, the bad, and the ugly

Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, page 100211, 2024. 1, 2 17 arXiv Template A PREPRINT

2024
[79]

Prompt Injection attack against LLM-integrated Applications, March

Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. Prompt Injection attack against LLM-integrated Applications, March
[80]

Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts

Jiahao Yu, Xingwei Lin, and Xinyu Xing. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253, 2023. 3.1

2023 arXiv
[81]

arXiv:2306.05499 [cs]. 3.2

arXiv
[82]

Prompt as triggers for backdoor attack: Examining the vulnerability in language models

Shuai Zhao, Jinming Wen, Luu Anh Tuan, Junbo Zhao, and Jie Fu. Prompt as triggers for backdoor attack: Examining the vulnerability in language models. arXiv preprint arXiv:2305.01219, 2023. 1, 2.1

2023 arXiv
[83]

InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents, March 2024

Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents, March 2024. arXiv:2403.02691 [cs]. 2.5

2024 arXiv
[85]

Zico Kolter, and Matt Fredrikson

Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models, December 2023. arXiv:2307.15043 [cs]. 2.1, 3.1, 5.4 18

2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

https://github.com

Github. https://github.com. Accessed: 2024-08-27. 3.1

2024

[2] [2]

https://scholar.google.com

Google scholar. https://scholar.google.com. Accessed: 2024-08-27. 3.1

2024

[3] [3]

https://simonwillison.net/ 2024/Mar/5/prompt-injection-jailbreaking/

Prompt injection and jailbreaking are not the same thing — simonwillison.net. https://simonwillison.net/ 2024/Mar/5/prompt-injection-jailbreaking/ . [Accessed 24-08-2024]. 2.3

2024

[4] [4]

https://www.reddit.com

Reddit. https://www.reddit.com. Accessed: 2024-08-27. 3.1

work page 2024

[5] [5]

https://www.pinecone.io/

The vector database to build knowledgeable AI | Pinecone — pinecone.io. https://www.pinecone.io/. [Accessed 25-08-2024]. 3.3

work page 2024

[6] [6]

https://twitter.com

Twitter. https://twitter.com. Accessed: 2024-08-27. 3.1

work page 2024

[7] [7]

https://learnprompting.org/docs/prompt_hacking/defensive_measures/ sandwich_defense, 2023

Sandwitch defense. https://learnprompting.org/docs/prompt_hacking/defensive_measures/ sandwich_defense, 2023. 2.4, 3.3

work page 2023

[8] [8]

Conversational health agents: A personalized llm-powered agent framework

Mahyar Abbasian, Iman Azimi, Amir M Rahmani, and Ramesh Jain. Conversational health agents: A personalized llm-powered agent framework. arXiv preprint arXiv:2310.02374, 2023. 1

arXiv 2023