REVIEW 19 cited by
A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large Language Model (LLM) systems are inherently compositional, with individual LLM serving as the core foundation with additional layers of objects such as plugins, sandbox, and so on. Along with the great potential, there are also increasing concerns over the security of such probabilistic intelligent systems. However, existing studies on LLM security often focus on individual LLM, but without examining the ecosystem through the lens of LLM systems with other objects (e.g., Frontend, Webtool, Sandbox, and so on). In this paper, we systematically analyze the security of LLM systems, instead of focusing on the individual LLMs. To do so, we build on top of the information flow and formulate the security of LLM systems as constraints on the alignment of the information flow within LLM and between LLM and other objects. Based on this construction and the unique probabilistic nature of LLM, the attack surface of the LLM system can be decomposed into three key components: (1) multi-layer security analysis, (2) analysis of the existence of constraints, and (3) analysis of the robustness of these constraints. To ground this new attack surface, we propose a multi-layer and multi-step approach and apply it to the state-of-art LLM system, OpenAI GPT4. Our investigation exposes several security issues, not just within the LLM model itself but also in its integration with other components. We found that although the OpenAI GPT4 has designed numerous safety constraints to improve its safety features, these safety constraints are still vulnerable to attackers. To further demonstrate the real-world threats of our discovered vulnerabilities, we construct an end-to-end attack where an adversary can illicitly acquire the user's chat history, all without the need to manipulate the user's input or gain direct access to OpenAI GPT4. Our demo is in the link: https://fzwark.github.io/LLM-System-Attack-Demo/
Forward citations
Cited by 19 Pith papers
-
Understanding the Supply Chain and Risks of Large Language Model Applications
A new benchmark dataset traces dependencies across 3,859 LLM applications, 109,211 models, 2,474 datasets, and 8,862 libraries, and finds widespread known vulnerabilities in application dependencies.
-
Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware
MARIN reduces API hallucinations in LLM-generated code by supplying project dependency context and constraining decoding to the project's valid APIs, as shown on a new 416-sample Java benchmark and Huawei internal projects.
-
Pirates of the RAG: Adaptively Attacking LLMs to Leak Knowledge Bases
A relevance-guided, fully automatic attack using open-source tools extracts most of a RAG system's private knowledge base without any access to the target's internals.
-
Requirements-Augmented Generation for Trustworthy Acceptance Testing of LLM-Based Software
REAG and a confidence-calibrated cascade generate context-aware test oracles for LLM-based software and produce statistically controlled verdict reliability, demonstrated on a production nutrition advisory app.
-
How Do You Choose Your AI Component? An Interview Study of Secure AI Integration in Practice
In interviews, 22 practitioners chose AI models primarily on functionality, cost, and trust in vendors, with security rarely a formal criterion, suggesting the industry is repeating early software supply chain mistakes.
-
Direct Behavior Optimization: Unlocking the Potential of Lightweight LLMs
A gradient-free Monte Carlo tree search over JSON key-step plans produces few-shot demonstrations that let LLaMA3-8B and LLaMA3.2-3B outperform GPT-3.5 on most of seven BIG-Bench Hard tasks.
-
Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling
Fine-tuned small language models can reach high accuracy on function calling, but zero-shot and few-shot performance is poor, and the study's few-shot results are compromised by using test-set examples in the prompt.
-
MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents
MELON detects indirect prompt injection in LLM agents by re-executing each step with the user prompt replaced by a neutral task prompt and flagging attacks when the original and masked runs produce similar tool calls.
-
Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models
Universal adversarial suffixes can shift any text's embedding toward a model's biased mean direction, breaking embedding-based LLM safety classifiers.
-
Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection
ROSI bakes the refusal direction into a model's weight matrices via a rank-one update, raising refusal and jailbreak robustness with minimal measured utility cost.
-
SAFEFLOW: A Principled Protocol for Trustworthy and Transactional Autonomous Agent Systems
SAFEFLOW wraps LLM/VLM agents in fine-grained information-flow control, verifier-gated trust adjustment, and transactional concurrency, and its authors report near-perfect safety on their own benchmark plus AgentHarm,...
-
Unveiling the Landscape of LLM Deployment in the Wild: An Empirical Study
An internet-wide scan of 320,102 public-facing LLM services across 15 frameworks finds widespread plain-HTTP exposure, weak TLS, and unauthenticated access to model and system APIs.
-
Detecting Malicious Source Code in PyPI Packages with LLMs: Does RAG Come in Handy?
Retrieval-augmented generation with YARA rules, GitHub advisories, and malicious code snippets failed to improve LLM detection of malicious PyPI packages, while fine-tuned LLaMA-3.1-8B reached 97% accuracy.
-
Prompt-in-Content Attacks: Exploiting Uploaded Inputs to Hijack LLM Behavior
Embedding a short 'system instruction' in a .docx file causes several commercial LLMs to refuse, substitute, redirect, or bias their output during summarization tasks.
-
A Survey on Model Extraction Attacks and Defenses for Large Language Models
A taxonomy of model extraction attacks and defenses for large language models, with proposed evaluation metrics and future research directions.
-
Automated Consistency Analysis of LLMs
LLMs frequently produce inconsistent answers to repeated cybersecurity questions, but the paper's validation methods rely on the models judging their own and each other's responses.
-
Next-Generation Phishing: How LLM Agents Empower Cyber Attackers
LLM-rephrased phishing emails evade current email detectors more often than original ones, and training on LLM-generated variants partly restores detection.
-
LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasures
This survey categorizes attacks on large language models by lifecycle phase and maps them to prevention and detection defenses, concluding that only a few defenses are highly effective.
-
Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions
The paper surveys recent work, models, applications, and challenges of using LLMs in intelligent transportation systems, without presenting new experimental results.
Discussion (0). Continue with ORCID to comment.