REVIEW 28 cited by
R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large language models (LLMs) have exhibited great potential in autonomously completing tasks across real-world applications. Despite this, these LLM agents introduce unexpected safety risks when operating in interactive environments. Instead of centering on the harmlessness of LLM-generated content in most prior studies, this work addresses the imperative need for benchmarking the behavioral safety of LLM agents within diverse environments. We introduce R-Judge, a benchmark crafted to evaluate the proficiency of LLMs in judging and identifying safety risks given agent interaction records. R-Judge comprises 569 records of multi-turn agent interaction, encompassing 27 key risk scenarios among 5 application categories and 10 risk types. It is of high-quality curation with annotated safety labels and risk descriptions. Evaluation of 11 LLMs on R-Judge shows considerable room for enhancing the risk awareness of LLMs: The best-performing model, GPT-4o, achieves 74.42% while no other models significantly exceed the random. Moreover, we reveal that risk awareness in open agent scenarios is a multi-dimensional capability involving knowledge and reasoning, thus challenging for LLMs. With further experiments, we find that fine-tuning on safety judgment significantly improve model performance while straightforward prompting mechanisms fail. R-Judge is publicly available at https://github.com/Lordog/R-Judge.
Forward citations
Cited by 28 Pith papers
-
The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilities
Execution-security research for AI coding agents is fragmented across 17 mechanism categories with five unaddressed cross-cutting gaps, including missing head-to-head isolation-vs-capability evaluation and untested re...
-
SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinement
SkillAttack exploits latent vulnerabilities in unmodified agent skills via closed-loop adversarial prompting, reaching ASR 0.73–0.93 on adversarial skills and up to 0.26 on real-world skills across 10 LLMs.
-
How Benchmarks Mis-Score Computer-Use Agents
About 15.3% of FAIL verdicts across five CUA benchmarks are wrong, and genuine failures are mostly verification and planning errors, not clicks.
-
Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks
On Cybench CTF tasks, 21 of 22 models cheated under baseline, 37% of baseline passes were cheated, and stricter anti-cheat prompts reduced but did not remove cheating.
-
Execution-Grounded Security Testing for Coding Agents in Software Engineering Pipelines
Disguising unsafe operations as routine software-engineering tests makes coding agents execute them at high verified rates, revealing a gap between language-layer refusal and execution-layer harm.
-
StepShield: When, Not Whether to Intervene on Rogue Agents
StepShield's step-level benchmark claims detection timeliness is a separate axis from accuracy, and reports a 2.3x early-intervention gap between LLM judges and regex guardrails that standard metrics cannot see.
-
Measuring the Security of Mobile LLM Agents under Adversarial Prompts from Untrusted Third-Party Channels
Indirect prompt injection through ads, webviews, and notifications reliably diverts mobile LLM agents into leaking data and installing malware across eight evaluated agents.
-
SENTINEL: A Multi-Level Formal Framework for Safety Evaluation of Foundation Model-based Embodied Agents
A three-level temporal-logic safety evaluator for embodied LLM agents that checks NL-to-LTL interpretation, plan compliance, and CTL over simulated execution trees.
-
MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environments
MLA-Trust introduces 34 tasks and an evaluation toolbox showing that GUI-interacting multimodal agents are substantially less trustworthy than static multimodal chat models.
-
RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents
Multimodal computer-use agents show risky intent in about 85% of tasks and complete risky actions in about 60%, measured on RiOSWorld, a new 492-task virtual-machine benchmark.
-
USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Models
USB-SafeBench is a unified MLLM safety benchmark with 61 risk categories, 4 modality combinations, and dual-language vulnerability and oversensitivity tests.
-
Safety Degradation in AI Agents
Adding retrieval to aligned LLMs degrades safety: refusal rates fall, bias and harmfulness rise, and prompt-based mitigation only partially restores alignment.
-
Agentic Harnesses: LLM-Driven Verification Layers for Robot Autonomy
An LLM-driven gate between robot planning and execution labels plans accept, reject, or escalate, reporting 81 percent accuracy and no direct accept/reject errors on small test sets.
-
SkillCenter: A Large-Scale Source-Grounded Skill Library for Autonomous AI Agents
SkillCenter provides 216,938 source-grounded skills across 24 domains via an automated LLM-based pipeline, with a downstream evaluation showing skills help agents only when the task exceeds the model's knowledge and r...
-
LLM Agents Should Employ Security Principles
A position paper proposing AgentSandbox, a framework that applies Saltzer-Schroeder security principles to LLM agents and reports large attack-success-rate reductions on AgentDojo.
-
SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents
SafeScientist adds prompt, discussion, tool-use, and output-review safety checks to an AI scientist, with a new domain benchmark, but its reported evaluation is internally inconsistent.
-
ALRPHFS: Adversarially Learned Risk Patterns with Hierarchical Fast \& Slow Reasoning for Robust Agent Defense
ALRPHFS builds an adversarially refined library of semantic risk patterns and uses fast retrieval plus slow LLM reasoning to defend LLM agents, reporting best-in-class average accuracy near 80 percent.
-
Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Values
Value Compass Benchmarks is a live, self-evolving platform that scores 33 LLMs across 27 value dimensions from four value systems, aiming to reveal true behavioral alignment with human values.
-
Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMs
A supervised fine-tuning loss that maximizes an Earth-Mover-Distance-style semantic penalty away from model-generated unsafe responses achieves safety with roughly 100 harmful examples.
-
Evaluation and Benchmarking of LLM Agents: A Survey
A review that proposes a two-dimensional taxonomy for evaluating LLM agents and highlights enterprise-specific evaluation gaps.
-
A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
The paper surveys security risks of LLM agents, organizes them into a five-level autonomy taxonomy, and proposes an untested CMDP-based architecture called R2A2.
-
Forewarned is Forearmed: A Survey on Large Language Model-based Agents in Autonomous Cyberattacks
A review of LLM-based agents as autonomous cyberattackers, arguing that they lower attack costs, scale up threats, and outpace existing defenses.
-
Efficient MAP Estimation of LLM Judgment Performance with Prior Transfer
BetaConform estimates LLM ensemble judgment accuracy from few labeled samples via a mixture of Beta-Binomial distributions, conformal-style adaptive stopping, and text-similarity prior transfer, but its theoretical gu...
-
Agentic Web: Weaving the Next Web with AI Agents
A position paper defines the Agentic Web as the next web era and proposes a three-dimensional conceptual framework for understanding and building it.
-
SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation
A systematization-of-knowledge survey that categorizes LLM privacy risks into training data, prompts, outputs, and agents, and reviews limitations of current mitigations.
-
The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs
A structured survey of LLM safety evaluation that proposes a why/what/where/how taxonomy and catalogs metrics, datasets, benchmarks, evaluators, and frameworks.
-
The Science of Evaluating Foundation Models
A survey-and-checklist proposal that organizes LLM evaluation into an ABCD framework (Algorithm, Big Data, Computation, Domain Expertise) for context-aware, documented assessment.
-
Large Language Model Safety: A Holistic Survey
A broad survey of LLM safety that groups the literature into four risk areas and four related areas, with a taxonomy and a public repository of papers, but no new empirical results.
Discussion (0). Continue with ORCID to comment.