REVIEW 14 cited by
EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Although language model (LM) agents have demonstrated increased performance in multiple domains, including coding and web-browsing, their success in cybersecurity has been limited. We present EnIGMA, an LM agent for autonomously solving Capture The Flag (CTF) challenges. We introduce new tools and interfaces to improve the agent's ability to find and exploit security vulnerabilities, focusing on interactive terminal programs. These novel Interactive Agent Tools enable LM agents, for the first time, to run interactive utilities, such as a debugger and a server connection tool, which are essential for solving these challenges. Empirical analysis on 390 CTF challenges across four benchmarks demonstrate that these new tools and interfaces substantially improve our agent's performance, achieving state-of-the-art results on NYU CTF, Intercode-CTF, and CyBench. Finally, we analyze data leakage, developing new methods to quantify it and identifying a new phenomenon we term soliloquizing, where the model self-generates hallucinated observations without interacting with the environment. Our code and development dataset are available at https://github.com/SWE-agent/SWE-agent/tree/v0.7 and https://github.com/NYU-LLM-CTF/NYU_CTF_Bench/tree/main/development respectively.
Forward citations
Cited by 14 Pith papers
-
RECEIPT: Deterministic, Reward-Hacking-Resistant Verification for White-Box Agentic XSS Discovery
A replay-based verifier with environment isolation, role separation, and a browser-execution sentinel lets white-box LLM agents report XSS exploits that are real, reproducible, attacker-to-victim vulnerabilities.
-
Veritas: Grounding LLM Agents for Reliable Vulnerability Reasoning over Stripped Binaries
Veritas detects out-of-bounds vulnerabilities in stripped binaries at 90% recall by grounding LLM reasoning in static witness-backed flows and runtime validation.
-
FirmPilot: Evidence-Guided Multi-Agent Environment Recovery for IoT Firmware Rehosting
An evidence-guided multi-agent rehosting loop raises web-service reachability on the LFwC corpus from 25.49% to 52.39% and network reachability from 39.30% to 71.93%.
-
CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution
CRAKEN, an LLM agent combining Self-RAG and Graph-RAG over a CTF writeup database, solves 22% of NYU CTF Bench challenges, three percentage points above the prior D-CIPHER baseline.
-
MM-Agent: LLM as Agents for Real-world Mathematical Modeling Problem
MM-Agent, a multi-stage LLM pipeline with a hierarchical modeling method library, is claimed to outperform prior agents and award-winning human solutions on a new 111-problem MCM/ICM-based mathematical modeling benchmark.
-
VEXAIoT: Autonomous IoT Vulnerability EXploitation using AI Agents
A multi-agent LLM framework autonomously discovers and exploits OWASP-mapped IoT vulnerabilities with 95% success across 260 trials in IoTGoat and Metasploitable2.
-
Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security
CIRCLE is a 1,260-prompt benchmark that measures how often commercial LLM code interpreters refuse, execute, or time out on resource-exhaustion tasks, revealing large and inconsistent safety gaps.
-
AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models
A new 70-challenge benchmark measures whether language models can autonomously discover and exploit AI/ML security vulnerabilities, finding Claude-3.7-Sonnet solves 61% of challenges while the best open-source model s...
-
AgentRec: Agent Recommendation Using Sentence Embeddings Aligned to Human Feedback
AgentRec routes a natural-language prompt to the right LLM agent by embedding the prompt and comparing it with per-agent prompt embeddings, reaching 92.2% top-1 accuracy on a synthetic test set.
-
Hacking CTFs with Plain Agents
A simple GPT-4o agent with ReAct prompting and up to five attempts solves 81 of 85 (95%) evaluable InterCode-CTF tasks, though benchmark exclusions and possible data contamination weaken the capability claim.
-
Forewarned is Forearmed: A Survey on Large Language Model-based Agents in Autonomous Cyberattacks
A review of LLM-based agents as autonomous cyberattackers, arguing that they lower attack costs, scale up threats, and outpace existing defenses.
-
AutoPentest: Enhancing Vulnerability Management With Autonomous LLM Agents
AutoPentest, an autonomous GPT-4o multi-agent pentesting tool, completes 15-26% of subtasks on three Hack The Box machines, mildly outperforming manual ChatGPT-4o at API cost of $96.20.
-
HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing
A two-module LLM agent with a planner and summarizer solves 41 of 120 PicoCTF and 32 of 80 OverTheWire challenges using GPT-4o, and the authors release both the agent and the benchmarks.
-
From Promise to Peril: Rethinking Cybersecurity Red and Blue Teaming in the Age of LLMs
LLMs can assist both attackers and defenders in cybersecurity, but context limits, hallucinations, and weak reasoning make them unsafe to deploy without human oversight and real-world evaluation.
Discussion (0). Continue with ORCID to comment.