Pith. sign in

REVIEW 16 cited by

EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.16165 v3 pith:VAN4SOPH submitted 2024-09-24 cs.AI

Talor Abramovich , Meet Udeshi , Minghao Shao , Kilian Lieret , Haoran Xi , Kimberly Milner , Sofija Jancheska , John Yang
show 8 more authors
Carlos E. Jimenez Farshad Khorrami Prashanth Krishnamurthy Brendan Dolan-Gavitt Muhammad Shafique Karthik Narasimhan Ramesh Karri Ofir Press
This is my paper
classification cs.AI
keywords agentinteractivetoolsagentschallengesdevelopmentenigmagithub
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Although language model (LM) agents have demonstrated increased performance in multiple domains, including coding and web-browsing, their success in cybersecurity has been limited. We present EnIGMA, an LM agent for autonomously solving Capture The Flag (CTF) challenges. We introduce new tools and interfaces to improve the agent's ability to find and exploit security vulnerabilities, focusing on interactive terminal programs. These novel Interactive Agent Tools enable LM agents, for the first time, to run interactive utilities, such as a debugger and a server connection tool, which are essential for solving these challenges. Empirical analysis on 390 CTF challenges across four benchmarks demonstrate that these new tools and interfaces substantially improve our agent's performance, achieving state-of-the-art results on NYU CTF, Intercode-CTF, and CyBench. Finally, we analyze data leakage, developing new methods to quantify it and identifying a new phenomenon we term soliloquizing, where the model self-generates hallucinated observations without interacting with the environment. Our code and development dataset are available at https://github.com/SWE-agent/SWE-agent/tree/v0.7 and https://github.com/NYU-LLM-CTF/NYU_CTF_Bench/tree/main/development respectively.

Discussion (0). Sign in to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. RECEIPT: Deterministic, Reward-Hacking-Resistant Verification for White-Box Agentic XSS Discovery

    cs.CR 2026-07 conditional novelty 8.0 of 10

    A replay-based verifier with environment isolation, role separation, and a browser-execution sentinel lets white-box LLM agents report XSS exploits that are real, reproducible, attacker-to-victim vulnerabilities.

  2. Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots

    cs.CR 2026-05 unverdicted novelty 7.0 of 10

    Honeyval evaluates LLM HTTP honeypots with AI attackers and shows they produce longer interactions, lower detection rates, and cost advantages over rule-based baselines.

  3. CyberEvolver: Structured Self-Evolution for Cybersecurity Agents On the Fly

    cs.CR 2026-05 unverdicted novelty 7.0 of 10

    CyberEvolver introduces a four-layer self-evolving agent architecture with trace-to-diagnosis and population beam search that raises seed agent success rates by 13.6% on CTF, exploitation, and penetration tasks across...

  4. Veritas: Grounding LLM Agents for Reliable Vulnerability Reasoning over Stripped Binaries

    cs.SE 2026-05 conditional novelty 7.0 of 10

    Veritas detects binary out-of-bounds vulnerabilities via witness-backed IR flows, dual-view LLM claims, and multi-agent runtime validation, reaching 90% recall and a confirmed Apple CVE.

  5. Empowering Autonomous Debugging Agents with Efficient Dynamic Analysis

    cs.SE 2026-04 unverdicted novelty 7.0 of 10

    ADI equips AI debugging agents with function-level interaction via a new execution trace structure, raising SWE-bench Verified resolution to 63.8% at $1.28 per task and delivering 6-18% gains when added to existing agents.

  6. CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge

    cs.CR 2026-04 unverdicted novelty 7.0 of 10

    CyberCertBench shows frontier LLMs reach human-expert performance on general IT and networking security but drop on vendor-specific and formal standards questions such as IEC 62443, with a new framework for producing ...

  7. FirmPilot: Evidence-Guided Multi-Agent Environment Recovery for IoT Firmware Rehosting

    cs.SE 2026-07 conditional novelty 6.0 of 10

    An evidence-guided multi-agent rehosting loop raises web-service reachability on the LFwC corpus from 25.49% to 52.39% and network reachability from 39.30% to 71.93%.

  8. Mastermind: Strategy-grounded Learning for Repository-Scale Vulnerability Reproduction

    cs.AI 2026-07 unverdicted novelty 6.0 of 10

    Mastermind's dual-loop planner learns transferable strategies via SFT and milestone GRPO, raising GPT-5.5 executor pass rate on 200 held-out CyberGym tasks from 60% to 84.5%.

  9. MetaPS: Adaptive Programmatic Strategy Selection for Market Agents

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    MetaPS trains models via simulation rollouts to select from programmatic strategy libraries for market agents, yielding better performance than fixed or direct LLM baselines across model sizes.

  10. Veritas: Grounding LLM Agents for Reliable Vulnerability Reasoning over Stripped Binaries

    cs.SE 2026-05 unverdicted novelty 6.0 of 10

    Veritas detects memory corruption vulnerabilities in stripped binaries by combining static value-flow slicing, dual-view LLM reasoning, and multi-agent runtime validation, reporting 90% recall, zero false positives on...

  11. Veritas: Grounding LLM Agents for Reliable Vulnerability Reasoning over Stripped Binaries

    cs.SE 2026-05 conditional novelty 6.0 of 10

    Veritas detects out-of-bounds vulnerabilities in stripped binaries at 90% recall by grounding LLM reasoning in static witness-backed flows and runtime validation.

  12. Patch2Vuln: Agentic Reconstruction of Vulnerabilities from Linux Distribution Binary Patches

    cs.CR 2026-05 unverdicted novelty 6.0 of 10

    An agentic pipeline localizes the security-relevant function in 10 of 20 Ubuntu binary security updates and produces an accepted root-cause classification in 11 of 20, limited mainly by binary differencing coverage.

  13. VEXAIoT: Autonomous IoT Vulnerability EXploitation using AI Agents

    cs.CR 2026-07 conditional novelty 5.5 of 10

    A multi-agent LLM framework autonomously discovers and exploits OWASP-mapped IoT vulnerabilities with 95% success across 260 trials in IoTGoat and Metasploitable2.

  14. Poisoned Playbooks: Demystifying Knowledge Poisoning Effects on AI Security Agents

    cs.CR 2026-06 unverdicted novelty 5.0 of 10

    Injecting one poisoned playbook into security knowledge sources causes systematic behavior changes in RAG AI agents across multiple LLMs and tasks, explained by a Verification Boundary classification.

  15. Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks

    cs.CR 2026-04 unverdicted novelty 5.0 of 10

    Claude 4.5 Opus reaches 59% solve rate on offensive cyber CTF tasks, with a Kali Linux environment adding 9.5 percentage points over Ubuntu while prompt engineering often hurts performance in equipped setups.

  16. Challenges and Future Directions in Agentic Reverse Engineering Systems

    cs.CR 2026-04 unverdicted novelty 3.0 of 10

    Agentic LLM systems for reverse engineering fail on obfuscation, timing, and unique architectures due to token limits and missing guardrails, with challenges and directions proposed.

Pith tools