REVIEW 5 cited by
Hacking CTFs with Plain Agents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We saturate a high-school-level hacking benchmark with plain LLM agent design. Concretely, we obtain 95% performance on InterCode-CTF, a popular offensive security benchmark, using prompting, tool use, and multiple attempts. This beats prior work by Phuong et al. 2024 (29%) and Abramovich et al. 2024 (72%). Our results suggest that current LLMs have surpassed the high school level in offensive cybersecurity. Their hacking capabilities remain underelicited: our ReAct&Plan prompting strategy solves many challenges in 1-2 turns without complex engineering or advanced harnessing.
Forward citations
Cited by 5 Pith papers
-
Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents
Security-agent success changes differently with budget: offensive CTF tasks improve with more compute, while defensive SOC work depends more on tool discipline than spend.
-
Asymmetry by Design: Boosting Cyber Defenders with Differential Access to AI
A policy framework for shaping access to AI cyber tools to favor defenders, with three access approaches and implementation guidance.
-
Evaluating AI cyber capabilities with crowdsourced elicitation
Crowdsourced AI teams solved nearly all challenges in a small CTF and beat 90% of registered human teams in a large one, pointing to open-market elicitation as a viable evaluation tool.
-
CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution
CRAKEN, an LLM agent combining Self-RAG and Graph-RAG over a CTF writeup database, solves 22% of NYU CTF Bench challenges, three percentage points above the prior D-CIPHER baseline.
-
AI Agent Governance: A Field Guide
A field guide that maps risks from AI agents and proposes a five-category taxonomy of governance interventions (alignment, control, visibility, security and robustness, societal integration).
Discussion (0). Sign in to comment.