Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Giving a planner-executor LLM agent an iterative retrieval loop over curated CTF solution writeups raises its solve rate on a 200-challenge benchmark from 19% to 22%, at an average cost increase of $0.34 per solved challenge.

desk verdict The 3% SOTA claim is not statistically established and the paper's own numbers disagree, but the open-sourced knowledge-retrieval recipe for CTF agents is a real contribution worth a careful revision. read the letter →

arxiv 2505.17107 v1 pith:YIB3VNHI submitted 2025-05-21 cs.CR cs.AIcs.LGcs.MA

Minghao Shao , Haoran Xi , Nanda Rani , Meet Udeshi , Venkata Sai Charan Putrevu , Kimberly Milner , Brendan Dolan-Gavitt , Sandeep Kumar Shukla
show 4 more authors
Prashanth Krishnamurthy Farshad Khorrami Ramesh Karri Muhammad Shafique
This is my paper
classification cs.CRcs.AIcs.LGcs.MA
keywords cybersecurityagentscapture-the-flagretrieval-augmentedgenerationself-RAGgraph-RAGknowledgeinjectionLLMoffensivesecurity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Cybersecurity LLM agents are limited by the fact that their training data stops at a cutoff date and does not contain step-by-step operational exploit knowledge. This paper claims that the gap can be closed without retraining by wiring a retrieval-augmented generation loop into a planner-executor agent: decompose each delegated task into a focused query, retrieve relevant CTF solution writeups from a curated database, grade the retrieved material and the generated hint, and rewrite the query if either check fails. On a public 200-challenge CTF benchmark the system solves 42 of 200 challenges (22%), four more than the previous best agent (38, or 19%), with an average cost increase of about $0.34 per solved challenge. If the pattern holds, it gives a modular template for injecting fresh domain knowledge into LLM-driven task planning systems generally.

What carries the argument

The load-bearing object is the recursive Self-RAG loop: retrieve, grade relevance, generate, grade for hallucination, rewrite, and retry, capped by a maximum recursion depth. A second mechanism, Graph-RAG, converts retrieved writeups into semantic triplets and a knowledge graph, so retrieval can follow connected concepts rather than only vector similarity. The two work over a curated knowledge database of 1,298 CTF writeups in markdown form, intentionally excluding writeups from the same competition family as the benchmark to reduce contamination. The retrieval trigger is the delegation step: the executor receives a knowledge hint, which is what lets the paper attribute gains to knowledge rather than to bigger prompts.

What would settle it

Run the same two configurations twenty times each with different random seeds on the 200-challenge benchmark and compare the distribution of solved counts; the claimed improvement is real only if the knowledge-based agent's count is above the baseline's upper quantile. Also swap the writeup database for a scrambled version and see if the 22% solve rate persists.

Watch

Extended reading notes

Core claim

The paper's central claim is that knowledge injection at execution time, not better tools or more context, is what pushes LLM agents past their current ceiling on multi-step security tasks. CRAKEN takes a standard planner-executor agent and adds a recursive retrieval pipeline at the point where the planner delegates a task to an executor. The pipeline extracts a search query from the executor's task description, retrieves from a database of 1,298 CTF writeups, grades the documents for relevance, generates a knowledge hint, grades the hint for hallucination, and rewrites the query until the hint passes or the recursion budget is exhausted. The passing hint is injected into the executor's context before it begins working. On the evaluation benchmark the best configuration, which combines the recursive retrieval with graph-structured retrieval, solves 22% of challenges, and the paper reports a 25–30% increase in the number of attack-technique categories the agent can handle.

Load-bearing premise

The headline comparison rests on one evaluation run per configuration, so the four-challenge gap that separates the knowledge-based agent from the best baseline may be within normal run-to-run variation.

Editorial extensions

If this is right

  • Knowledge injection at execution time outperforms knowledge injection at planning time (21% vs 17% for the best base model), so the retriever is most useful where fine-grained, context-specific information is needed.
  • A writeup database focused on step-by-step operational knowledge beats code-snippet and payload databases (21% vs 17.5% and 16%), meaning the content type drives the gain.
  • Graph-based retrieval adds two additional solved challenges over classic vector retrieval while keeping the average cost comparable.
  • The retrieval system is modular and can be attached to any planner-executor agent without retraining, so the paper positions it as a general upgrade path for knowledge-intensive automation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own transition analysis shows that 72.7% of generated hints fail hallucination grading, which suggests generation quality, not document retrieval, is the main bottleneck; improving the generator could yield a larger gain than expanding the database.
  • Writeup-driven knowledge injection should transfer to other domains with a canon of worked solutions, such as exploit development, malware analysis, or competitive programming, where a 'writeup' culture already exists.
  • Because the cost increase per solved challenge is moderate, the technique could make smaller, cheaper LLMs competitive when paired with a strong retriever, a configuration the paper tests only lightly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript introduces CRAKEN, a planner-executor multi-agent system for CTF solving that augments the D-CIPHER agent with a Self-RAG/Graph-RAG retrieval pipeline over curated CTF writeups, payloads, and code. The retrieval process decomposes executor context into queries, applies iterative retrieval, grading, and query rewriting, and injects the resulting knowledge hints at task-delegation time. The evaluation on NYU CTF Bench reports 22% solved with Graph-RAG on Claude 3.5 Sonnet versus 19% for D-CIPHER, and the authors claim state-of-the-art performance with a modest cost increase. Additional results cover MITRE ATT&CK technique coverage and ablations over RAG type, knowledge database, planning versus execution injection, and mixed LLM configurations. The framework and datasets are released as open source.

Significance. If the reported gains are reproducible, CRAKEN is a useful modular engineering contribution: it integrates retrieval into an existing agent framework, releases an open dataset, and the ablation set is thoughtful and informative. The explicit exclusion of CSAW writeups from the knowledge database is a reasonable anti-leakage measure. However, the central state-of-the-art claim is not currently established because the headline comparison rests on a single run per configuration without variance information, the cost accounting is internally inconsistent, and the causal role of the retrieved knowledge is not separated from prompt-formatting or pretraining effects. The main value at this stage is the architecture and the retrieval ablations, not the point estimate of 22%.

major comments (4)
  1. [Table 2, Section 5, Section 6] The state-of-the-art claim depends on a single run per configuration. With 200 challenges and solve rates near 0.2, the standard error of the difference between two independent runs is on the order of 8 solved challenges (about 4 percentage points); the observed four-challenge gap between D-CIPHER (38 solved, 19%) and Graph-RAG CRAKEN (44 solved, 22%) is therefore within one standard error. The paper reports no seeds, temperature settings, confidence intervals, or significance tests, and Section 5.1 states the Graph-RAG improvement as exactly two additional named challenges. Please repeat the main comparisons over multiple seeds (and ideally over challenge subsamples), report variance, or explicitly weaken the SOTA claim to a single-run point estimate.
  2. [Section 5.1, Table 2, Section 6] The cost analysis is internally inconsistent. Table 2 lists the default CRAKEN/Claude 3.5 Sonnet row at $0.68 and the Graph-RAG row at $0.86, but Section 5.1 says Graph-RAG's cost "matches the CRAKEN default configuration, i.e., $0.82," and the mixture-of-LLMs subsection says the default setup solves 21.0% at $0.80. The conclusion's "average cost increase of $0.34" is consistent with $0.52 to $0.86 but not with the stated default cost. Please correct the cost figures in the text and table, and state whether the reported cost is per challenge, per solved challenge, or total API spend.
  3. [Section 3, Section 5.1] The paper's central mechanism claim is that CRAKEN gains capability by accessing knowledge beyond the model's training data. This is not tested: the writeup database consists of public GitHub and Hugging Face texts, and the LLM may already know many of these techniques from pretraining. The higher performance of the writeup database over the code and payload databases could equally come from retrieval quality, prompt formatting, or the model's ability to recall similar public solutions. A control condition, such as retrieving non-informative or scrambled contexts through the same pipeline, or evaluating on challenges whose writeups are held out and were published after the model's training cutoff, would separate knowledge access from format effects. Without such a control, the "knowledge-based execution" claim is a plausible interpretation rather than an established result.
  4. [Section 5, Appendix D, Table 3] The claim that CRAKEN solves 25-30% more MITRE ATT&CK techniques than prior work is based on the "Total" row of Table 3, but that row sums challenge-technique occurrences over TIDs, not the number of distinct techniques solved. Many TIDs appear with zero solves, and the same technique can be counted multiple times across challenges. The phrasing "more techniques" therefore overstates technique breadth. Please report distinct TIDs solved, and clarify that the current totals are occurrence counts rather than unique techniques.
minor comments (5)
  1. [Figure 3, Section 5] The Venn diagram labels "CRAKEN (42)" while Table 2 reports 44 solved challenges for Graph-RAG; this is not necessarily an inconsistency if the figure refers to the default CRAKEN configuration, but the paper never says so. Label the configuration explicitly and explain the "Total (51)" value in the figure.
  2. [Algorithm 1] The line "if HALLUCINATION GRADER (a, R) then continue" is ambiguous: if the grader returns True for hallucinated content, the pseudocode should state that convention explicitly; as written, it appears to retry on grounded answers. Clarify the return semantics of the grader.
  3. [Section 4, Table 2] The metric labeled "$ cost" is described in Section 4 as both "average cost per solved CTF" and "total dollar cost of LLM API calls." Please define the denominator and state clearly which quantity is reported in Table 2.
  4. [Appendix D, Table 3] The table title contains a typo ("CRACKEN" instead of "CRAKEN"), and the paper alternates between "EniGMA" and "EnIGMA" in the text and tables; a careful proofread is needed.
  5. [Reproducibility] Please provide the exact repository commit, configuration files, and API model snapshots used for the Graph-RAG 22% run, so that the single reported number can be reproduced independently.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CRAKEN's claimed gains are empirical comparisons against an external benchmark and a reproduced no-RAG baseline, not derivations that reduce to their inputs.

full rationale

This is a systems paper whose central claims are empirical evaluation results, not derivations from definitions or fitted equations. The headline result (22% with Graph-RAG vs 19% for D-CIPHER on NYU CTF Bench) is measured on an external benchmark, with D-CIPHER results reproduced in Table 2 rather than merely cited. The knowledge database deliberately excludes CSAW writeups because CSAW challenges appear in NYU CTF Bench, which is an explicit anti-leakage step rather than an input smuggled into the output. The use of D-CIPHER as the underlying multi-agent framework is a real ablation control: CRAKEN adds retrieval to an otherwise unchanged agent, so the comparison directly tests the retrieval contribution, and the paper even reports configurations where CRAKEN performs worse than D-CIPHER (e.g., GPT-4.1), showing the outcome is not forced. The MITRE ATT&CK mapping is inherited from the authors' prior D-CIPHER work, but it is a fixed annotation applied to both systems' solved challenges, so it does not make CRAKEN's capability claim equivalent to the mapping itself. The self-citations to D-CIPHER, NYU CTF Bench, and the MITRE mapping are normal and not load-bearing for the empirical conclusion. Concerns about single-run evaluation and run-to-run variance are statistical robustness issues, not circularity, and therefore do not raise the circularity score.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The framework has several hand-set retrieval hyperparameters and relies on domain assumptions about benchmark validity and run stability.

free parameters (5)
  • chunk_size = 4096
    Vector retrieval chunk size, chosen by hand. Affects whether relevant writeup content is returned.
  • chunk_overlap = 100
    Overlap between consecutive chunks in the vector index, a hand-set configuration choice.
  • max_recursion_depth = dM (exact value not reported)
    Limits the Self-RAG iteration loop in Algorithm 1; affects how often failed retrievals can be rewritten.
  • max_cost_budget = $3.0 per challenge
    Hard budget for API calls per CTF. Since many agents exit on Max Cost, this budget partly determines the solve counts.
  • max_rounds = not reported in text
    Borrowed from D-CIPHER's agent loop; agents exit on Max Rounds, which affects solve rates.
assumptions (5)
  • domain assumption CTF solve rate is a valid proxy for cybersecurity capability
    The paper evaluates CRAKEN purely by solved CTF flags and MITRE technique counts; this assumes CTF performance transfers to real-world defensive and offensive security.
  • domain assumption NYU CTF Bench is a representative and fair benchmark for comparing CTF agents
    The benchmark was created by the same research group (NYU), and the current paper is an evaluation on that benchmark. Its use is standard but introduces ownership bias.
  • domain assumption Excluding CSAW writeups from the knowledge database prevents benchmark leakage
    The authors exclude CSAW CTF writeups because NYU CTF Bench is built from CSAW challenges. This assumes no other benchmark challenges or similar writeups leak partial solutions through the remaining 1,298 writeups.
  • domain assumption The D-CIPHER MITRE ATT&CK mapping is accurate and reusable
    The ATT&CK technique mapping is taken from their prior D-CIPHER paper, and CRAKEN's technique coverage is measured against that mapping.
  • domain assumption LLM API responses are deterministic enough that single runs are representative
    All experiments appear to be single runs with stochastic LLM sampling, yet conclusions are stated without variance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution." pith.science (2026). https://pith.science/paper/YIB3VNHI

@misc{pith2026250517107,
  author       = {Pith},
  title        = {Pith review of: CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YIB3VNHI}},
  note         = {Machine review of arXiv:2505.17107}
}
read the original abstract

Large Language Model (LLM) agents can automate cybersecurity tasks and can adapt to the evolving cybersecurity landscape without re-engineering. While LLM agents have demonstrated cybersecurity capabilities on Capture-The-Flag (CTF) competitions, they have two key limitations: accessing latest cybersecurity expertise beyond training data, and integrating new knowledge into complex task planning. Knowledge-based approaches that incorporate technical understanding into the task-solving automation can tackle these limitations. We present CRAKEN, a knowledge-based LLM agent framework that improves cybersecurity capability through three core mechanisms: contextual decomposition of task-critical information, iterative self-reflected knowledge retrieval, and knowledge-hint injection that transforms insights into adaptive attack strategies. Comprehensive evaluations with different configurations show CRAKEN's effectiveness in multi-stage vulnerability detection and exploitation compared to previous approaches. Our extensible architecture establishes new methodologies for embedding new security knowledge into LLM-driven cybersecurity agentic systems. With a knowledge database of CTF writeups, CRAKEN obtained an accuracy of 22% on NYU CTF Bench, outperforming prior works by 3% and achieving state-of-the-art results. On evaluation of MITRE ATT&CK techniques, CRAKEN solves 25-30% more techniques than prior work, demonstrating improved cybersecurity capabilities via knowledge-based execution. We make our framework open source to public https://github.com/NYU-LLM-CTF/nyuctf_agents_craken.

Figures

Figures reproduced from arXiv: 2505.17107 by the authors.

Figure 1
Figure 1. Architecture of CRAKEN composed of two parts: 1. Planner-Executor based [ [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Graph-RAG Retrieval Implementation. We implement the retrieval process us￾ing the LangChain framework. We integrate Milvus [22] for efficient vector-based similarity search, and Neo4j [24] for managing graph knowledge relationships for Graph￾RAG. This technological foundation enables CRAKEN to decompose complex tasks, retrieve domain-specific knowl￾edge, and execute multi-step solutions across diverse secu￾rity chal… view at source ↗
Figure 3
Figure 3. Overlap of CTFs solved by three agents on NYU CTF Bench. Retrieval Process Analysis [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Transition diagram visualizing the RAG process. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: CRAKEN exit analysis by category on Claude 3.5 Sonnet, Claude 3.7 Sonnet, GPT 4o and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Ransomware 3.0: Self-Composing and LLM-Orchestrated

    cs.CR 2025-08 conditional novelty 8.0 of 10

    A prototype LLM-orchestrated ransomware successfully executes reconnaissance, payload selection, encryption/exfiltration/destruction, and personalized extortion across three environments, with open-source models.

  2. Veritas: Grounding LLM Agents for Reliable Vulnerability Reasoning over Stripped Binaries

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    Veritas detects out-of-bounds vulnerabilities in stripped binaries at 90% recall by grounding LLM reasoning in static witness-backed flows and runtime validation.

  3. Determinants and Limits of LLM Security-Tool Orchestration: A Study with HexStrike-AI

    cs.SE 2026-07 conditional novelty 6.0 of 10

    For a fixed DeepSeek model, the MCP client alone produced a 2.1× solve-rate gap on HexStrike-AI CTF trials, and bundled tool/behavior fixes lifted overall success from 55.4% to 72.0%.

Reference graph

Works this paper leans on

69 extracted references · 42 canonical work pages · cited by 3 Pith papers

  1. [1]

    Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik Narasimhan, Ramesh Karri, and Ofir Press

    Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner, Sofija Jancheska, John Yang, Carlos E. Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik Narasimhan, Ramesh Karri, and Ofir Press. Interactive tools substantially assist LM agents in finding security vulnerabilities, 2...

    arXiv 2025

  2. [2]

    Sumona, Masud Mohiuddin, Md Tanzim Reza, and Md Mizanur Rahman

    Vishwanath Akuthota, Raghunandan Kasula, Sabiha T. Sumona, Masud Mohiuddin, Md Tanzim Reza, and Md Mizanur Rahman. Vulnerability detection and monitoring using LLM. In Women in Engineering Conference on Electrical and Computer Engineering, pages 309–314. IEEE, 2023

    work page 2023

  3. [3]

    Self-rag: Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In International Conference on Learning Representations, 2023

    work page 2023

  4. [4]

    CyberSecEval 2: A wide-ranging cybersecurity evaluation suite for large language models, 2024

    Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, David Molnar, Spencer Whitman, and Joshua Saxe. CyberSecEval 2: A wide-ranging cybersecurity evaluation suite for large language models, 2024. URL https://arxiv.org/abs/2404.13161v1

    arXiv 2024

  5. [5]

    RepairAgent: An autonomous, LLM-based agent for program repair, 2024

    Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. RepairAgent: An autonomous, LLM-based agent for program repair, 2024. URL https://arxiv.org/abs/2403.17134v2

    arXiv 2024

  6. [6]

    P. V . Sai Charan, Hrushikesh Chunduri, P. Mohan Anand, and Sandeep K Shukla. From text to mitre techniques: Exploring the malicious use of large language models for generating cyber attack payloads, 2023

    work page 2023

  7. [7]

    Using facebook’s open source capture the flag platform as a hands-on learning and assessment tool for cybersecurity education

    Rhonda Chicone et al. Using facebook’s open source capture the flag platform as a hands-on learning and assessment tool for cybersecurity education. International Journal of Conceptual Structures and Smart Applications, 6(1):18–32, 2018

    work page 2018

  8. [8]

    DARPA cyber grand challenge

    DARPA. DARPA cyber grand challenge. https://www.darpa.mil/program/ cyber-grand-challenge, 2016. URL https://www.darpa.mil/program/ cyber-grand-challenge

    work page 2016

Show all 69 references
  1. [9]

    DARPA AIxCC

    DARPA. DARPA AIxCC. https://aicyberchallenge.com/about/, 2024. URL https: //aicyberchallenge.com/about/

    2024

  2. [10]

    PentestGPT: An LLM-empowered automatic penetration testing tool, 2024

    Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. PentestGPT: An LLM-empowered automatic penetration testing tool, 2024. URL https://arxiv.org/abs/2308.06782

    2024 arXiv

  3. [11]

    Vul-rag: Enhancing llm-based vulnerability detection via knowledge-level rag

    Xueying Du, Geng Zheng, Kaixin Wang, Jiayi Feng, Wentai Deng, Mingwei Liu, Bihuan Chen, Xin Peng, Tao Ma, and Yiling Lou. Vul-rag: Enhancing llm-based vulnerability detection via knowledge-level rag. arXiv preprint arXiv:2406.11147, 2024

    2024 arXiv

  4. [12]

    From rag to multi-agent systems: A survey of modern approaches in llm development, 2025

    Gustavo de Aquino e Aquino, Nádila da Silva de Azevedo, Leandro Youiti Silva Okimoto, Leonardo Yuto Suzuki Camelo, Hendrio Luis de Souza Bragança, Rubens Fernandes, Andre Printes, Fábio Cardoso, Raimundo Gomes, and Israel Gondres Torné. From rag to multi-agent systems: A surve...

    2025

  5. [13]

    Outside the comfort zone: Analysing LLM capabilities in software vulnerability detection

    Yuejun Guo, Constantinos Patsakis, Qiang Hu, Qiang Tang, and Fran Casino. Outside the comfort zone: Analysing LLM capabilities in software vulnerability detection. In European symposium on research in computer security, pages 271–289. Springer, 2024. 10

    2024

  6. [14]

    Grag: Graph retrieval-augmented generation

    Yuntong Hu, Zhihan Lei, Zheng Zhang, Bo Pan, Chen Ling, and Liang Zhao. Grag: Graph retrieval-augmented generation. arXiv preprint arXiv:2405.16506, 2024

    2024 arXiv

  7. [15]

    Matei, and Elisa Bertino

    Diane Jackson, Sorin A. Matei, and Elisa Bertino. Artificial intelligence ethics education in cybersecurity: Challenges and opportunities: a focus group report, 2023

    2023

  8. [16]

    A study on the implementation method of an agent-based advanced rag system using graph

    Cheonsu Jeong. A study on the implementation method of an agent-based advanced rag system using graph. arXiv preprint arXiv:2407.19994, 2024

    2024 arXiv

  9. [17]

    Active retrieval augmented generation

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992, 2023

    2023

  10. [18]

    Long-context llms meet rag: Overcoming challenges for long inputs in rag

    Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. Long-context llms meet rag: Overcoming challenges for long inputs in rag. In The Thirteenth International Conference on Learning Representations, 2024

    2024

  11. [19]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems...

    2020

  12. [20]

    Attention is all you need for LLM-based code vulnerability localization, 2024

    Yue Li, Xiao Li, Hao Wu, Yue Zhang, Xiuzhen Cheng, Sheng Zhong, and Fengyuan Xu. Attention is all you need for LLM-based code vulnerability localization, 2024. URL https: //arxiv.org/abs/2410.15288v1

    2024 arXiv

  13. [21]

    GRACE: Empowering LLM-based software vulnerability detection with graph structure and in-context learning

    Guilong Lu, Xiaolin Ju, Xiang Chen, Wenlong Pei, and Zhilong Cai. GRACE: Empowering LLM-based software vulnerability detection with graph structure and in-context learning. Journal of Systems and Software, 212:112031, 2024

    2024

  14. [22]

    Milvus | High-Performance Vector Database Built for Scale

    Milvus. Milvus | High-Performance Vector Database Built for Scale. https://milvus.io/,

  15. [23]

    HackSynth: LLM agent and evaluation framework for autonomous penetration testing, 2024

    Lajos Muzsai, David Imolai, and András Lukács. HackSynth: LLM agent and evaluation framework for autonomous penetration testing, 2024. URL https://arxiv.org/abs/2412. 01778v1

    2024

  16. [24]

    Neo4j Graph Database & Analytics

    Neo4j, Inc. Neo4j Graph Database & Analytics. https://neo4j.com/, 2025. URL https: //neo4j.com/

    2025

  17. [25]

    Graph retrieval-augmented generation: A survey

    Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. Graph retrieval-augmented generation: A survey. arXiv preprint arXiv:2408.08921, 2024

    2024 arXiv

  18. [26]

    Friend or foe – the impact of ChatGPT on capture the flag competitions

    Heloise Pieterse. Friend or foe – the impact of ChatGPT on capture the flag competitions. In International Conference on Cyber Warfare and Security, volume 19, pages 268–276, 2024

    2024

  19. [27]

    Earp, Sven Nyholm, John Danaher, Nikolaj Møller, Hi- lary Bowman-Smart, Joshua Hatherley, Julian Koplin, Monika Plozza, Daniel Rodger, et al

    Sebastian Porsdam Mann, Brian D. Earp, Sven Nyholm, John Danaher, Nikolaj Møller, Hi- lary Bowman-Smart, Joshua Hatherley, Julian Koplin, Monika Plozza, Daniel Rodger, et al. Generative AI entails a credit–blame asymmetry, 2023

    2023

  20. [28]

    A rag-based question-answering solution for cyber-attack investigation and attribution

    Sampath Rajapaksha, Ruby Rani, and Erisa Karafili. A rag-based question-answering solution for cyber-attack investigation and attribution. In Computer Security. ESORICS 2024 International Workshops: SECAI, DisA, CPS4CIP , and SecAssure, Bydgoszcz, Poland, September 1...

    2024

  21. [29]

    A rag-based question-answering so- lution for cyber-attack investigation and attribution

    Sampath Rajapaksha, Ruby Rani, and Erisa Karafili. A rag-based question-answering so- lution for cyber-attack investigation and attribution. In Computer Security. ESORICS 2024 International Workshops, pages 238–256, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-82362-6. 11

    2024

  22. [30]

    ISBN 978-3-031-82361-9

    Springer-Verlag. ISBN 978-3-031-82361-9. doi: 10.1007/978-3-031-82362-6_15. URL https://doi.org/10.1007/978-3-031-82362-6_15

    doi

  23. [31]

    An empirical evaluation of LLMs for solving offensive security challenges, 2024

    Minghao Shao, Boyuan Chen, Sofija Jancheska, Brendan Dolan-Gavitt, Siddharth Garg, Ramesh Karri, and Muhammad Shafique. An empirical evaluation of LLMs for solving offensive security challenges, 2024. URL https://arxiv.org/abs/2402.11814v1

    2024 arXiv

  24. [32]

    Savin, Ammar Asseri, Josiah Dykstra, Jonathan Goohs, Anthony Melaragno, and William Casey

    Georgel M. Savin, Ammar Asseri, Josiah Dykstra, Jonathan Goohs, Anthony Melaragno, and William Casey. Battle ground: Data collection and labeling of CTF games to understand human cyber operators. In Cyber Security Experimentation and Test Workshop, pages 32–40. Association for...

    2023

  25. [33]

    PentestAgent: Incorporating LLM agents to automated penetration testing, 2024

    Xiangmin Shen, Lingzhi Wang, Zhenyuan Li, Yan Chen, Wencheng Zhao, Dawei Sun, Jiashui Wang, and Wei Ruan. PentestAgent: Incorporating LLM agents to automated penetration testing, 2024. URL https://arxiv.org/abs/2411.05185v1

    2024 arXiv

  26. [34]

    NYU CTF Bench: A scalable open-source benchmark dataset for evaluating LLMs in offensive security

    Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Muhammad Shafique. NYU CTF Bench: A scalable open-source benchmark dataset for eval...

    2024

  27. [35]

    Agentic retrieval-augmented generation: A survey on agentic rag

    Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. Agentic retrieval-augmented generation: A survey on agentic rag. arXiv preprint arXiv:2501.09136, 2025

    2025 arXiv

  28. [36]

    Morse: Bridging the gap in cybersecurity expertise with retrieval augmented generation

    Marco Simoni, Andrea Saracino, Mauro Conti, et al. Morse: Bridging the gap in cybersecurity expertise with retrieval augmented generation. arXiv preprint arXiv:2407.15748, 2024

    2024 arXiv

  29. [37]

    MITRE ATT&CK

    The MITRE Corporation. MITRE ATT&CK. https://attack.mitre.org/, 2015. URL https://attack.mitre.org/

    2015

  30. [38]

    Seah, and Ee-Chien Chang

    Wesley Tann, Yuancheng Liu, Jun Heng Sim, Choon M. Seah, and Ee-Chien Chang. Using large language models for cybersecurity capture-the-flag challenges and certification questions,

  31. [39]

    D-CIPHER: Dynamic collaborative intelligent multi-agent system with planner and heterogeneous executors for offensive security,

    Meet Udeshi, Minghao Shao, Haoran Xi, Nanda Rani, Kimberly Milner, Venkata Sai Charan Putrevu, Brendan Dolan-Gavitt, Sandeep Kumar Shukla, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Muhammad Shafique. D-CIPHER: Dynamic collaborative intelligent multi-agent sy...

  32. [40]

    Benefits and pitfalls of using capture the flag games in university courses

    Jan Vykopal, Valdemar Švábenský, and Ee-Chien Chang. Benefits and pitfalls of using capture the flag games in university courses. In Technical Symposium on Computer Science Education, page 752–758. Association for Computing Machinery, 2020. doi: 10.1145/3328778.3366893. URL ht...

    2020

  33. [41]

    Hacking CTFs with plain agents, 2024

    Rustem Turtayev, Artem Petrov, Dmitrii V olkov, and Denis V olk. Hacking CTFs with plain agents, 2024. URL https://arxiv.org/abs/2412.02776v1

    2024 arXiv

  34. [42]

    Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Annual Meeting of the A...

    2023 doi

  35. [43]

    URL https://arxiv.org/abs/2502.10931v2

    arXiv

  36. [44]

    Coderag-bench: Can retrieval augment code generation? arXiv preprint arXiv:2406.14497, 2024

    Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F Xu, Yiqing Xie, Graham Neubig, and Daniel Fried. Coderag-bench: Can retrieval augment code generation? arXiv preprint arXiv:2406.14497, 2024

    2024 arXiv

  37. [45]

    CYBERSECEV AL 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models, 2024

    Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, Vlad Ionescu, Yue Li, and Joshua Saxe. CYBERSECEV AL 3: Advancing the evaluation of cybersecurity risks and capabilitie...

    2024 arXiv

  38. [46]

    Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT

    Chunqiu Steven Xia and Lingming Zhang. Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. In International Symposium on Software Testing and Analysis, pages 819–831. Association for Computing Machinery, 2024

    2024

  39. [47]

    Searching for best practices in retrieval- augmented generation

    Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, et al. Searching for best practices in retrieval- augmented generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language P...

    2024

  40. [48]

    Autopwn: Artifact- assisted heap exploit generation for ctf pwn competitions

    Dandan Xu, Kai Chen, Miaoqian Lin, Chaoyang Lin, and Xiaofeng Wang. Autopwn: Artifact- assisted heap exploit generation for ctf pwn competitions. IEEE Transactions on Information Forensics and Security, 19:293–306, 2024. doi: 10.1109/TIFS.2023.3322319

    2024

  41. [49]

    Unveiling security, privacy, and ethical concerns of ChatGPT

    Xiaodong Wu, Ran Duan, and Jianbing Ni. Unveiling security, privacy, and ethical concerns of ChatGPT. Journal of Information and Intelligence, 2(2):102–115, 2024. doi: https://doi.org/ 10.1016/j.jiixd.2023.10.007. URL https://www.sciencedirect.com/science/article/ pii/S2949715...

    2024 doi

  42. [50]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models, 2022. URL https: //openreview.net/forum?id=tvI4u1ylcqs

    2022

  43. [51]

    ReWOO: Decoupling reasoning from observations for efficient augmented language models,

    Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, and Dongkuan Xu. ReWOO: Decoupling reasoning from observations for efficient augmented language models,

  44. [52]

    URL https://arxiv.org/abs/2305.18323v1

    arXiv

  45. [53]

    step-back

    Chengshuai Zhao, Garima Agrawal, Tharindu Kumarage, Zhen Tan, Yuli Deng, Ying-Chih Chen, and Huan Liu. Ontology-aware rag for improved question-answering in cybersecurity education, 2024. URL https://arxiv.org/abs/2412.14191. 13 A RAG Algorithms Supported in CRAKEN Beyond Self...

    2024

  46. [54]

    Narasimhan

    John Yang, Akshara Prabhakar, Shunyu Yao, Kexin Pei, and Karthik R. Narasimhan. Language agents as hackers: Evaluating cybersecurity skills with capture the flag, 2023. URL https: //openreview.net/forum?id=KOZwk7BFc3

    2023

  47. [56]

    Zhang, Neil Perry, Riya Dulepet, Eliot Jones, Justin W

    Andy K. Zhang, Neil Perry, Riya Dulepet, Eliot Jones, Justin W. Lin, Joey Ji, Celeste Menders, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Mike Yang, Teddy Zhang, Rishi Allu...

    2024 arXiv

  48. [57]

    An empirical study of automated vulnerability localization with large language models, 2024

    Jian Zhang, Chong Wang, Anran Li, Weisong Sun, Cen Zhang, Wei Ma, and Yang Liu. An empirical study of automated vulnerability localization with large language models, 2024. URL https://arxiv.org/abs/2404.00287v1

    2024

  49. [59]

    A concise task description: What is the primary challenge objective?

  50. [60]

    A search query: What specific technical information would help solve this challenge?

  51. [61]

    yes" or “no

    Keywords: Extract 3-5 relevant keywords related to techniques needed. {format_instructions} You should only output the dictionary or json format content. No other content is allowed in the answer. default task: Please continue based on your best judgment. Retrieval Prompts RAG...

  52. [62]

    dogestore

    Predictable or Constant Initialization Vectors (IVs) As shown in the Google CTF “dogestore” challenge: iv = get_iv(); openssl::symm::decrypt( openssl::symm::cipher::aes_256_ctr(), &key, &some(&iv, data) ) The vulnerability here is that the IV remains constant across different ...

  53. [63]

    Keystream Reuse Vulnerability From the WhiteHat challenge about A5/1 cipher: • The implementation only changed the timestamp every 30 seconds • Without a counter (like in CTR mode), the keystream remained constant for that period • This allowed attackers to recover the keystre...

  54. [64]

    hack-tac-toe

    Improper State Maintenance in RC4 The “hack-tac-toe” challenge demonstrates another critical weakness: 400ea7a58971b0f78fa9c6ed298764a8 400ea7a58971b0f78fa9c6ed298764a8 400ea7a58971b0f78fa9c6ed298764a8 The keystream repeats after just 16 bytes, meaning: • The internal state wa...

  55. [65]

    journey2

    Implementation Flaws and Code Tampering From the “journey2” challenge, a vulnerability where a single byte in the RC4 algorithm was modified: - k = s[(s[i] + s[j]) % 256] + k = s[(s[j] - s[i]) % 256] This shows: 16 • Even a small change in state management (+ vs -) completely ...

  56. [66]

    Lack of state updates: Rust implementations might fail to properly update internal states between encryption operations

  57. [67]

    Improper key scheduling: Many RC4 implementations do not correctly initialize or refresh the state

  58. [68]

    Memory safety issues: Rust provides memory safety, but misuse of state arrays can still cause logic flaws

  59. [69]

    Static or predictable seeds: Without proper randomness, cipher security is severely weak- ened. Recommended Mitigations • Ensure IVs are never reused for the same key • Implement proper counters for stream ciphers • Verify the correctness of key scheduling algorithms • Use cry...

    2017

  60. [2023]

    URL https://arxiv.org/abs/2308.10443

    arXiv

  61. [2025]

    URL https://milvus.io/

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.