{"id":"c6b95e6e-b594-49de-badd-70711983b891","arxiv_id":"2505.17107","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CRAKEN, an LLM agent combining Self-RAG and Graph-RAG over a CTF writeup database, solves 22% of NYU CTF Bench challenges, three percentage points above the prior D-CIPHER baseline.","lead":"The paper introduces CRAKEN, an LLM agent that adds a retrieval-augmented knowledge base of CTF writeups to a planner-executor system, raising solve rate on NYU CTF Bench from 19% (D-CIPHER) to 22%. A smart generalist might read it to see whether giving AI agents access to a searchable database of past exploits meaningfully improves their security-testing skills.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 3% SOTA gain rests on single unrepeated runs; Table 2's 22% (44 solved) also conflicts with Figure 3's CRAKEN (42), so the four-to-six challenge difference against D-CIPHER is within plausible LLM-agent run-to-run variance.","rationale":"The reader's weakest assumption is the same load-bearing concern I identify: the headline comparison is built from single runs of stochastic LLM agents, so the 3% improvement over D-CIPHER may be noise. I agree with that assessment. I additionally note a concrete internal inconsistency that strengthens the concern: Table 2's Graph-RAG row of 22.0% implies 44 solved challenges, while Figure 3 and Appendix E show CRAKEN with 42 solved, meaning the paper's own reporting is not self-consistent about which configuration produced the headline number. This does not change the appropriate verdict, which remains CONDITIONAL: the framework is coherent, open-sourced, and well ablated, but the SOTA claim needs repeated runs with variance reporting, a clarified baseline provenance, and a resolved count discrepancy. Because the reader already conditioned acceptance on these additions, no verdict adjustment is needed.","tokens_in":19249,"tokens_out":10272,"duration_ms":88753,"concrete_test":"Run D-CIPHER and CRAKEN with Graph-RAG, both using Claude 3.5 Sonnet, five independent times each on the full 200-challenge NYU CTF Bench under the same harness, Docker environment, API settings, and budget cap, logging temperature and seeds; then report per-run solved counts and a bootstrap 95% confidence interval for the difference in solve rate. Also reconcile the solved-count discrepancy by re-deriving Table 2, Figure 3, and Appendix E from the same run logs. If the confidence interval includes zero, or if the 44-versus-42 mismatch persists, the abstract's state-of-the-art claim should be qualified accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, repeated in the abstract and Section 6, is that CRAKEN with Graph-RAG reaches 22% on NYU CTF Bench versus D-CIPHER's 19%, a 3% state-of-the-art improvement. Table 2 reports one run per configuration and no seed or temperature information. For a 200-challenge benchmark with per-challenge solve probability near 0.2, the standard error of the difference between two independent runs is roughly 0.04, so a 0.03 or even 0.06 gap is less than two standard errors and comfortably within run-to-run noise. The report is internally inconsistent as well: Table 2's 22% corresponds to 44 solved challenges, while Figure 3 and Appendix E depict CRAKEN at 42 solved, and Section 5.1 says Graph-RAG adds only two challenges over the classic-RAG default. Either the figure is stale or the 22% headline is not the configuration being analyzed. Because the claimed advantage over D-CIPHER is exactly the kind of small, stochastic difference that single-shot LLM-agent evaluations cannot establish, the state-of-the-art assertion is not currently supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces CRAKEN, a planner-executor multi-agent system for CTF solving that augments the D-CIPHER agent with a Self-RAG/Graph-RAG retrieval pipeline over curated CTF writeups, payloads, and code. The retrieval process decomposes executor context into queries, applies iterative retrieval, grading, and query rewriting, and injects the resulting knowledge hints at task-delegation time. The evaluation on NYU CTF Bench reports 22% solved with Graph-RAG on Claude 3.5 Sonnet versus 19% for D-CIPHER, and the authors claim state-of-the-art performance with a modest cost increase. Additional results cover MITRE ATT&CK technique coverage and ablations over RAG type, knowledge database, planning versus execution injection, and mixed LLM configurations. The framework and datasets are released as open source.","tokens_in":19487,"tokens_out":9526,"duration_ms":83712,"significance":"If the reported gains are reproducible, CRAKEN is a useful modular engineering contribution: it integrates retrieval into an existing agent framework, releases an open dataset, and the ablation set is thoughtful and informative. The explicit exclusion of CSAW writeups from the knowledge database is a reasonable anti-leakage measure. However, the central state-of-the-art claim is not currently established because the headline comparison rests on a single run per configuration without variance information, the cost accounting is internally inconsistent, and the causal role of the retrieved knowledge is not separated from prompt-formatting or pretraining effects. The main value at this stage is the architecture and the retrieval ablations, not the point estimate of 22%.","major_comments":[{"comment":"The state-of-the-art claim depends on a single run per configuration. With 200 challenges and solve rates near 0.2, the standard error of the difference between two independent runs is on the order of 8 solved challenges (about 4 percentage points); the observed four-challenge gap between D-CIPHER (38 solved, 19%) and Graph-RAG CRAKEN (44 solved, 22%) is therefore within one standard error. The paper reports no seeds, temperature settings, confidence intervals, or significance tests, and Section 5.1 states the Graph-RAG improvement as exactly two additional named challenges. Please repeat the main comparisons over multiple seeds (and ideally over challenge subsamples), report variance, or explicitly weaken the SOTA claim to a single-run point estimate.","section":"Table 2, Section 5, Section 6"},{"comment":"The cost analysis is internally inconsistent. Table 2 lists the default CRAKEN/Claude 3.5 Sonnet row at $0.68 and the Graph-RAG row at $0.86, but Section 5.1 says Graph-RAG's cost \"matches the CRAKEN default configuration, i.e., $0.82,\" and the mixture-of-LLMs subsection says the default setup solves 21.0% at $0.80. The conclusion's \"average cost increase of $0.34\" is consistent with $0.52 to $0.86 but not with the stated default cost. Please correct the cost figures in the text and table, and state whether the reported cost is per challenge, per solved challenge, or total API spend.","section":"Section 5.1, Table 2, Section 6"},{"comment":"The paper's central mechanism claim is that CRAKEN gains capability by accessing knowledge beyond the model's training data. This is not tested: the writeup database consists of public GitHub and Hugging Face texts, and the LLM may already know many of these techniques from pretraining. The higher performance of the writeup database over the code and payload databases could equally come from retrieval quality, prompt formatting, or the model's ability to recall similar public solutions. A control condition, such as retrieving non-informative or scrambled contexts through the same pipeline, or evaluating on challenges whose writeups are held out and were published after the model's training cutoff, would separate knowledge access from format effects. Without such a control, the \"knowledge-based execution\" claim is a plausible interpretation rather than an established result.","section":"Section 3, Section 5.1"},{"comment":"The claim that CRAKEN solves 25-30% more MITRE ATT&CK techniques than prior work is based on the \"Total\" row of Table 3, but that row sums challenge-technique occurrences over TIDs, not the number of distinct techniques solved. Many TIDs appear with zero solves, and the same technique can be counted multiple times across challenges. The phrasing \"more techniques\" therefore overstates technique breadth. Please report distinct TIDs solved, and clarify that the current totals are occurrence counts rather than unique techniques.","section":"Section 5, Appendix D, Table 3"}],"minor_comments":[{"comment":"The Venn diagram labels \"CRAKEN (42)\" while Table 2 reports 44 solved challenges for Graph-RAG; this is not necessarily an inconsistency if the figure refers to the default CRAKEN configuration, but the paper never says so. Label the configuration explicitly and explain the \"Total (51)\" value in the figure.","section":"Figure 3, Section 5"},{"comment":"The line \"if HALLUCINATION GRADER (a, R) then continue\" is ambiguous: if the grader returns True for hallucinated content, the pseudocode should state that convention explicitly; as written, it appears to retry on grounded answers. Clarify the return semantics of the grader.","section":"Algorithm 1"},{"comment":"The metric labeled \"$ cost\" is described in Section 4 as both \"average cost per solved CTF\" and \"total dollar cost of LLM API calls.\" Please define the denominator and state clearly which quantity is reported in Table 2.","section":"Section 4, Table 2"},{"comment":"The table title contains a typo (\"CRACKEN\" instead of \"CRAKEN\"), and the paper alternates between \"EniGMA\" and \"EnIGMA\" in the text and tables; a careful proofread is needed.","section":"Appendix D, Table 3"},{"comment":"Please provide the exact repository commit, configuration files, and API model snapshots used for the Graph-RAG 22% run, so that the single reported number can be reproduced independently.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competent engineering contribution, but the D-CIPHER baseline is from largely the same group and the MITRE mapping is inherited from that prior work; the editor may wish to ensure the comparison is independently checked. The main risk is overclaiming state-of-the-art status from a single stochastic run. A revision that adds repeated runs, corrects the cost accounting, and clarifies the MITRE metric would make the paper's contribution substantially stronger."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the headline result — 22% vs D-CIPHER's 19% — is not statistically established, and the paper's own numbers don't agree on whether the winning configuration solves 42 or 44 challenges. That's a real problem. But the engineering contribution is genuine: an open-sourced framework that injects retrieval-based knowledge at executor time, a new writeup corpus with CSAW excluded, and ablations that separate the effects of knowledge source, injection point, and model.\n\nWhat's new and good: combining Self-RAG and Graph-RAG in a planner-executor CTF agent is a sensible application of known techniques, and they did the structural things right. The writeup-only database beats code and payload mixes, execution-time injection beats planning-time injection, and the leakage control (excluding CSAW writeups from the corpus) is the right instinct.\n\nWhere it's soft: Table 2 gives one run per configuration, no seeds, no confidence intervals. For 200 challenges with solve rate ~0.2, the standard error on the difference between two independent runs is around 0.05, so a 3-point gap is within noise. The paper also contradicts itself: Table 2 says 22% (44 solved) for Graph-RAG, Figure 3 and Appendix E show CRAKEN at 42 solved, and Section 5.1 says Graph-RAG adds only two challenges over classic RAG. That's consistent with 44 vs 42, but then Figure 3's 42 is not the reported best config. Section 5.1 also quotes the default config cost at $0.82 while Table 2 says $0.68. These need to be fixed. The MITRE ATT&CK claim (25-30% more techniques) relies on D-CIPHER's mapping and the same single run, so it inherits the same fragility.\n\nBottom line: this deserves a serious referee, not a desk reject, but the manuscript is not ready in present form. The referee should ask for multi-seed runs with variance reporting, a consistent result table, and a direct leakage test between the writeup corpus and the benchmark. The core idea is plausible and the artifact is reusable; it just needs a sound evaluation to support its own claims.","headline":"The 3% SOTA claim is not statistically established and the paper's own numbers disagree, but the open-sourced knowledge-retrieval recipe for CTF agents is a real contribution worth a careful revision.","tokens_in":20127,"tokens_out":5113,"would_cite":true,"duration_ms":40464,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Giving a planner-executor LLM agent an iterative retrieval loop over curated CTF solution writeups raises its solve rate on a 200-challenge benchmark from 19% to 22%, at an average cost increase of $0.34 per solved challenge.","keywords":["cybersecurity agents","capture-the-flag","retrieval-augmented generation","self-RAG","graph-RAG","knowledge injection","LLM agents","offensive security"],"falsifier":"Run the same two configurations twenty times each with different random seeds on the 200-challenge benchmark and compare the distribution of solved counts; the claimed improvement is real only if the knowledge-based agent's count is above the baseline's upper quantile. Also swap the writeup database for a scrambled version and see if the 22% solve rate persists.","tokens_in":19050,"feed_emoji":"🛡️","tokens_out":5617,"duration_ms":46512,"temperature":0.7,"pith_summary":"Cybersecurity LLM agents are limited by the fact that their training data stops at a cutoff date and does not contain step-by-step operational exploit knowledge. This paper claims that the gap can be closed without retraining by wiring a retrieval-augmented generation loop into a planner-executor agent: decompose each delegated task into a focused query, retrieve relevant CTF solution writeups from a curated database, grade the retrieved material and the generated hint, and rewrite the query if either check fails. On a public 200-challenge CTF benchmark the system solves 42 of 200 challenges (22%), four more than the previous best agent (38, or 19%), with an average cost increase of about $0.34 per solved challenge. If the pattern holds, it gives a modular template for injecting fresh domain knowledge into LLM-driven task planning systems generally.","feed_headline":"With writeup retrieval, LLM agent solves 22% of CTF benchmark","feed_subtitle":"A curated knowledge database added at execution time beats the previous best agent by four solved challenges.","key_machinery":"The load-bearing object is the recursive Self-RAG loop: retrieve, grade relevance, generate, grade for hallucination, rewrite, and retry, capped by a maximum recursion depth. A second mechanism, Graph-RAG, converts retrieved writeups into semantic triplets and a knowledge graph, so retrieval can follow connected concepts rather than only vector similarity. The two work over a curated knowledge database of 1,298 CTF writeups in markdown form, intentionally excluding writeups from the same competition family as the benchmark to reduce contamination. The retrieval trigger is the delegation step: the executor receives a knowledge hint, which is what lets the paper attribute gains to knowledge rather than to bigger prompts.","core_discovery":"The paper's central claim is that knowledge injection at execution time, not better tools or more context, is what pushes LLM agents past their current ceiling on multi-step security tasks. CRAKEN takes a standard planner-executor agent and adds a recursive retrieval pipeline at the point where the planner delegates a task to an executor. The pipeline extracts a search query from the executor's task description, retrieves from a database of 1,298 CTF writeups, grades the documents for relevance, generates a knowledge hint, grades the hint for hallucination, and rewrites the query until the hint passes or the recursion budget is exhausted. The passing hint is injected into the executor's context before it begins working. On the evaluation benchmark the best configuration, which combines the recursive retrieval with graph-structured retrieval, solves 22% of challenges, and the paper reports a 25–30% increase in the number of attack-technique categories the agent can handle.","pith_inferences":["The paper's own transition analysis shows that 72.7% of generated hints fail hallucination grading, which suggests generation quality, not document retrieval, is the main bottleneck; improving the generator could yield a larger gain than expanding the database.","Writeup-driven knowledge injection should transfer to other domains with a canon of worked solutions, such as exploit development, malware analysis, or competitive programming, where a 'writeup' culture already exists.","Because the cost increase per solved challenge is moderate, the technique could make smaller, cheaper LLMs competitive when paired with a strong retriever, a configuration the paper tests only lightly."],"forward_implications":["Knowledge injection at execution time outperforms knowledge injection at planning time (21% vs 17% for the best base model), so the retriever is most useful where fine-grained, context-specific information is needed.","A writeup database focused on step-by-step operational knowledge beats code-snippet and payload databases (21% vs 17.5% and 16%), meaning the content type drives the gain.","Graph-based retrieval adds two additional solved challenges over classic vector retrieval while keeping the average cost comparable.","The retrieval system is modular and can be attached to any planner-executor agent without retraining, so the paper positions it as a general upgrade path for knowledge-intensive automation."],"supporting_citations":[{"why":"Supplies the 200-challenge benchmark that all solve rates and cost numbers are measured on.","marker":"[32]"},{"why":"Provides the planner-executor agent architecture CRAKEN is built on and the prior baseline it must beat.","marker":"[39]"},{"why":"Supplies the Self-RAG iterative retrieval-grading-rewriting loop that is the core of the retrieval system.","marker":"[3]"},{"why":"Supplies the graph-based retrieval method used in the best-performing configuration.","marker":"[25]"},{"why":"Prior interactive-tools agent whose solved-challenge overlap with CRAKEN is compared in the solution distribution analysis.","marker":"[1]"},{"why":"Establishes retrieval-augmented generation, the general paradigm the knowledge injection builds on.","marker":"[19]"}],"fun_headline_variants":["CTF agent hits 22% with injected writeup knowledge","Knowledge injection lifts LLM agent past prior CTF best","CRAKEN: recursive retrieval boosts LLM cyber agent to 22%","LLM agent with writeup knowledge solves 22% CTF challenges","Writeup-based knowledge execution beats prior LLM CTF agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison rests on one evaluation run per configuration, so the four-challenge gap that separates the knowledge-based agent from the best baseline may be within normal run-to-run variation.","fun_headline_variants_meta":{"raw":{"variants":["CTF agent hits 22% with injected writeup knowledge","Knowledge injection lifts LLM agent past prior CTF best","CRAKEN: recursive retrieval boosts LLM cyber agent to 22%","LLM agent with writeup knowledge solves 22% CTF challenges","Writeup-based knowledge execution beats prior LLM CTF agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1469,"prompt_tokens":984,"completion_tokens":485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":395}},"tokens_in":600,"tokens_out":485,"duration_ms":4508,"temperature":1.0,"reasoning_tokens":395,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:18:30.831705+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same two configurations twenty times each with different random seeds on the 200-challenge benchmark and compare the distribution of solved counts; the claimed improvement is real only if the knowledge-based agent's count is above the baseline's upper quantile. Also swap the writeup database for a scrambled version and see if the 22% solve rate persists.","supporting_citations":[{"cited_title":"Savin, Ammar Asseri, Josiah Dykstra, Jonathan Goohs, Anthony Melaragno, and William Casey","cited_arxiv_id":null,"evidence_quote":"Supplies the 200-challenge benchmark that all solve rates and cost numbers are measured on."},{"cited_title":"D-CIPHER: Dynamic collaborative intelligent multi-agent system with planner and heterogeneous executors for offensive security,","cited_arxiv_id":null,"evidence_quote":"Provides the planner-executor agent architecture CRAKEN is built on and the prior baseline it must beat."},{"cited_title":"Self-rag: Learning to retrieve, generate, and critique through self-reflection","cited_arxiv_id":null,"evidence_quote":"Supplies the Self-RAG iterative retrieval-grading-rewriting loop that is the core of the retrieval system."}],"review_version":1}