{"id":"35beaa0a-63cd-47b9-b565-a9347ccfeaea","arxiv_id":"2506.17644","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A benchmark and agent for CTF solving, but the agent's retrieval database appears to contain the answers to the test challenges, undermining the reported improvements.","lead":"Researchers built a 3,992-question benchmark to separate what LLMs know about security techniques from how well they apply that knowledge in CTF challenges, and then built an agent, CTFAgent, that combines retrieval and an upgraded command-line environment to solve CTF challenges.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains may be direct retrieval: the RAG database is built from write-ups of the same competitions as the test sets, and Figure 8 shows DB-Exploiting returning an exact exploit payload for a test challenge, so the 39→73 and 8→18 improvements cannot yet be attributed to the proposed…","rationale":"The reader's weakest assumption is the load-bearing one, and the manuscript's internal evidence supports it. §3.2 describes collecting over 10,000 write-ups from CTFtime covering over 700 competitions, including picoCTF and CSAW by ecosystem; §5.1 explicitly ties CTFKnow into CTFAgent's RAG system; and Figure 8 shows a concrete test challenge, 'puffin' from CSAW, receiving the exact overflow payload from DB-Exploiting. That is not a hypothetical contamination risk but a demonstrated one. Because the paper's headline result is the attribution of large gains to RAG and Environmental Augmentation, this overlap directly undermines the central claim. I am not raising the point as a charge of misconduct; the prose in §5.1 is simply internally inconsistent with the later claim of testing on the NYU dataset. A provenance-based holdout experiment would settle the matter cleanly. The CTFKnow measurement portion may still be valuable, and the manual verification of a random sample of 323 questions is genuine support for that benchmark, but the agent evaluation as presented cannot support the 85% and 120% claims.","tokens_in":29110,"tokens_out":4981,"duration_ms":61205,"concrete_test":"Build a provenance table mapping each of the reported ~2,078 RAG knowledge entries back to its source write-up from §3.2, then intersect that source set with the 300 test challenges in Intercode-CTF and NYU CTF by challenge name/identifier (and, where ambiguous, by vulnerable-code embedding similarity). Rerun Tables 2 and 3 with (a) every knowledge entry whose source write-up matches a test challenge removed from DB-Understanding and DB-Exploiting, and (b) the same maximum round budget for the baseline and CTFAgent. If the 39→73 and 8→18 gains persist under this holdout, contamination is not the explanation; if they shrink substantially, the reported improvement reflects retrieval rather than the modules' capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CTFAgent's two modules cause the 85% and 120% gains, but the evaluation does not rule out direct retrieval of test-challenge solutions. Per §3.2, the RAG database is built from write-ups of 'over 700 large-scale international CTF competitions', and §5.1 states that CTFKnow is 'already maintained in CTFAgent's RAG system'. The test sets are Intercode-CTF (picoCTF challenges) and NYU CTF (CSAW challenges), both ecosystems heavily represented in CTFtime write-ups. The paper's own Figure 8 shows DB-Exploiting returning, for the CSAW challenge 'puffin', the exact payload \"A\"x16 . \"B\"x4 . \"C\"x108 — that is, the answer to a test challenge. Unless 'puffin' is the only such case, the gap between CTFAgent and CTFAgent-w/o-RAG in Tables 2 and 3 measures how often the database contains the solution, not the value of knowledge augmentation. The ablation design cannot expose this because removing RAG removes the contaminated retrieval channel. The unmatched round budget visible in Figure 9 (baseline solves nothing beyond round 4 while CTFAgent is allowed many more rounds) is a second confound, but the decisive flaw is provenance overlap between the RAG corpus and the evaluation sets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper makes two contributions: (1) CTFKnow, a benchmark of 1,996 technical knowledge points and 3,992 single-choice/open-ended questions built from 1,084 CTF write-ups, used to measure LLMs' CTF knowledge and its application; and (2) CTFAgent, an LLM agent with two-stage RAG and interactive Environmental Augmentation, claimed to improve solved challenges by 85% on Intercode-CTF (39/100 to 73/100) and by 120% on NYU CTF (8/200 to 18/200), plus a top-23.6% ranking at picoCTF2024. The measurement study finds that LLMs perform well on single-choice CTF knowledge questions but much worse on open-ended application questions. The central empirical claim about CTFAgent is not supportable as presented because the RAG corpus is constructed from write-ups of the same competitions used for testing, with at least one documented case of retrieving the exact exploit payload for a test challenge; additional confounds include unmatched interaction-round budgets and an uncontrolled NYU baseline.","tokens_in":29358,"tokens_out":5687,"duration_ms":63819,"significance":"If the CTFAgent results were valid, the paper would make a substantial contribution: it would demonstrate that targeted retrieval and environment improvements can turn LLMs into credible automatic CTF solvers, and CTFKnow would be a useful benchmark for isolating technical knowledge from reasoning in security tasks. The CTFKnow construction pipeline is a genuine strength: it includes LLM-based knowledge extraction and filtering, a second-model cross-check, and manual verification of 323 sampled items. However, the framework evaluation is the load-bearing part of the paper, and its central numbers are compromised by test-set contamination and uncontrolled comparisons. The significance of the work therefore cannot be assessed from the reported experiments; the paper would need a substantially decontaminated and controlled evaluation to support its claims.","major_comments":[{"comment":"The RAG database directly overlaps with the test sets. The corpus is built from CTFtime write-ups of 'over 700 large-scale international CTF competitions' (§3.2), and the two evaluation datasets are picoCTF (Intercode-CTF) and CSAW (NYU CTF) challenges, both of which are exactly the type of competitions documented on CTFtime. §5.1 states that CTFKnow is already maintained in CTFAgent's RAG system, so the retrieval database also incorporates the 1,084 write-ups used to build CTFKnow. Figure 8, presented as the workflow example for the CSAW test challenge 'puffin', shows DB-Exploiting returning the exact payload 'A'x16 . 'B'x4 . 'C'x108 as a hint, i.e., the solution to that test challenge. The paper reports no decontamination step, such as excluding write-ups for the specific test challenges from the retrieval corpus. Consequently, the reported gains from 39 to 73 on Intercode-CTF and from 8 to 18 on NYU CTF can be explained by retrieval of solution write-ups rather than by the proposed modules. This issue is load-bearing for the paper's central claim and must be resolved before the results can be interpreted.","section":"§3.2, §4.2, §5.1, Figure 8"},{"comment":"The ablation design cannot distinguish knowledge augmentation from answer retrieval. Removing RAG entirely (CTFAgent-w/o-RAG) also removes the contaminated retrieval channel, so the performance gap between CTFAgent and CTFAgent-w/o-RAG in Tables 2 and 3 measures how often the database contains a solution to the test challenge, not the marginal value of the two-stage RAG mechanism. A proper ablation should keep the retrieval channel intact while using a decontaminated corpus (e.g., excluding write-ups originating from the same competitions or the same challenge names), or an unrelated control corpus, so that the retrieval process is exercised without leaking test solutions.","section":"§5.3"},{"comment":"The interaction-round budget is not matched between the compared systems. §5.1 sets CTFAgent's maximum interaction rounds to 30 for both datasets, while Figure 9 shows the Intercode baseline solving no challenges beyond the fourth round. If the baseline is capped at a much smaller number of rounds, the comparison conflates the effect of the two modules with the effect of a substantially larger action budget. The authors should either grant the baseline the same 30-round budget or report solve rates at a common round limit. This confound applies independently of the contamination issue.","section":"§5.1, Figure 9"},{"comment":"The NYU CTF baseline is not evaluated in a controlled setting. The note under Table 3 states, 'We obtained experimental data of NYU CTF directly from the paper by Shao et al. [71].' This means the model version, tooling, environment, and execution protocol are not matched between CTFAgent and the NYU baseline, so the claimed 120% improvement rests on an externally reported number rather than a head-to-head comparison. The same concern applies to the picoCTF2024 comparison in Table 5, where no evidence is provided that the NYU framework was run under identical conditions by the same pipeline.","section":"Table 3"}],"minor_comments":[{"comment":"The text reports 'over 700 large-scale international CTF competitions' and 'over 10,000 write-ups,' yet after filtering only 1,084 write-ups remain; please clarify whether the 700-competition and 10,000-write-up figures refer to the raw scrape and explain how the filtered set is distributed across competitions.","section":"§3.2"},{"comment":"The final panel shows 'casw{...}' as the flag, while the text states the flag format is 'csawctf{....}'; this typo should be corrected.","section":"Figure 8"},{"comment":"The word 'disscussed' appears in the Introduction and should be 'discussed'.","section":"§1"},{"comment":"The human evaluation row is based on a small sampled subset (90 single-choice and 30 open-ended questions per participant) with different open-ended scoring procedures than the LLM evaluation; please state the variance or confidence intervals so the human-LLM comparison is not over-interpreted.","section":"Table 1"},{"comment":"The o1-preview evaluation is only run on challenges unsolved by GPT-4-Turbo, and the statement that challenges solved by GPT-4-Turbo 'can be readily addressed by the o1 model' is not demonstrated by any reported run; please qualify this claim or provide the supporting data.","section":"§5.2, Table 4"}],"recommendation":"reject","confidential_remarks":"The decisive issue is test-set contamination of the RAG corpus. The paper's own Figure 8 shows retrieval of the exact exploit payload for a test challenge, and no decontamination procedure is described. The necessary fix is a substantial re-evaluation with a corpus that excludes write-ups of the test challenges, plus matched interaction budgets and a same-environment baseline for NYU CTF. These are not local presentation fixes, so I cannot recommend major revision within the current scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the CTFKnow benchmark is a genuine contribution, but the CTFAgent evaluation is circular. The RAG database is built from write-ups of the same competitions that make up the test sets, and Figure 8 shows the agent retrieving the exact payload for a test challenge. So the headline 85% and 120% gains cannot be attributed to the proposed modules.\n\nWhat's new: CTFKnow is a focused, knowledge-level benchmark with 3,992 single-choice and open-ended questions constructed from 1,084 CTF write-ups, with filtering, manual verification of a 323-item sample, and a small human baseline. The main finding—LLMs answer multiple-choice questions about CTF well but struggle when asked to produce the knowledge from scratch—is plausible and useful, and the execution-log analysis about missing tools and unfriendly environments is informative.\n\nSoft spots: the agent evaluation is the load-bearing claim and it does not hold. The database is assembled from CTFtime write-ups of over 700 competitions (§3.2), and the test sets are picoCTF and CSAW challenges. The paper itself notes CTFKnow is already in CTFAgent's RAG system (§5.1). Figure 8 shows DB-Exploiting returning the exact payload 'A'x16 . 'B'x4 . 'C'x108 for the CSAW challenge 'puffin'. Unless the authors can show that is the only overlap, the solve-rate gains measure retrieval of stored solutions, not capability. The ablation removes RAG entirely, which removes the contaminated channel, so it cannot isolate the value of knowledge augmentation. The NYU baseline is taken from the prior paper rather than re-run, and the round budgets are unmatched: the baseline solves nothing beyond round four while CTFAgent is allowed up to thirty rounds (Figure 9). The picoCTF2024 result is a real event, but without a deduplicated RAG corpus and matched conditions it is hard to interpret.\n\nThe stress-test note holds up on reading the paper. The problem is not a missing control; it is that the design makes contamination systematic.\n\nWho it's for: readers working on LLM security evaluation or RAG for offensive security. CTFKnow is worth a serious look. The agent claims need a proper holdout: remove any write-up for a challenge in the test set from the RAG corpus, re-run both baselines with matched round budgets, and report per-challenge provenance. I would send this to peer review—the benchmark and the failure analysis justify referee time—but the current empirical claims should not survive without that re-evaluation. My recommendation: major revision with the holdout experiment.","headline":"CTFKnow is a real benchmark, but the agent evaluation leaks test-solution payloads through its own RAG database, so the headline 85%/120% gains don't stand.","tokens_in":29981,"tokens_out":3558,"would_cite":true,"duration_ms":38791,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs know CTF theory but fail to apply it; a two-stage retrieval and an interactive shell close most of the gap.","keywords":["Capture-the-Flag","LLM benchmark","retrieval-augmented generation","offensive security","LLM agents","CTF automation","knowledge application"],"falsifier":"A concrete falsifier is a temporal split evaluation: build the RAG database only from write-ups published before 2023, run CTFAgent on the 2024 picoCTF and the newest Intercode-CTF challenges, and check whether the solve rate drop is large. If the drop is large, the reported gains measure retrieval from a corpus that overlaps the test set rather than CTF capability; if the drop is small, the two-stage RAG transfers general technique and the claim survives. A second check is to count how many CTFAgent solutions contain payload strings verbatim identical to RAG database entries rather than semantically equivalent variants.","tokens_in":28855,"feed_emoji":"🏴","tokens_out":3320,"duration_ms":29129,"temperature":0.7,"pith_summary":"This paper argues that what blocks large language models from solving Capture-the-Flag challenges is not missing technical knowledge but the application of that knowledge to concrete scenarios and the ability to act on feedback from the environment. To show this, the authors build CTFKnow, a 3,992-question benchmark of single-choice and open-ended questions distilled from over a thousand CTF write-ups, and measure five LLMs. The measurement finds that LLMs answer around 80 percent of single-choice questions but drop by roughly half on open-ended questions, and that they degrade further as challenge difficulty rises. The paper then proposes CTFAgent, which pairs a two-stage retrieval-augmented generation module with an interactive environment module, and reports that it raises solved challenges from 39 to 73 out of 100 on Intercode-CTF, from 8 to 18 out of 200 on the NYU CTF Dataset, and reaches the top 23.6 percent of nearly 7,000 teams in picoCTF2024.","feed_headline":"LLMs know CTF theory but fail to apply it","feed_subtitle":"A two-stage retrieval and an interactive shell raise CTF solve rates by 85 percent on standard benchmarks.","key_machinery":"The operating mechanism is the two-stage RAG pipeline. RAG-Understanding takes the code the agent reads from the challenge files, embeds it, and returns the two most similar vulnerability knowledge trunks; RAG-Exploiting takes each exploit idea the model proposes, embeds it, and returns the closest exploit-method knowledge, often including a ready-made payload. The interactive Environmental Augmentation module complements this by giving the model persistent netcat sessions with immediate, line-by-line feedback, prebuilt tool-use hints, and a modern decompiler, so that retrieved knowledge can actually be executed and corrected. CTFKnow, the measurement benchmark, is built by extracting 2,078 knowledge points from 1,084 write-ups with GPT-4, filtering them with a different LLM, and generating paired single-choice and open-ended questions, with 323 items manually verified at 99.38 percent accuracy.","core_discovery":"The central claim is that LLMs already possess most of the technical knowledge needed for CTF challenges, but they fail to match that knowledge to specific scenarios and to adapt their actions based on environment feedback. The paper supports this with CTFKnow, which separates knowledge recall (single-choice) from knowledge application (open-ended), and with CTFAgent, which supplies the missing pieces: a two-stage RAG that retrieves vulnerability-identification hints from code snippets in the understanding phase and exploit-knowledge hints from the model's own exploit ideas in the exploiting phase, plus an interactive environment with real-time netcat sessions, tool-use hints, and upgraded decompilation. The paper reports that this combination improves end-to-end solving by 85 percent on Intercode-CTF and by more than 120 percent on the NYU CTF Dataset, and that the gains concentrate in challenges that previously required long, multi-round interactions.","pith_inferences":["A direct test of whether the RAG database is leaking solutions would be to rebuild the knowledge base from write-ups that postdate the test challenges and rerun the evaluation; if gains persist, the two-stage retrieval is genuinely transferring technique rather than memorized answers.","The gap between single-choice and open-ended accuracy suggests that the same measurement design could be applied to other skill domains where LLMs look capable on multiple-choice tests but falter in practice, such as penetration testing or secure code review.","One could extend CTFAgent by feeding the final successful payload back into the RAG database as a new knowledge trunk, turning each solved challenge into a reusable exploit template; the paper does not propose this self-expanding loop.","The failure analysis's observation that models emit irrelevant content instead of retrying after a failed approach points to a testable fix: an output filter that detects non-task text and forces the agent to emit a concrete next action, which would likely push the solve rate higher on both datasets."],"forward_implications":["If the 85 percent gain is real, then external knowledge retrieval plus an interactive execution environment is the main lever for turning LLMs into competent CTF solvers, and other benchmarks should be redesigned to include both.","The less-than-50 percent performance on open-ended questions implies that knowledge recall benchmarks overstate readiness: any evaluation of LLMs for offensive security should test application, not just recognition.","The finding that the baseline solved no challenges beyond round four while CTFAgent solved some after twenty rounds suggests that preserving context and maintaining a session over many turns is itself a bottleneck that future agents need to address.","The picoCTF2024 result, with CTFAgent beating the NYU framework's rank by nearly half, implies that a RAG-plus-environment design can transfer from curated datasets to a live competition without special adaptation.","Because CTFAgent's RAG retrieves from the same write-up ecosystem that produced the test challenges, the reported gains should be read as the upper bound of what retrieval support can do, not as a measure of unaided reasoning."],"supporting_citations":[{"why":"Provides the Intercode-CTF dataset and baseline environment used to measure CTFAgent's 39-to-73 improvement.","marker":"[89]"},{"why":"Provides the NYU CTF Dataset, its baseline solve count, and the execution logs the paper analyses for environment-failure causes.","marker":"[71]"},{"why":"Supplies the other prior empirical evaluation of LLMs on offensive security challenges whose logs are analysed for missing tools and unfriendly environments.","marker":"[70]"},{"why":"The GPT-4 model used for CTF knowledge extraction, question generation, and as the main backbone of CTFAgent.","marker":"[35]"},{"why":"The Deepseek model used as an independent filter for hallucinated knowledge points and inaccurate questions.","marker":"[12]"},{"why":"The o1-preview model whose reasoning strength solves an additional 11 Intercode-CTF challenges under the CTFAgent ReAct adaptation.","marker":"[22]"},{"why":"The survey that defines the standard RAG indexing-retrieval-generation workflow that CTFAgent's two-stage design modifies.","marker":"[49]"}],"fun_headline_variants":["CTFAgent: RAG and interactive shell boost LLM solve rates 85%","LLMs know CTF theory, struggle to apply it; agent fixes that","Two-stage retrieval and env feedback up LLM CTF success by 85%","From CTF knowledge to action: agent improves LLMs by 85%","LLMs lack CTF application skills; CTFAgent bridges gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the CTF write-up corpus used to build the RAG database does not effectively contain the answers to the test challenges; for at least one test challenge the figure showing the pipeline returns the exact exploit payload suggests this assumption may be violated at a level that could inflate the reported solve rates.","fun_headline_variants_meta":{"raw":{"variants":["CTFAgent: RAG and interactive shell boost LLM solve rates 85%","LLMs know CTF theory, struggle to apply it; agent fixes that","Two-stage retrieval and env feedback up LLM CTF success by 85%","From CTF knowledge to action: agent improves LLMs by 85%","LLMs lack CTF application skills; CTFAgent bridges gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000312,"raw_usage":{"total_tokens":1824,"prompt_tokens":1046,"completion_tokens":778,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":676}},"tokens_in":662,"tokens_out":778,"duration_ms":7839,"temperature":1.0,"reasoning_tokens":676,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:30:42.032038+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete falsifier is a temporal split evaluation: build the RAG database only from write-ups published before 2023, run CTFAgent on the 2024 picoCTF and the newest Intercode-CTF challenges, and check whether the solve rate drop is large. If the drop is large, the reported gains measure retrieval from a corpus that overlaps the test set rather than CTF capability; if the drop is small, the two-stage RAG transfers general technique and the claim survives. A second check is to count how many CTFAgent solutions contain payload strings verbatim identical to RAG database entries rather than semantically equivalent variants.","supporting_citations":[{"cited_title":"Learning to Reason with LLMs | OpenAI","cited_arxiv_id":null,"evidence_quote":"The o1-preview model whose reasoning strength solves an additional 11 Intercode-CTF challenges under the CTFAgent ReAct adaptation."}],"review_version":1}