Pith. sign in

REVIEW 4 major objections 5 minor 98 references

Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs know CTF theory but fail to apply it; a two-stage retrieval and an interactive shell close most of the gap.

desk verdict CTFKnow is a real benchmark, but the agent evaluation leaks test-solution payloads through its own RAG database, so the headline 85%/120% gains don't stand. read the letter →

arxiv 2506.17644 v1 pith:U52ITS5N submitted 2025-06-21 cs.AI

Zimo Ji , Daoyuan Wu , Wenyuan Jiang , Pingchuan Ma , Zongjie Li , Shuai Wang This is my paper
classification cs.AI
keywords Capture-the-FlagLLMbenchmarkretrieval-augmentedgenerationoffensivesecurityagentsCTFautomationknowledgeapplication
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that what blocks large language models from solving Capture-the-Flag challenges is not missing technical knowledge but the application of that knowledge to concrete scenarios and the ability to act on feedback from the environment. To show this, the authors build CTFKnow, a 3,992-question benchmark of single-choice and open-ended questions distilled from over a thousand CTF write-ups, and measure five LLMs. The measurement finds that LLMs answer around 80 percent of single-choice questions but drop by roughly half on open-ended questions, and that they degrade further as challenge difficulty rises. The paper then proposes CTFAgent, which pairs a two-stage retrieval-augmented generation module with an interactive environment module, and reports that it raises solved challenges from 39 to 73 out of 100 on Intercode-CTF, from 8 to 18 out of 200 on the NYU CTF Dataset, and reaches the top 23.6 percent of nearly 7,000 teams in picoCTF2024.

What carries the argument

The operating mechanism is the two-stage RAG pipeline. RAG-Understanding takes the code the agent reads from the challenge files, embeds it, and returns the two most similar vulnerability knowledge trunks; RAG-Exploiting takes each exploit idea the model proposes, embeds it, and returns the closest exploit-method knowledge, often including a ready-made payload. The interactive Environmental Augmentation module complements this by giving the model persistent netcat sessions with immediate, line-by-line feedback, prebuilt tool-use hints, and a modern decompiler, so that retrieved knowledge can actually be executed and corrected. CTFKnow, the measurement benchmark, is built by extracting 2,078 knowledge points from 1,084 write-ups with GPT-4, filtering them with a different LLM, and generating paired single-choice and open-ended questions, with 323 items manually verified at 99.38 percent accuracy.

What would settle it

A concrete falsifier is a temporal split evaluation: build the RAG database only from write-ups published before 2023, run CTFAgent on the 2024 picoCTF and the newest Intercode-CTF challenges, and check whether the solve rate drop is large. If the drop is large, the reported gains measure retrieval from a corpus that overlaps the test set rather than CTF capability; if the drop is small, the two-stage RAG transfers general technique and the claim survives. A second check is to count how many CTFAgent solutions contain payload strings verbatim identical to RAG database entries rather than semantically equivalent variants.

Watch

Extended reading notes

Core claim

The central claim is that LLMs already possess most of the technical knowledge needed for CTF challenges, but they fail to match that knowledge to specific scenarios and to adapt their actions based on environment feedback. The paper supports this with CTFKnow, which separates knowledge recall (single-choice) from knowledge application (open-ended), and with CTFAgent, which supplies the missing pieces: a two-stage RAG that retrieves vulnerability-identification hints from code snippets in the understanding phase and exploit-knowledge hints from the model's own exploit ideas in the exploiting phase, plus an interactive environment with real-time netcat sessions, tool-use hints, and upgraded decompilation. The paper reports that this combination improves end-to-end solving by 85 percent on Intercode-CTF and by more than 120 percent on the NYU CTF Dataset, and that the gains concentrate in challenges that previously required long, multi-round interactions.

Load-bearing premise

The paper assumes that the CTF write-up corpus used to build the RAG database does not effectively contain the answers to the test challenges; for at least one test challenge the figure showing the pipeline returns the exact exploit payload suggests this assumption may be violated at a level that could inflate the reported solve rates.

Editorial extensions

If this is right

  • If the 85 percent gain is real, then external knowledge retrieval plus an interactive execution environment is the main lever for turning LLMs into competent CTF solvers, and other benchmarks should be redesigned to include both.
  • The less-than-50 percent performance on open-ended questions implies that knowledge recall benchmarks overstate readiness: any evaluation of LLMs for offensive security should test application, not just recognition.
  • The finding that the baseline solved no challenges beyond round four while CTFAgent solved some after twenty rounds suggests that preserving context and maintaining a session over many turns is itself a bottleneck that future agents need to address.
  • The picoCTF2024 result, with CTFAgent beating the NYU framework's rank by nearly half, implies that a RAG-plus-environment design can transfer from curated datasets to a live competition without special adaptation.
  • Because CTFAgent's RAG retrieves from the same write-up ecosystem that produced the test challenges, the reported gains should be read as the upper bound of what retrieval support can do, not as a measure of unaided reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of whether the RAG database is leaking solutions would be to rebuild the knowledge base from write-ups that postdate the test challenges and rerun the evaluation; if gains persist, the two-stage retrieval is genuinely transferring technique rather than memorized answers.
  • The gap between single-choice and open-ended accuracy suggests that the same measurement design could be applied to other skill domains where LLMs look capable on multiple-choice tests but falter in practice, such as penetration testing or secure code review.
  • One could extend CTFAgent by feeding the final successful payload back into the RAG database as a new knowledge trunk, turning each solved challenge into a reusable exploit template; the paper does not propose this self-expanding loop.
  • The failure analysis's observation that models emit irrelevant content instead of retrying after a failed approach points to a testable fix: an output filter that detects non-task text and forces the agent to emit a concrete next action, which would likely push the solve rate higher on both datasets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper makes two contributions: (1) CTFKnow, a benchmark of 1,996 technical knowledge points and 3,992 single-choice/open-ended questions built from 1,084 CTF write-ups, used to measure LLMs' CTF knowledge and its application; and (2) CTFAgent, an LLM agent with two-stage RAG and interactive Environmental Augmentation, claimed to improve solved challenges by 85% on Intercode-CTF (39/100 to 73/100) and by 120% on NYU CTF (8/200 to 18/200), plus a top-23.6% ranking at picoCTF2024. The measurement study finds that LLMs perform well on single-choice CTF knowledge questions but much worse on open-ended application questions. The central empirical claim about CTFAgent is not supportable as presented because the RAG corpus is constructed from write-ups of the same competitions used for testing, with at least one documented case of retrieving the exact exploit payload for a test challenge; additional confounds include unmatched interaction-round budgets and an uncontrolled NYU baseline.

Significance. If the CTFAgent results were valid, the paper would make a substantial contribution: it would demonstrate that targeted retrieval and environment improvements can turn LLMs into credible automatic CTF solvers, and CTFKnow would be a useful benchmark for isolating technical knowledge from reasoning in security tasks. The CTFKnow construction pipeline is a genuine strength: it includes LLM-based knowledge extraction and filtering, a second-model cross-check, and manual verification of 323 sampled items. However, the framework evaluation is the load-bearing part of the paper, and its central numbers are compromised by test-set contamination and uncontrolled comparisons. The significance of the work therefore cannot be assessed from the reported experiments; the paper would need a substantially decontaminated and controlled evaluation to support its claims.

major comments (4)
  1. [§3.2, §4.2, §5.1, Figure 8] The RAG database directly overlaps with the test sets. The corpus is built from CTFtime write-ups of 'over 700 large-scale international CTF competitions' (§3.2), and the two evaluation datasets are picoCTF (Intercode-CTF) and CSAW (NYU CTF) challenges, both of which are exactly the type of competitions documented on CTFtime. §5.1 states that CTFKnow is already maintained in CTFAgent's RAG system, so the retrieval database also incorporates the 1,084 write-ups used to build CTFKnow. Figure 8, presented as the workflow example for the CSAW test challenge 'puffin', shows DB-Exploiting returning the exact payload 'A'x16 . 'B'x4 . 'C'x108 as a hint, i.e., the solution to that test challenge. The paper reports no decontamination step, such as excluding write-ups for the specific test challenges from the retrieval corpus. Consequently, the reported gains from 39 to 73 on Intercode-CTF and from 8 to 18 on NYU CTF can be explained by retrieval of solution write-ups rather than by the proposed modules. This issue is load-bearing for the paper's central claim and must be resolved before the results can be interpreted.
  2. [§5.3] The ablation design cannot distinguish knowledge augmentation from answer retrieval. Removing RAG entirely (CTFAgent-w/o-RAG) also removes the contaminated retrieval channel, so the performance gap between CTFAgent and CTFAgent-w/o-RAG in Tables 2 and 3 measures how often the database contains a solution to the test challenge, not the marginal value of the two-stage RAG mechanism. A proper ablation should keep the retrieval channel intact while using a decontaminated corpus (e.g., excluding write-ups originating from the same competitions or the same challenge names), or an unrelated control corpus, so that the retrieval process is exercised without leaking test solutions.
  3. [§5.1, Figure 9] The interaction-round budget is not matched between the compared systems. §5.1 sets CTFAgent's maximum interaction rounds to 30 for both datasets, while Figure 9 shows the Intercode baseline solving no challenges beyond the fourth round. If the baseline is capped at a much smaller number of rounds, the comparison conflates the effect of the two modules with the effect of a substantially larger action budget. The authors should either grant the baseline the same 30-round budget or report solve rates at a common round limit. This confound applies independently of the contamination issue.
  4. [Table 3] The NYU CTF baseline is not evaluated in a controlled setting. The note under Table 3 states, 'We obtained experimental data of NYU CTF directly from the paper by Shao et al. [71].' This means the model version, tooling, environment, and execution protocol are not matched between CTFAgent and the NYU baseline, so the claimed 120% improvement rests on an externally reported number rather than a head-to-head comparison. The same concern applies to the picoCTF2024 comparison in Table 5, where no evidence is provided that the NYU framework was run under identical conditions by the same pipeline.
minor comments (5)
  1. [§3.2] The text reports 'over 700 large-scale international CTF competitions' and 'over 10,000 write-ups,' yet after filtering only 1,084 write-ups remain; please clarify whether the 700-competition and 10,000-write-up figures refer to the raw scrape and explain how the filtered set is distributed across competitions.
  2. [Figure 8] The final panel shows 'casw{...}' as the flag, while the text states the flag format is 'csawctf{....}'; this typo should be corrected.
  3. [§1] The word 'disscussed' appears in the Introduction and should be 'discussed'.
  4. [Table 1] The human evaluation row is based on a small sampled subset (90 single-choice and 30 open-ended questions per participant) with different open-ended scoring procedures than the LLM evaluation; please state the variance or confidence intervals so the human-LLM comparison is not over-interpreted.
  5. [§5.2, Table 4] The o1-preview evaluation is only run on challenges unsolved by GPT-4-Turbo, and the statement that challenges solved by GPT-4-Turbo 'can be readily addressed by the o1 model' is not demonstrated by any reported run; please qualify this claim or provide the supporting data.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported CTFAgent gains are partially circular: the RAG knowledge base is built from write-ups of the same competitions as the test sets, and Figure 8 shows a test challenge's exact exploit payload being retrieved from DB-Exploiting, so the 39-to-73 and 8-to-18 improvements are not shown to exceed stored-solution retrieval.

  1. other [§2.1, §3.2, §5.1, §4.4 and Figure 8]
    "In this paper, all collected write-ups have been sourced from CTFtime. ... We first select over 700 large-scale international CTF competitions from the past five years ... To clarify, we do not use CTFKnow here, as it is designed for measuring LLMs' knowledge acquisition ability (and it is already maintained in CTFAgent's RAG system), not for evaluating their end-to-end CTF solving ability. ... When RAG's DB-Exploit receives this idea, it searches its database for the most closely related piece of technical knowledge, which includes an example payload for buffer overflow. ..."

    The RAG corpus is CTFKnow, assembled from CTFtime write-ups of competitions that include the evaluation ecosystems: Intercode-CTF is a picoCTF collection and NYU CTF is a CSAW collection. The paper never reports a holdout removing write-ups whose challenges appear in the test sets. Figure 8 demonstrates the mechanism concretely: for the NYU CSAW test challenge 'puffin', DB-Exploiting returns the exact exploit payload ('A'x16 . 'B'x4 . 'C'x108). A solved challenge can therefore be a retrieval of a stored solution rather than a capability produced by the proposed two-stage RAG and Environmental Augmentation. The ablation variant 'CTFAgent-w/o-RAG' removes the contaminated retrieval channel, so the gaps in Tables 2 and 3 (39 to 73, 8 to 18) measure corpus overlap as much as framework value.

full rationale

The strongest evidence of circularity is provenance overlap between the RAG database and the evaluation sets, confirmed by the paper's own Figure 8: a test challenge ('puffin' from NYU CSAW) receives its exact exploit payload from DB-Exploiting. Because CTFKnow is described as 'already maintained in CTFAgent's RAG system' and is built from CTFtime write-ups of over 700 competitions, the evaluation does not establish that the proposed modules generalize beyond stored-solution lookup. This makes the headline performance gain partially circular, even though the RAG module also supplies generic knowledge that could aid unrelated challenges. The CTFKnow measurement study itself (single-choice and open-ended knowledge questions) is largely self-contained and does not rely on self-citation chains; findings about knowledge recall versus scenario-matching have independent content. The unmatched round budgets visible in Figure 9 (baseline solving nothing after round 4 while CTFAgent uses up to 30 rounds) are an additional confound affecting internal validity, but they are a comparison-protocol issue rather than a circularity of the derivation chain. Because only one concrete contamination case is exhibited and the paper does not quantify corpus overlap, the circularity is substantial but not total: score 6 rather than 8 or 10.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on five hand-chosen settings (top-k, rounds, extraction limit, filter threshold, sample size) and on several domain assumptions about benchmark validity, judge neutrality, and evaluation fairness. The most fragile assumption is that the RAG database does not contain write-ups of the test challenges, which Figure 8 directly contradicts. No new physical or mathematical entities are postulated; CTFAgent's modules are software components, not invented entities in the sense of this ledger.

free parameters (5)
  • RAG top-k = 2
    Two most related knowledge pieces returned as hints (Figure 7, §4.2); no sensitivity analysis reported.
  • Max interaction rounds = 30
    30 rounds for CTFAgent on both datasets (§5.1); baseline max rounds not reported, so gains may partly reflect longer budget.
  • Knowledge points per write-up = up to 2
    Prompt instructs GPT-4 to extract up to 2 knowledge points per write-up (§3.2 step 2), limiting benchmark coverage.
  • DeepSeek filtering threshold = score 5
    Only knowledge points scored 5 ('fully matches') are kept (§3.2 step 3 and Appendix C).
  • Manual verification sample size = 323
    Used to claim 99.38% reliability; a random sample, but only two inaccuracies found (§3.2 step 5).
assumptions (5)
  • domain assumption LLM-extracted and LLM-filtered knowledge points accurately reflect the original write-ups.
    Benchmark validity depends on GPT-4 extraction and DeepSeek filtering being accurate; only 323 of 1,996 instances were manually checked (§3.2).
  • domain assumption The RAG knowledge base does not contain write-up-derived solutions for the specific test challenges.
    This is required for a valid evaluation but contradicted by the puffin example (§4.4, Figure 8), where the retrieved payload matches the test challenge.
  • domain assumption The LLM judge (GPT-4-Turbo, cross-checked with Qwen) evaluates open-ended answers without systematic bias.
    Open-ended correctness rates depend on this; only GPT-4-Turbo's answers were cross-checked with a second judge, not all models (§3.5).
  • domain assumption Single-choice questions with deliberately obvious wrong answers measure knowledge rather than recognition of surface cues.
    The question generation prompt requires three options with 'very obvious mistakes' (Appendix C), which may inflate accuracy.
  • domain assumption Prior baseline numbers (NYU CTF) are comparable despite different model versions and environments.
    Table 3 uses results 'obtained directly from the paper by Shao et al.' without re-running in the same environment (§5.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges." pith.science (2026). https://pith.science/paper/U52ITS5N

@misc{pith2026250617644,
  author       = {Pith},
  title        = {Pith review of: Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U52ITS5N}},
  note         = {Machine review of arXiv:2506.17644}
}
read the original abstract

Capture-the-Flag (CTF) competitions are crucial for cybersecurity education and training. As large language models (LLMs) evolve, there is increasing interest in their ability to automate CTF challenge solving. For example, DARPA has organized the AIxCC competition since 2023 to advance AI-powered automated offense and defense. However, this demands a combination of multiple abilities, from knowledge to reasoning and further to actions. In this paper, we highlight the importance of technical knowledge in solving CTF problems and deliberately construct a focused benchmark, CTFKnow, with 3,992 questions to measure LLMs' performance in this core aspect. Our study offers a focused and innovative measurement of LLMs' capability in understanding CTF knowledge and applying it to solve CTF challenges. Our key findings reveal that while LLMs possess substantial technical knowledge, they falter in accurately applying this knowledge to specific scenarios and adapting their strategies based on feedback from the CTF environment. Based on insights derived from this measurement study, we propose CTFAgent, a novel LLM-driven framework for advancing CTF problem-solving. CTFAgent introduces two new modules: two-stage Retrieval Augmented Generation (RAG) and interactive Environmental Augmentation, which enhance LLMs' technical knowledge and vulnerability exploitation on CTF, respectively. Our experimental results show that, on two popular CTF datasets, CTFAgent both achieves over 80% performance improvement. Moreover, in the recent picoCTF2024 hosted by CMU, CTFAgent ranked in the top 23.6% of nearly 7,000 participating teams. This reflects the benefit of our measurement study and the potential of our framework in advancing LLMs' capabilities in CTF problem-solving.

Figures

Figures reproduced from arXiv: 2506.17644 by the authors.

Figure 1
Figure 1. The typical progress of solving a CTF challenge. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The standard workflow of RAG. In this paper, we enhance LLMs using a customized version of Retrieval-Augmented Generation (RAG) technology, which incor￾porates knowledge from external databases [49]. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The workflow of building our benchmark. 3.2 CTFKnow Design Motivation. We recognize the necessity for a benchmark that specifically measures LLM capabilities in CTF. Yet, existing bench￾marks, such as Intercode-CTF [89] and NYU CTF Dataset [71], focus on evaluating LLMs’ overall performance in solving CTF challenges. We see this as a limitation, as it is often challenging to disentan￾gle the technical knowledge from… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: A benchmark example, showing extracted technical knowledge along with the single-choice question. The open-ended [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: LLMs’ correctness rate by difficulty of original chal [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: The architecture of two-stage RAG system. Firstly, [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: An example illustrating the working progress of [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Cumulative distribution function for the number [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Average score distribution of all teams with 1,875 points, CTFAgent, and NYU Framework in picoCTF 2024. and principles of CTFAgent. We can draw the same conclusion from the experimental results of the NYU CTF Dataset. Simultaneously, we observe that the performance of…
Figure 11
Figure 11. Figure 11: Visualization of the 173 clusters obtained by apply￾ing Affinity Propagation to 2,013 CTF technical knowledge points; see the detailed cluster results on our project website. 80 60 40 20 0 20 40 60 t-SNE Dimension 1 40 20 0 20 40 t-SNE Dimension 2 173 Clusters of CTF …
Figure 12
Figure 12. Figure 12: C System Prompts In §3, we designed customized system prompts for the tasks of knowledge extraction, knowledge filtering, question generation (for [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 12
Figure 12. Figure 12: Distribution of challenges of various write-up collected in §3 by Year, Type, and Difficulty. Difficulty is determined [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

98 extracted references · 50 canonical work pages

  1. [71]

    Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2023. Examining zero-shot vulnerability repair with large language models. In2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2339–2356

    2023

  2. [1]

    Cybercrime To Cost The World $10.5 Trillion Annually By

    2020. Cybercrime To Cost The World $10.5 Trillion Annually By

    2020

  3. [2]

    AI Cyber Challenge Opens Registration, Adds $4 Million in Prizes, Shows Scoring Algorithm and Challenge Exemplar

    2023. AI Cyber Challenge Opens Registration, Adds $4 Million in Prizes, Shows Scoring Algorithm and Challenge Exemplar. https://www.darpa.mil/news/2023/ ai-cyber-challenge-opens

    2023

  4. [3]

    DEF CON®27 Hacking Conference Contests & Events

    2023. DEF CON®27 Hacking Conference Contests & Events. https://defcon. org/html/defcon-27/dc-27-ce.html

    2023

  5. [4]

    0CTF 2024

    2024. 0CTF 2024. https://ctf.0ops.sjtu.cn/. https://ctf.0ops.sjtu.cn/

    2024

  6. [5]

    All about CTF

    2024. All about CTF. https://ctftime.org/. https://ctftime.org/

    2024

  7. [6]

    Assistants API Overview

    2024. Assistants API Overview. https://platform.openai.com/docs/assistants/ overview?context=with-streaming. https://platform.openai.com/docs/assistants/ overview?context=with-streaming

    2024

  8. [7]

    2024. BUUCTF. https://buuoj.cn/. https://buuoj.cn/

    2024

Show all 98 references
  1. [8]

    Capture the Flag

    2024. Capture the Flag. https://www.csaw.io/ctf. https://www.csaw.io/ctf

    2024

  2. [9]

    Capture the Flag for Empowered Cybersecurity Training

    2024. Capture the Flag for Empowered Cybersecurity Training. https://ine.com/ blog/capture-the-flag-for-empowered-cybersecurity-training

    2024

  3. [10]

    CGC: Cyber Grand Challenge

    2024. CGC: Cyber Grand Challenge. https://www.darpa.mil/research/programs/ cyber-grand-challenge

    2024

  4. [11]

    Claude 3.5 Sonnet

    2024. Claude 3.5 Sonnet. hhttps://www.anthropic.com/news/claude-3-5-sonnet. https://www.anthropic.com/news/claude-3-5-sonnet

    2024

  5. [12]

    DeepSeek

    2024. DeepSeek. https://www.deepseek.com/

    2024

  6. [13]

    2024. DEFCON. https://defcon.org/. https://defcon.org/

    2024

  7. [14]

    Function calling

    2024. Function calling. https://platform.openai.com/docs/guides/function-calling. https://platform.openai.com/docs/guides/function-calling

    2024

  8. [15]

    2024. Ghidra. https://ghidra-sre.org/. https://ghidra-sre.org/

    2024

  9. [16]

    Google CTF

    2024. Google CTF. https://capturetheflag.withgoogle.com/. https://capturetheflag. withgoogle.com/

    2024

  10. [17]

    gpt-3-5-turbo

    2024. gpt-3-5-turbo. https://platform.openai.com/docs/models/gpt-3-5-turbo. https://platform.openai.com/docs/models/gpt-3-5-turbo

    2024

  11. [18]

    2024. gpt-4. https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4. https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4

    2024

  12. [19]

    2024. gpt-4o. https://platform.openai.com/docs/models/gpt-4o. https://platform. openai.com/docs/models/gpt-4o

    2024

  13. [20]

    2024. HITCON. https://hitcon.org/2024/CMT/. https://hitcon.org/2024/CMT/

    2024

  14. [21]

    2024. IDA. https://hex-rays.com/ida-free/

    2024

  15. [22]

    Learning to Reason with LLMs | OpenAI

    2024. Learning to Reason with LLMs | OpenAI. https://openai.com/index/ learning-to-reason-with-llms/

    2024

  16. [23]

    Meet Llama 3.1

    2024. Meet Llama 3.1. https://llama.meta.com/. https://llama.meta.com/

    2024

  17. [24]

    Mixtral of experts | Mistral AI | Frontier AI in your hands

    2024. Mixtral of experts | Mistral AI | Frontier AI in your hands. https://mistral. ai/news/mixtral-of-experts/. https://mistral.ai/news/mixtral-of-experts/

    2024

  18. [25]

    picoCTF - CMU Cybersecurity Competition

    2024. picoCTF - CMU Cybersecurity Competition. https://picoctf.org/. https: //picoctf.org/

    2024

  19. [26]

    picoCTF2024

    2024. picoCTF2024. https://play.picoctf.org/events/73/scoreboards. https://play. picoctf.org/events/73/scoreboards

    2024

  20. [27]

    Top 10 Cyber Hacking Competitions - Capture the Flag (CTF)

    2024. Top 10 Cyber Hacking Competitions - Capture the Flag (CTF). https://www. geeksforgeeks.org/top-cyber-hacking-competitions-capture-the-flag-ctf/

    2024

  21. [28]

    UIUCTF 2024

    2024. UIUCTF 2024. https://2024.uiuc.tf/. https://2024.uiuc.tf/

    2024

  22. [29]

    VicOne & Block Harbor Spearhead Biggest Automotive Cap- ture the Flag Competition for Cybersecurity Enthusiasts World- wide

    2024. VicOne & Block Harbor Spearhead Biggest Automotive Cap- ture the Flag Competition for Cybersecurity Enthusiasts World- wide. https://vicone.com/company/press-releases/vicone-and-block- 14 Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challen...

    2024

  23. [30]

    Burp Suite - Application Security Testing Software

    2025. Burp Suite - Application Security Testing Software. https://portswigger. net/burp

    2025

  24. [31]

    Microsoft Security Copilot Blog

    2025. Microsoft Security Copilot Blog. https://techcommunity.microsoft.com/ blog/securitycopilotblog/advancing-security-copilot-with-magic-automating- self-correction-in-nl2kql-and-b/4390932

    2025

  25. [32]

    Proactive Defense: The Role of Offensive Security in Cybersecurity

    2025. Proactive Defense: The Role of Offensive Security in Cybersecurity. https: //cloudsecurityalliance.org/artifacts/using-ai-for-offensive-security

    2025

  26. [33]

    Using AI for Offensive Security

    2025. Using AI for Offensive Security. https://cloudsecurityalliance.org/artifacts/ using-ai-for-offensive-security

    2025

  27. [34]

    What is Automated Vulnerability Remediation? https: //www.sentinelone.com/cybersecurity-101/cybersecurity/what-is-automated- vulnerability-remediation/

    2025. What is Automated Vulnerability Remediation? https: //www.sentinelone.com/cybersecurity-101/cybersecurity/what-is-automated- vulnerability-remediation/

    2025

  28. [35]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv:2303.08774(2023)

    2023 arXiv

  29. [36]

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. 2024. Graph of thoughts: Solving elaborate problems with large language models. InProc. AAAI

    2024

  30. [37]

    Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, et al. 2024. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models.arXiv preprint arXiv:2404...

    2024 arXiv

  31. [38]

    David Brumley. 2018. The cyber grand challenge and the future of cyber- autonomy.USENIX Login43, 2 (2018), 6–9

    2018

  32. [39]

    Tanner J Burns, Samuel C Rios, Thomas K Jordan, Qijun Gu, and Trevor Un- derwood. 2017. Analysis and exercises for engaging beginners in online CTF competitions for security education. InUSENIX Workshop on Advances in Security Education

    2017

  33. [40]

    Daihang Chen, Yonghui Liu, Mingyi Zhou, Yanjie Zhao, Haoyu Wang, Shuai Wang, Xiao Chen, Tegawendé F Bissyandé, Jacques Klein, and Li Li. 2024. LLM for Mobile: An Initial Roadmap.arXiv preprint arXiv:2407.06573(2024)

    2024 arXiv

  34. [41]

    Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Fan Yang, Zenan Zhou, Weipeng Chen, Haofen Wang, Jeff Z Pan, et al . 2025. Learning to Reason with Search for LLMs via Reinforcement Learning.arXiv preprint arXiv:2503.19470(2025)

    2025 arXiv

  35. [42]

    Kevin Chung and Julian Cohen. 2014. Learning obstacles in the capture the flag model. InUSENIX 3GSE

    2014

  36. [43]

    Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for rag systems. InProc. ACM SIGIR

    2024

  37. [44]

    Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. 2024. PentestGPT: Evaluating and harnessing large language models for automated penetration testing. InProc. USENIX Security

    2024

  38. [45]

    Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2023. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. InProceedings of the 32nd ACM SIGSOFT international symposium on software testing and ...

    2023

  39. [46]

    Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shu- jing Yang, and Lingming Zhang. 2024. Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries. In Proc. IEEE/ACM ICSE

    2024

  40. [47]

    Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang. 2024. Teams of LLM Agents can Exploit Zero-Day Vulnerabilities.arXiv preprint arXiv:2406.01637(2024)

    2024 arXiv

  41. [48]

    Brendan J Frey and Delbert Dueck. 2007. Clustering by passing messages between data points.science315, 5814 (2007), 972–976

    2007

  42. [49]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997(2023)

    2023 arXiv

  43. [50]

    Zeyu Gao, Hao Wang, Yuchen Zhou, Wenyu Zhu, and Chao Zhang. 2023. How far have we gone in vulnerability detection using large language models.arXiv preprint arXiv:2311.12420(2023)

    2023 arXiv

  44. [51]

    Andreas Happe and Jürgen Cito. 2023. Getting pwn’d by ai: Penetration testing with large language models. InProc. ACM FSE

    2023

  45. [52]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language under- standing.arXiv preprint arXiv:2009.03300(2020)

    2020 arXiv

  46. [53]

    Junjie Huang and Quanyan Zhu. 2024. PenHeal: A Two-Stage LLM Framework for Automated Pentesting and Optimal Remediation.arXiv:2407.17788(2024)

    2024 arXiv

  47. [54]

    Patrick Hulin, Andy Davis, Rahul Sridhar, Andrew Fasano, Cody Gallagher, Aaron Sedlacek, Tim Leek, and Brendan Dolan-Gavitt. 2017. {AutoCTF}: Creating diverse pwnables via automated bug injection. InWOOT

    2017

  48. [55]

    Pengfei Jing, Mengyun Tang, Xiaorong Shi, Xing Zheng, Sen Nie, Shi Wu, Yong Yang, and Xiapu Luo. 2024. SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity.arXiv:2412.20787(2024)

    2024 arXiv

  49. [56]

    Avishree Khare, Saikat Dutta, Ziyang Li, Alaia Solko-Breslin, Rajeev Alur, and Mayur Naik. 2023. Understanding the effectiveness of large language models in detecting security vulnerabilities.arXiv preprint arXiv:2311.16169(2023)

    2023 arXiv

  50. [57]

    Sung-Kyung Kim, Eun-Tae Jang, Hanjin Park, and Ki-Woong Park. 2023. Pwnable- Sherpa: An interactive coaching system with a case study of pwnable challenges. Computers & Security125 (2023), 103009

    2023

  51. [58]

    Stela Kucek and Maria Leitner. 2020. An empirical survey of functions and configurations of open-source capture the flag (ctf) environments.Journal of Network and Computer Applications151 (2020), 102470

    2020

  52. [59]

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. 2025. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383(2025)

    2025 arXiv

  53. [60]

    Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. 2024. Personal llm agents: Insights and survey about the capability, efficiency and security.arXiv preprint arXiv:2401.05459(2024)

    2024 arXiv

  54. [61]

    Zongjie Li, Wenying Qiu, Pingchuan Ma, Yichen Li, You Li, Sijia He, Baozheng Jiang, Shuai Wang, and Weixi Gu. 2024. On the Accuracy and Robustness of Large Language Models in Chinese Industrial Scenarios. InProc. ACM/IEEE IPSN

    2024

  55. [62]

    Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. 2025. From system 1 to system 2: A survey of reasoning large language models.arXiv preprint arXiv:2502.17419(2025)

    2025 arXiv

  56. [63]

    Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, et al. 2025. Advances and Chal- lenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems.arXiv pr...

    2025 arXiv

  57. [64]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruc- tion tuning.Advances in neural information processing systems36 (2024)

    2024

  58. [65]

    Ye Liu, Yue Xue, Daoyuan Wu, Yuqiang Sun, Yi Li, Miaolei Shi, and Yang Liu

  59. [66]

    Pingchuan Ma, Rui Ding, Shuai Wang, Shi Han, and Dongmei Zhang. 2023. InsightPilot: An LLM-empowered automated data exploration system. InEMNLP: System Demonstrations

    2023

  60. [67]

    PropertyGPT: LLM-driven Formal Verification of Smart Contracts through Retrieval-Augmented Property Generation. InProc. ISOC NDSS

  61. [68]

    Ruijie Meng, Martin Mirchev, Marcel Böhme, and Abhik Roychoudhury. 2024. Large language model guided protocol fuzzing. InProc. ISOC NDSS

    2024

  62. [69]

    Wei Ma, Daoyuan Wu, Yuqiang Sun, Tianwen Wang, Shangqing Liu, Jian Zhang, Yue Xue, and Yang Liu. 2025. Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justifications. InProc. IEEE/ACM ICSE

    2025

  63. [70]

    Minghao Shao, Boyuan Chen, Sofija Jancheska, Brendan Dolan-Gavitt, Siddharth Garg, Ramesh Karri, and Muhammad Shafique. 2024. An empirical evaluation of llms for solving offensive security challenges.arXiv:2402.11814(2024)

    2024 arXiv

  64. [72]

    Teotino Gomes Soares, Azhari Azhari, Nur Rokhman, and E Wonarko. 2021. Edu- cation question answering systems: a survey. InProceedings of The International MultiConference of Engineers and Computer Scientists

    2021

  65. [73]

    Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krish- namurthy, et al. 2024. NYU CTF Dataset: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security.arXi...

    2024 arXiv

  66. [74]

    Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Wei Ma, Lyuye Zhang, Miaolei Shi, and Yang Liu. 2024. LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs’ Vulnerability Reasoning.arXiv:2401.16185(2024)

    2024 arXiv

  67. [75]

    Nicholas Springer and Wu-chang Feng. 2021. Thunder CTF: Learning Cloud Security on a Dime.arXiv preprint arXiv:2107.12566(2021)

    2021 arXiv

  68. [76]

    Wesley Tann, Yuancheng Liu, Jun Heng Sim, Choon Meng Seah, and Ee-Chien Chang. 2023. Using large language models for cybersecurity capture-the-flag challenges and certification questions.arXiv preprint arXiv:2308.10443(2023)

    2023 arXiv

  69. [77]

    Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Haijun Wang, Zhengzi Xu, Xiaofei Xie, and Yang Liu. 2024. Gptscan: Detecting logic vulnerabilities in smart contracts by combining gpt with program analysis. InProc. IEEE/ACM ICSE

    2024

  70. [78]

    Alba Thaqi, Arbena Musa, and Blerim Rexha. 2024. Leveraging AI for CTF Challenge Optimization. InProc. IEEE CIEES

    2024

  71. [79]

    Chandra Thapa, Seung Ick Jang, Muhammad Ejaz Ahmed, Seyit Camtepe, Josef Pieprzyk, and Surya Nepal. 2022. Transformer-based language models for soft- ware vulnerability detection. InProc. ACM ACSAC

    2022

  72. [80]

    Saad Ullah, Mingji Han, Saurabh Pujar, Hammond Pearce, Ayse Coskun, and Gianluca Stringhini. 2023. Can large language models identify and reason about security vulnerabilities? not yet.arXiv preprint arXiv:2312.12575(2023)

    2023 arXiv

  73. [81]

    Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain, and Merouane Debbah

  74. [82]

    Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research9, 11 (2008)

    2008

  75. [83]

    Jan Vykopal, Valdemar Švábensk `y, and Ee-Chien Chang. 2020. Benefits and pitfalls of using capture the flag games in university courses. InProceedings of the 51st ACM Technical symposium on computer science education. 752–758

    2020

  76. [84]

    Department of Health and Human Services

    U.S. Department of Health and Human Services. 2018. Federal Policy for the Protection of Human Subjects (’Common Rule’). https://www.ecfr.gov/current/ title-45/subtitle-A/subchapter-A/part-46#p-46.104(d)(2) Title 45 Code of Federal 15 CCS ’25, October 13–17, 2025, Taipei, Taiw...

    2018

  77. [85]

    SeongIl Wi, Jaeseung Choi, and Sang Kil Cha. 2018. Git-based{CTF}: A Simple and Effective Approach to Organizing{In-Course}{ Attack-and-Defense} Secu- rity Competition. In2018 USENIX Workshop on Advances in Security Education

    2018

  78. [86]

    Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2024. Fuzz4all: Universal fuzzing with large language models. InProceed- ings of the IEEE/ACM 46th International Conference on Software Engineering

    2024

  79. [87]

    Liwen Wang, Yuanyuan Yuan, Ao Sun, Zongjie Li, Pingchuan Ma, Daoyuan Wu, and Shuai Wang. 2024. Benchmarking Multi-Modal LLMs for Testing Visual Deep Learning Systems Through the Lens of Image Mutation.arXiv preprint arXiv:2404.13945(2024)

    2024 arXiv

  80. [88]

    John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. 2024. Intercode: Standardizing and benchmarking interactive coding with execution feedback.Advances in Neural Information Processing Systems36 (2024)

    2024

  81. [89]

    John Yang, Akshara Prabhakar, Shunyu Yao, Kexin Pei, and Karthik R Narasimhan

  82. [90]

    Dandan Xu, Kai Chen, Miaoqian Lin, Chaoyang Lin, and Xiaofeng Wang. 2023. Autopwn: Artifact-assisted heap exploit generation for ctf pwn competitions. IEEE Transactions on Information Forensics and Security(2023)

    2023

  83. [91]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629(2022)

    2022 arXiv

  84. [92]

    Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. 2023. Chain-of-note: Enhancing robustness in retrieval-augmented language models.arXiv preprint arXiv:2311.09210(2023)

    2023 arXiv

  85. [93]

    Lyuye Zhang, Kaixuan Li, Kairan Sun, Daoyuan Wu, Ye Liu, Haoye Tian, and Yang Liu. 2024. Acfix: Guiding llms with mined common rbac practices for context-aware repair of access control vulnerabilities in smart contracts.arXiv preprint arXiv:2403.06838(2024)

    2024 arXiv

  86. [94]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models.NeurIPS(2024)

    2024

  87. [98]

    The knowledge fully matches the write- up and accurately reflects its content

    Li Zhong, Zilong Wang, and Jingbo Shang. 2024. Ldb: A large language model debugger via verifying runtime execution step-by-step.arXiv preprint arXiv:2402.16906(2024). Appendix A Knowledge Topics To visually illustrate the topics covered by the final 2,013 knowledge points in ...

    2024 arXiv

  88. [2023]

    InMulti-Agent Security Workshop@ NeurIPS’23

    Language agents as hackers: Evaluating cybersecurity skills with capture the flag. InMulti-Agent Security Workshop@ NeurIPS’23

  89. [2024]

    Cybermetric: A benchmark dataset for evaluating large language models knowledge in cybersecurity.arXiv preprint arXiv:2402.07688(2024)

    2024 arXiv

  90. [2025]

    https://cybersecurityventures.com/cybercrime-will-cost-the-world-16-4- billion-a-day-in-2021/

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.