Pith. sign in

REVIEW 3 major objections 4 minor 36 references

RedTeamLLM: an Agentic AI framework for offensive security

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A separated reasoning step lets an LLM agent complete more penetration-testing steps with fewer tool calls, the paper claims.

desk verdict The measured ablation (reasoning cuts tool calls in 4/5 CTFs) is believable and worth reporting, but the paper overclaims an unevaluated architecture and the headline percentages are not yet reproducible. read the letter →

arxiv 2505.06913 v1 pith:JKZLZLNB submitted 2025-05-11 cs.CR cs.AIcs.CY

Brian Challita , Pierre Parrend This is my paper · ORCID
classification cs.CRcs.AIcs.CY
keywords AgenticAIoffensivesecuritypenetrationtestingLLMagentsReActcontextwindowmemorymanagementplancorrection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes RedTeamLLM, an agentic-AI architecture for automated penetration testing, and evaluates a proof-of-concept built around a ReAct-style loop with three separate LLM sessions for summarizing, reasoning, and acting. It claims that this implementation outperforms the state-of-the-art PentestGPT in 60% of the five entry-level capture-the-flag use cases tested, and that adding a dedicated reasoning step reduces required tool calls in 80% of cases while improving offensive capability in 80% of cases. The broader architecture—recursive planning, plan correction, and memory management—is specified to address context-window limits and recovery from failed subtasks, but only the ReAct-based pipeline is evaluated. The intended contribution is a reference model the community can use to understand and defend against LLM-driven offensive operations.

What carries the argument

The load-bearing mechanism is the three-step pipeline: a Summarizer LLM session compresses long command outputs, a Reasoning LLM session plans the next action and analyzes the latest output, and an Act LLM session treats the reasoning output as an assistant message and issues exactly one tool call to a quasi-interactive root terminal. This separation keeps planning in a dedicated session so the executor does not drift into commentary or approval-seeking, and it lets the agent watch the terminal's full output without exhausting the context window. The surrounding architecture—Launcher, RedTeamAgent, Memory Manager, ADaPT Enhanced, Plan Corrector, and Planner—is specified as a tree-structured task decomposition with stored execution traces, but it is not what the reported numbers exercise.

What would settle it

Run RedTeamLLM and PentestGPT on the same five entry-level VMs under identical prompts, temperatures, and stopping criteria, and have a blinded judge score the transcripts against the write-up steps. If PentestGPT completes at least as many steps on Victim1, WestWild, and CTF4, or if the reasoned agent's advantage disappears when only the first run is counted, the paper's comparative claim fails.

Watch

Extended reading notes

Core claim

The central claim is that separating reasoning from acting into distinct LLM sessions materially changes agent behavior in offensive tasks. With reasoning, the agent formulates an explicit plan before each command, feeds the output back into the reasoner, and only then executes through a root-privileged terminal session; without it the agent repeats thoughts, changes strategy mid-run, and pauses to ask permission. Measured against the published write-up steps for five easy-category capture-the-flag VMs, the reasoned agent completes more steps in 4 of 5 cases, reaching 4 of the write-up's levels on Victim1 and WestWild, and cuts tool calls by 37–68% in 4 of 5 cases at the cost of a 291% increase on CTF4. The paper also claims that the full architecture—with ADaPT-style recursive decomposition, a memory manager that stores execution trees, and a plan corrector—addresses plan correction, memory, context-window limits, and generality, though these components are not part of the reported evaluation.

Load-bearing premise

The results assume that the human-written solution guides are a complete, unambiguous checklist of the correct steps, so the fraction of write-up steps an agent's best of five runs completes is a fair measure of offensive capability.

Editorial extensions

If this is right

  • If the 60% comparative claim holds, an LLM agent with a separated reasoning session is a viable baseline for entry-level penetration testing, completing more write-up steps than PentestGPT on Victim1, WestWild, and CTF4.
  • If the reasoning ablation is right, adding a dedicated reasoning step before each terminal action cuts tool calls by roughly a third to two-thirds on most tasks while improving completion, so the cost per successful action drops.
  • If the architecture-level claims are taken at face value, storing execution trees as embedded task descriptions should let the agent narrow to the right path on re-runs and avoid previously failed branches.
  • If the security model is honored, the resulting system is meant to run with user command validation, network isolation, append-only logs, and a kill switch, so the offensive capability is contained.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The comparison to PentestGPT reuses published benchmark numbers rather than running both agents under identical conditions, so a direct head-to-head with matched prompts and stopping rules would be a stronger test of the 60% claim.
  • Because the completion score is the best of five runs against human write-ups, the reported capability may overstate what a single red-team engagement would deliver; reporting first-run or median results would make the gain concrete.
  • The memory and plan-correction components are unevaluated, so the paper's contribution is best read as evidence for the reasoning step, not for the full agentic framework.
  • A natural extension is to vary the underlying model, since the paper's results come from GPT-4o and it is an open question whether the reasoning advantage persists with weaker open-weight models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes RedTeamLLM, an agentic AI framework for offensive security built around a three-step pipeline (summarize, reason, act). The full architecture includes a memory manager, an ADaPT-based recursive planner, and a plan corrector, but the evaluated proof-of-concept uses only the ReAct-style terminal-execution component. The evaluation runs the agent on five entry-level VULNHUB CTF virtual machines, compares its completion against PentestGPT numbers taken from the TAPT benchmark, and performs an ablation of the reasoning component. The central reported results are that reasoning reduces tool calls in 4 of 5 use cases and improves completion in 4 of 5 use cases, and that the implementation 'beats state-of-the-art competitors like PentestGPT in 60% of the use cases.'

Significance. If the quantitative claims hold, the paper provides a useful piece of evidence that a separate, dedicated reasoning session in a terminal-operating LLM agent improves both efficiency and task completion on entry-level CTF tasks. The authors also deserve credit for choosing a reproducible benchmark (VULNHUB VMs with public write-ups), for releasing the implementation on GitHub, and for running an explicit ablation of the reasoning component. The broader architectural contributions — memory management, recursive planning, and plan correction — are conceptually interesting but are explicitly not evaluated, so the paper's significance rests primarily on the ReAct-based evaluation. The small scale (five use cases, max-of-five aggregation) makes the reported percentages fragile, and the comparison to PentestGPT depends on unstated assumptions about the comparability of benchmark conditions.

major comments (3)
  1. [Section 6.3] The completion measurement is not sufficiently specified for the central claims. Section 6.3 states that completion is computed from write-up step lists and that the maximum number of steps achieved over 5 runs is considered, but it does not publish the step lists, the procedure for matching agent actions to steps, or the per-run distributions. With only five use cases, a single reclassified step can move the headline result from 4/5 to 3/5, so the 80% and 60% claims in Section 7 need a reproducible scoring rubric and ideally per-run results. Please provide the write-up step inventories, the matching rules, raw logs, and the distribution of completion scores over runs.
  2. [Section 6.2] The comparison with PentestGPT is not controlled. The paper borrows PentestGPT numbers from the TAPT benchmark (Isozaki et al., 2024) but does not state whether the prompts, model versions, tool sets, stopping criteria, or scoring conventions match those used for RedTeamLLM. Because Section 7's 'beats PentestGPT in 60% of use cases' claim depends entirely on this comparison, the authors should either run PentestGPT under matched conditions in their own environment or clearly justify why the TAPT numbers are directly comparable. Without this, the comparative claim is not yet supported.
  3. [Abstract and Section 5] The abstract claims the framework 'addresses four open challenges: plan correction, memory management, context window constraint, and generality vs. specialization,' but Section 5 states that ADaPT, memory management, and plan correction 'are less mature, and not evaluated here.' The paper should either temper the abstract to reflect that these are design features rather than evaluated contributions, or provide at least a preliminary evaluation of one of them. As written, the claim overstates the evidence in the paper.
minor comments (4)
  1. [Section 6.2] The text says Figure 6 reports totals 'after 10 tests on each VM,' while Section 6.3 describes 5 runs with reasoning and 5 without. Please clarify whether these are the same 10 runs and make the run counts consistent.
  2. [Section 6.2] The CTF4 comparison reports '3.5 vs. 2' and '3.5 vs. 1' steps; please explain how a half step can be awarded, since the completion metric in Section 6.3 is described in terms of write-up steps.
  3. [Section 5.1] The summarizer is described as stateless and 'sometimes omits important information'; this limitation is not mentioned in the evaluation discussion, even though the summarizer is part of the evaluated pipeline. A sentence in Sections 6 or 7 acknowledging the potential impact on the results would be appropriate.
  4. [Various] There are numerous typos and minor errors, including 'RefTeam LLM' (Section 4.4), 'avalaible' (Section 5), 'read teaming' (Section 2.1), 'peek' for 'peak' (Section 6.2), and inconsistent capitalization of PentestGPT. A careful proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is anchored to external artifacts (VULNHUB VMs, write-ups, TAPT benchmark) and no fitted parameter or self-citation chain is used to produce the reported results.

full rationale

The paper makes no derived prediction that reduces to its inputs. Section 6.3 defines the completion measure as: 'The degree of completion is computed for each use cases, using the write-up, which contains the listing of correct steps to complete the security challenge, as reference' — this anchors the score to external solution write-ups for five VULNHUB VMs, and the 'maximum number of steps achieved over the 5 runs' is an aggregation rule, not a parameter fitted to the target claim. The comparative result against PentestGPT in Section 6.2 uses step counts taken from the external TAPT benchmark ([Isozaki et al., 2024]), so the comparison is not self-referential. The reasoning ablation compares measured tool-call counts and completion levels with and without the reasoning session; both quantities are operationally defined and observed, not produced by fitting the model to those same observations. There is no self-citation that is load-bearing: ADaPT, ReAct, and PentestGPT are cited as independent prior work, and no uniqueness theorem from the authors' own previous publications is invoked. Section 5 concedes that memory management, ADaPT, and plan correction are 'less mature, and not evaluated here,' which narrows the abstract's claim that the framework 'addresses four open challenges' — but that is a scope/overclaiming issue, not a circular reduction. The main risks are reproducibility and comparability: the action-to-write-up matching procedure is not published, and the PentestGPT baseline conditions are not controlled. These are methodological concerns, not evidence that any result is equivalent to its input by construction. Therefore no circular step is established.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No fitted numerical parameters are introduced, but the central claims depend on hand-chosen evaluation decisions: best-of-5 aggregation, write-up-based scoring, easy-category VMs, cross-paper baselines, and a benevolent-user assumption that excludes prompt injection. Each is load-bearing for the Section 6 and 7 claims, and each is either undisclosed in procedure or explicitly outside the evaluated scope.

free parameters (1)
  • Run aggregation rule = maximum over 5 runs, n=5 per condition per VM
    Section 6.3 states 'The maximum number of steps achieved over the 5 runs is considered, i.e. the better execution.' This hand-chosen aggregation determines all completion claims in Sections 6 and 7, and no full distributions are reported.
assumptions (4)
  • domain assumption Human-authored VULNHUB write-ups fully enumerate the correct steps for each challenge, and matching agent actions to those steps measures offensive capability.
    Section 6.3 computes 'degree of completion' using the write-up as reference. The matching procedure is not described, so the yardstick for every completion claim is assumed valid.
  • domain assumption PentestGPT results from the TAPT benchmark (Isozaki et al. 2024) are directly comparable to RedTeamLLM's own runs on the same VMs.
    Section 6.2 adopts those numbers without evidence that model versions, stopping criteria, terminal conditions, or scoring rules were matched across papers.
  • domain assumption Five entry-level VULNHUB VMs are representative of the penetration-testing tasks the framework is claimed to handle.
    Section 6.1 restricts to the 'easy' category, while Section 3 claims genericity and automation beyond the evidence range.
  • domain assumption The LLM behaves as a benevolent user in default configuration, making prompt injection out of scope.
    Section 4.4 states this explicitly, excluding the main realistic threat for an agent with root terminal access; the security model is therefore untested against adversaries.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RedTeamLLM: an Agentic AI framework for offensive security." pith.science (2026). https://pith.science/paper/JKZLZLNB

@misc{pith2026250506913,
  author       = {Pith},
  title        = {Pith review of: RedTeamLLM: an Agentic AI framework for offensive security},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JKZLZLNB}},
  note         = {Machine review of arXiv:2505.06913}
}
read the original abstract

From automated intrusion testing to discovery of zero-day attacks before software launch, agentic AI calls for great promises in security engineering. This strong capability is bound with a similar threat: the security and research community must build up its models before the approach is leveraged by malicious actors for cybercrime. We therefore propose and evaluate RedTeamLLM, an integrated architecture with a comprehensive security model for automatization of pentest tasks. RedTeamLLM follows three key steps: summarizing, reasoning and act, which embed its operational capacity. This novel framework addresses four open challenges: plan correction, memory management, context window constraint, and generality vs. specialization. Evaluation is performed through the automated resolution of a range of entry-level, but not trivial, CTF challenges. The contribution of the reasoning capability of our agentic AI framework is specifically evaluated.

Figures

Figures reproduced from arXiv: 2505.06913 by the authors.

Figure 1
Figure 1. Process diagram of ReAct ADaPT [Prasad et al., 2023] takes a greedy approach to decomposition: it keeps decomposing the task until it reaches subtasks that can be executed, through recursive decompo￾sition which avoids a saturation of agent capability. The de￾composition stops either when a task can be executed directly, or when a max depth is reached. Unlike ReAct and P&E, ADaPT can’t be a prompting method as it is… view at source ↗
Figure 2
Figure 2. Software Architecture for Red Team LLM Model [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Database schema for Memory management Model [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Security layers wrapping the LLM agent The LLM itself is used in its default configuration, and with a benevolent user that have not intend to abuse it. Consequently, typical threats like prompt injection attacks [Labunets et al., ] or app store abuses [Hou et al., 202…
Figure 6
Figure 6. Figure 6: Number of API calls in Summarizer, Reason, Act steps for [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 8
Figure 8. Figure 8: CTF level completed by the RedTeamLLM framework [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 7
Figure 7. Figure 7: Number of tool calls without and with reasoning for the 5 [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 13 canonical work pages

  1. [1]

    Agentic ai: Au- tonomous intelligence for complex goals–a comprehen- sive survey

    [Acharya et al., 2025] Deepak Bhaskar Acharya, Karthigeyan Kuppan, and B Divya. Agentic ai: Au- tonomous intelligence for complex goals–a comprehen- sive survey. IEEE Access,

    work page 2025

  2. [4]

    {PentestGPT}: Evaluating and harnessing large language models for automated penetration testing

    [Deng et al., 2024] Gelei Deng, Yi Liu, V ´ıctor Mayoral- Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. {PentestGPT}: Evaluating and harnessing large language models for automated penetration testing. In33rd USENIX Security Symposium (USENIX Security

    work page 2024

  3. [5]

    Decoding the threat landscape: Chatgpt, fraudgpt, and wormgpt in social engi- neering attacks

    [Falade, 2023] Polra Victor Falade. Decoding the threat landscape: Chatgpt, fraudgpt, and wormgpt in social engi- neering attacks. arXiv preprint arXiv:2310.05595,

    arXiv 2023

  4. [7]

    Llm agents can autonomously hack websites

    [Fang et al., 2024b] Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang. Llm agents can autonomously hack websites. arXiv preprint arXiv:2402.06664,

    arXiv

  5. [8]

    Countering autonomous cyber threats

    [Heckel and Weller, 2024] Kade M Heckel and Adrian Weller. Countering autonomous cyber threats. arXiv preprint arXiv:2410.18312,

    arXiv 2024

  6. [9]

    On the (in) security of llm app stores

    [Hou et al., 2024] Xinyi Hou, Yanjie Zhao, and Haoyu Wang. On the (in) security of llm app stores. arXiv preprint arXiv:2407.08422,

    arXiv 2024

  7. [10]

    Ai agents and agen- tic systems: A multi-expert analysis

    [Hughes et al., 2025] Laurie Hughes, Yogesh K Dwivedi, Tegwen Malik, Mazen Shawosh, Mousa Ahmed Al- bashrawi, Il Jeon, Vincent Dutot, Mandanna Appan- deranda, Tom Crick, Rahul De’, et al. Ai agents and agen- tic systems: A multi-expert analysis. Journal of Computer Information Systems, pages 1–29,

    work page 2025

  8. [11]

    Toward robust secu- rity orchestration and automated response in security op- erations centers with a hyper-automation approach using agentic ai

    [Ismail et al., 2025] Ismail Ismail, Rahmat Kurnia, Zil- mas Arjuna Brata, Ghitha Afina Nelistiani, Shinwook Heo, Hyeongon Kim, and Howon Kim. Toward robust secu- rity orchestration and automated response in security op- erations centers with a hyper-automation approach using agentic ai

    work page 2025

Show all 36 references
  1. [12]

    Towards automated penetra- tion testing: Introducing llm benchmark, analysis, and im- provements

    [Isozaki et al., 2024] Isamu Isozaki, Manil Shrestha, Rick Console, and Edward Kim. Towards automated penetra- tion testing: Introducing llm benchmark, analysis, and im- provements. arXiv preprint arXiv:2410.17141,

    2024 arXiv

  2. [13]

    Darkbert: A language model for the dark side of the internet

    [Jin et al., 2023] Youngjin Jin, Eugene Jang, Jian Cui, Jin- Woo Chung, Yongjae Lee, and Seungwon Shin. Darkbert: A language model for the dark side of the internet. arXiv preprint arXiv:2305.08596,

    2023 arXiv

  3. [14]

    Security threats in agentic ai system

    [Khan et al., 2024] Raihan Khan, Sayak Sarkar, Sainik Ku- mar Mahata, and Edwin Jose. Security threats in agentic ai system. arXiv preprint arXiv:2410.14728,

    2024 arXiv

  4. [15]

    Fun- tuning: Characterizing the vulnerability of proprietary llms to optimization-based prompt injection attacks via the fine-tuning interface

    [Labunets et al., ] Andrey Labunets, Nishit V Pandya, Ashish Hooda, Xiaohan Fu, and Earlence Fernandes. Fun- tuning: Characterizing the vulnerability of proprietary llms to optimization-based prompt injection attacks via the fine-tuning interface. [Laney, 2024] Samuel P Laney....

    2024

  5. [16]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    [Lewis et al., 2020] Patrick Lewis, Ethan Perez, Aleksan- dra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨aschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Info...

    2020

  6. [18]

    Evocodebench: An evolving code generation benchmark with domain-specific evaluations

    [Li et al., 2024] Jia Li, Ge Li, Xuanming Zhang, Yunfei Zhao, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, and Yongbin Li. Evocodebench: An evolving code generation benchmark with domain-specific evaluations. Advances in Neural Information Processing Systems, 37:57619–57641,

    2024

  7. [19]

    Structured chain-of-thought prompting for code genera- tion

    [Li et al., 2025] Jia Li, Ge Li, Yongmin Li, and Zhi Jin. Structured chain-of-thought prompting for code genera- tion. ACM Transactions on Software Engineering and Methodology, 34(2):1–23,

    2025

  8. [20]

    Artificial intelligence (ai) cybersecurity dimensions: a comprehensive framework for understanding adversarial and offensive ai

    [Malatji and Tolah, 2024] Masike Malatji and Alaa Tolah. Artificial intelligence (ai) cybersecurity dimensions: a comprehensive framework for understanding adversarial and offensive ai. AI and Ethics, pages 1–28,

    2024

  9. [21]

    Hacksynth: Llm agent and evaluation framework for autonomous penetration testing

    [Muzsai et al., 2024] Lajos Muzsai, David Imolai, and Andr´as Luk ´acs. Hacksynth: Llm agent and evaluation framework for autonomous penetration testing. arXiv preprint arXiv:2412.01778,

    2024 arXiv

  10. [22]

    Agentic ai and the cyber arms race

    [Oesch et al., 2025] Sean Oesch, Jack Hutchins, Phillipe Austria, and Amul Chaulagain. Agentic ai and the cyber arms race. Computer, 58(5):82–85,

    2025

  11. [23]

    Adapt: As-needed decomposi- tion and planning with language models

    [Prasad et al., 2023] Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. Adapt: As-needed decomposi- tion and planning with language models. arXiv preprint arXiv:2311.05772,

    2023 arXiv

  12. [24]

    A systematic survey of prompt engineering in large language models: Techniques and applications

    [Sahoo et al., 2024] Pranab Sahoo, Ayush Kumar Singh, Sri- parna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927,

    2024 arXiv

  13. [25]

    Practices for governing agentic ai sys- tems

    [Shavit et al., 2023] Yonadav Shavit, Sandhini Agarwal, Miles Brundage, Steven Adler, Cullen O’Keefe, Rosie Campbell, Teddy Lee, Pamela Mishkin, Tyna Eloundou, Alan Hickey, et al. Practices for governing agentic ai sys- tems. Research Paper, OpenAI,

    2023

  14. [26]

    Pentestagent: Incorpo- rating llm agents to automated penetration testing

    [Shen et al., 2024] Xiangmin Shen, Lingzhi Wang, Zhenyuan Li, Yan Chen, Wencheng Zhao, Dawei Sun, Jiashui Wang, and Wei Ruan. Pentestagent: Incorpo- rating llm agents to automated penetration testing. arXiv preprint arXiv:2411.05185,

    2024 arXiv

  15. [27]

    Pearl: Prompting large language models to plan and execute actions over long documents

    [Sun et al., 2023] Simeng Sun, Yang Liu, Shuohang Wang, Chenguang Zhu, and Mohit Iyyer. Pearl: Prompting large language models to plan and execute actions over long documents. arXiv preprint arXiv:2305.14564,

    2023 arXiv

  16. [28]

    Trustllm: Trust- worthiness in large language models

    [Sun et al., 2024] Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wen- han Lyu, Yixuan Zhang, Xiner Li, et al. Trustllm: Trust- worthiness in large language models. arXiv preprint arXiv:2401.05561,

    2024 arXiv

  17. [29]

    Cyberseceval 3: Advancing the eval- uation of cybersecurity risks and capabilities in large lan- guage models

    [Wan et al., 2024] Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, et al. Cyberseceval 3: Advancing the eval- uation of cybersecurity risks and capabilities in large l...

    2024 arXiv

  18. [30]

    Self-consistency improves chain of thought reasoning in language models

    [Wang et al., 2022] Xuezhi Wang, Jason Wei, Dale Schu- urmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171,

    2022 arXiv

  19. [32]

    Tdag: A multi-agent framework based on dynamic task decomposition and agent genera- tion

    [Wang et al., 2025] Yaoxiang Wang, Zhiyong Wu, Junfeng Yao, and Jinsong Su. Tdag: A multi-agent framework based on dynamic task decomposition and agent genera- tion. Neural Networks, page 107200,

    2025

  20. [33]

    Chain-of-thought prompting elicits reasoning in large language models

    [Wei et al., 2022] Jason Wei, Xuezhi Wang, Dale Schuur- mans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837,

    2022

  21. [34]

    Autoattacker: A large language model guided system to implement automatic cyber-attacks

    [Xu et al., 2024] Jiacen Xu, Jack W Stokes, Geoff McDon- ald, Xuesong Bai, David Marshall, Siyue Wang, Adith Swaminathan, and Zhou Li. Autoattacker: A large language model guided system to implement automatic cyber-attacks. arXiv preprint arXiv:2403.01038,

    2024 arXiv

  22. [35]

    A survey on large language model (llm) security and privacy: The good, the bad, and the ugly

    [Yao et al., 2024] Yifan Yao, Jinhao Duan, Kaidi Xu, Yuan- fang Cai, Zhibo Sun, and Yue Zhang. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, page 100211,

    2024

  23. [36]

    Automatic chain of thought prompting in large language models

    [Zhang et al., 2022] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022

    2022 arXiv

  24. [2020]

    Chain of code: Reasoning with a language model-augmented code emulator

    [Li et al., 2023] Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter. Chain of code: Reasoning with a language model-augmented code emulator. arXiv preprint arXiv:2312.04474,

    2023 arXiv

  25. [2022]

    Tdag: A multi-agent framework based on dynamic task decomposition and agent genera- tion

    [Wang et al., 2024] Yaoxiang Wang, Zhiyong Wu, Junfeng Yao, and Jinsong Su. Tdag: A multi-agent framework based on dynamic task decomposition and agent genera- tion. arXiv preprint arXiv:2402.10178,

    2024 arXiv

  26. [2023]

    Llm agents can au- tonomously exploit one-day vulnerabilities.arXiv preprint arXiv:2404.08144,

    [Fang et al., 2024a] Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. Llm agents can au- tonomously exploit one-day vulnerabilities.arXiv preprint arXiv:2404.08144,

    arXiv

  27. [2024]

    Breaking down the defenses: A comparative survey of attacks on large language models

    [Chowdhury et al., 2024] Arijit Ghosh Chowdhury, Md Mofijul Islam, Vaibhav Kumar, Faysal Hossain Shezan, Vinija Jain, and Aman Chadha. Breaking down the defenses: A comparative survey of attacks on large language models. arXiv preprint arXiv:2403.04786,

    2024

  28. [2025]

    When do program-of-thought works for reasoning? In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 38, pages 17691–17699,

    [Bi et al., 2024] Zhen Bi, Ningyu Zhang, Yinuo Jiang, Shumin Deng, Guozhou Zheng, and Huajun Chen. When do program-of-thought works for reasoning? In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 38, pages 17691–17699,

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.