Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges

Andr\'as Luk\'acs; David Imolai; Lajos Muzsai

REVIEW 5 major objections 5 minor 35 references

Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that GRPO reinforcement learning on a procedurally generated cryptographic CTF dataset can raise a tool-augmented 8B model's Pass@8 on unseen crypto challenges from 0.35 to 0.88, with transfer to external benchmarks.

desk verdict A useful dataset and a plausible RL result, but the headline Pass@8 comparison mixes task subsets and models, and needs a corrected baseline before it can be taken at face value. read the letter →

arxiv 2506.02048 v2 pith:OPJ54KWQ submitted 2025-06-01 cs.CR cs.AI

Lajos Muzsai , David Imolai , András Lukács This is my paper

classification cs.CRcs.AI

keywords ReinforcementlearningLLMagentsCryptographyCaptureTheFlagGRPOTool-augmentedreasoningCybersecurityProceduraldataset

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that GRPO-style reinforcement learning on a procedurally generated cryptographic CTF dataset can turn a tool-augmented 8B language model into a much stronger solver of unseen crypto challenges. Pass@8, the fraction of tasks solved in at least one of eight attempts, rises from 0.35 to 0.88 on previously unseen Random-Crypto tasks, with the detailed easy-subset table showing the same final level from a 0.10 baseline. The training also transfers to external benchmarks: picoCTF Pass@8 rises from 0.10 to 0.18, and the curriculum-trained agent reaches 0.19 on AICrypto MCQ, a multiple-choice benchmark never seen in training. If true, this matters because it offers a scalable, verifiable training recipe for security agents that does not depend on huge proprietary models or manually labeled examples.

What carries the argument

The two coupled mechanisms are Random-Crypto and GRPO with a shaped reward. Random-Crypto generates over five thousand training tasks from fifty randomized cryptographic scheme families, each with a unique flag and an LLM-written narrative, so the agent practices on infinitely varying verifiable challenges. GRPO, a PPO variant that ranks candidate outputs within a group, compares eight trajectories per prompt and updates the policy by relative reward, while the reward function gives 1.0 for a correct flag and smaller rewards for boxed-answer formatting, valid JSON tool calls, and error-free Python execution, with explicit penalties for hallucinated flags. The agent interacts with a persistent Python REPL server, allowing up to four tool calls per challenge, and training is made compute-efficient with QLoRA on a single A100 GPU.

What would settle it

Train the identical GRPO setup with the accuracy reward set to zero, or replaced with a reward for any boxed flag, while keeping all tool-format and execution rewards unchanged, and evaluate Pass@8 on the same easy test split. If it remains near 0.88, the gain is format compliance, not cryptanalysis; if it collapses to near base-model levels, the accuracy signal is doing the work. A complementary check is to evaluate the base model with few-shot demonstrations of the exact tool-call and boxed-answer format; if its Pass@8 jumps from 0.10 to near 0.88 without RL, format alone explains the result.

Watch

Extended reading notes

Core claim

The core claim is that cryptographic CTF challenges form an ideal reinforcement learning environment—precise flag validation, structured multi-step reasoning, and deterministic tool execution—and that a procedurally generated supply of such challenges makes RL training practical. Fine-tuning Llama-3.1-8B with GRPO on easy Random-Crypto tasks, using a Python execution server and a composite reward for accuracy, answer format, valid tool calls, and clean execution, produces an agent that solves previously unseen easy crypto tasks at Pass@8 0.88–0.90, where the tool-augmented base model scored 0.10. The authors attribute the gains to improved procedural reasoning and tool use, shown by cleaner code execution and format adherence, and report that hint-free or curriculum training generalizes to picoCTF and to a multiple-choice cryptography benchmark the model never encountered during training.

Load-bearing premise

The headline result rests on the assumption that a held-out challenge sharing the same prompt format, tool interface, and procedural generation style as the training tasks is solved because of genuine cryptographic reasoning, rather than because the model learned to produce the expected tool-call and answer format.

Editorial extensions

If this is right

A small 8B open model trained this way can approach the Pass@8 of far larger proprietary models on unseen easy crypto challenges, suggesting a compute-efficient route to security-agent competence.
RL on procedural crypto tasks generalizes beyond the training distribution, improving performance on picoCTF's heterogeneous challenges and on a multiple-choice cryptography format the model never saw during training.
The early training gains consist mostly of learning to follow the tool protocol—code that runs without errors and flags that are not hallucinated—so format compliance is a learnable and necessary component of agent performance.
Curriculum training with hints first and no hints later gives the best transfer to new formats, while pure no-hint training gives higher Maj@8 consistency on the training distribution.
The observed memory-exhaustion failures when the agent generates heavy enumeration code imply that RL tool-use environments need resource sandboxing alongside accuracy rewards.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The 0.35-to-0.88 Pass@8 headline may substantially reflect the base model's unfamiliarity with the tool-call prompt format; a controlled ablation that removes the accuracy reward would reveal how much of the gain is format compliance versus cryptanalytic skill.
If format compliance is the main driver, the same GRPO recipe could transfer to any domain with a verifiable final answer and a tool interface, making Random-Crypto a testbed for tool-use RL rather than for cryptography specifically.
The easy-only training choice hints that reward density, not raw task difficulty, governs RL success; procedurally generating tasks with calibrated difficulty could extend this approach to harder crypto and non-crypto CTF categories.
The authors' own sandboxing warning suggests a natural extension: make safe execution, through timeouts and memory limits, part of the reward so agents learn to avoid resource-exhausting code paths.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The new thing here is Random-Crypto: a procedural generator for crypto CTF tasks, 50 families, 5,000 training instances, plus code and data on GitHub. That is a real contribution. Using GRPO plus QLoRA to fine-tune an 8B model on this kind of verifiable tool-use environment is a sensible and practical direction, and the authors report honest external transfer to picoCTF and AICrypto MCQ. The paper deserves credit for releasing the benchmark and for documenting the execution-safety issues they hit.

The soft spots are real, and they mostly sit on the headline claim. The abstract says Pass@8 goes from 0.35 to 0.88. In the body, the Llama-3.1-8B baseline with tools and no hint on the full Random-Crypto test set is 0.10 (Table 2), while the 0.88 post-training number is on the easy subset (Table 3). The 0.35 in the abstract is close to GPT-4.1's score, not to the Llama baseline. So the before/after comparison is not matched: it mixes task difficulty and possibly models. That is a load-bearing inconsistency, not a typo.

There are smaller issues. Table 4's text says picoCTF Pass@8 goes from 0.10 to 0.18, but the improvement column implies a baseline of 0.07. No error bars appear anywhere, and the Maj@8 values (0.14 to 0.24) suggest high variance across the eight samples. The reward function gives 0.5 for successful execution and 0.2 for a valid tool call, so a model can collect most of the reward without solving anything; the paper's own Figure 2 shows format and execution rewards jumping early while solve rate climbs slowly. Part of the gain is likely prompt-format compliance. That does not kill the result, but it means the current tables overstate how much is cryptanalytic.

I do not think this is circular. The Random-Crypto test split is a held-out set from the generator, and the gains on picoCTF and AICrypto MCQ are external. The direction is sound. The paper is just not ready as-is: the baseline comparison needs to be corrected, results need variance reporting, and the easy-subset vs. full-set distinction needs to be explicit in the abstract.

Who is this for? Anyone working on LLM agents for security or procedural RL environments. I would want to see a corrected version before citing the performance numbers, but the dataset itself is worth citing.

Recommendation: send to peer review. It deserves a serious referee, and the revisions are manageable.

Referee Report

5 major / 5 minor

Summary. The paper introduces Random-Crypto, a procedurally generated cryptographic CTF dataset with 50 subtypes and 5,000+ training tasks, and uses GRPO to fine-tune a Python-tool-augmented Llama-3.1-8B model. The authors report that the trained agent improves Pass@8 on previously unseen crypto challenges from 0.35 to 0.88, and that the gains transfer to picoCTF and AICrypto MCQ. They attribute the improvement to enhanced tool usage and procedural reasoning, and release the benchmark and training code.

Significance. If the central quantitative claim were properly supported, this would be a valuable contribution: a procedurally generated, verifiable environment for RL in a security domain, with a modest-compute recipe (QLoRA on one A100, 250 steps) and credible transfer experiments. The release of dataset and code is a concrete strength. The RL-to-tool-use training recipe and the discussion of sandboxing risks are also useful. However, the headline result is currently not backed by a matched baseline, and the evidence is confined to an easy test subset with no statistical confidence information, so the significance of the claimed improvement is not yet established.

major comments (5)

[Abstract and Section 4.2 / Table 3] The abstract states that Pass@8 improves 'from 0.35 to 0.88' on previously unseen challenges, but this comparison does not appear anywhere in the body for Llama-3.1-8B. Table 2 reports Llama-3.1-8B with tool use and no hint at Pass@8 = 0.10 on the full 50-task test set, while Table 3 reports post-training Pass@8 = 0.88 on the easy subset, with an implied baseline of 0.10. The value 0.35 is close to GPT-4.1's Table 2 score (0.34) and is not a same-model baseline. The abstract must be reworded to state the exact baseline model, task subset, and metric, or the missing matched baseline must be reported.
[Section 4.2 / Table 3 vs Table 2] The reported RL gain is computed on the easy subset of Random-Crypto, whereas the baselines in Table 2 are on the full 50-task set. The paper never reports post-training Pass@8 on the full test set, so the claim that RL improves performance on 'previously unseen challenges' in general is unsupported. The authors should either report full-set before/after results or explicitly restrict the claim to the easy subset, and provide the easy-subset baseline separately.
[Table 4 and Section 4.3] Table 4 is internally inconsistent: the text says the hinted and curriculum models 'increase Pass@8 from 0.10 to 0.18', but the improvement column (+0.11) implies a baseline of 0.07, and the w/o-hints row implies the same 0.07 baseline. The reader cannot tell whether the true picoCTF baseline is 0.10 or 0.07. This inconsistency directly affects the generalization claim and must be corrected.
[Section 4.2 / Table 3] No error bars, confidence intervals, or significance tests are reported anywhere. The test set has only 50 tasks, and the easy subset is smaller; Pass@8 with eight samples can be highly variable. The low Maj@8 (0.24 for the w/o-hints model despite Pass@8 = 0.88) shows that most tasks are solved by only one of eight samples, so the headline Pass@8 may overstate consistent capability. The paper should report per-sample accuracy, confidence intervals, or a bootstrap analysis.
[Section 3.2 / Table 1] The reward design gives substantial weight to execution success (0.5) and format/tool-call compliance (0.2 + 0.1) relative to the accuracy reward (1.0). The paper itself reports that format adherence rose from ~40% to ~80% and execution success from ~39% to ~56%, while the accuracy reward rose only from 10% to 30% during training. This means a large part of the Pass@8 gain may reflect learned format and tool-call compliance rather than cryptanalytic skill. The authors should isolate the accuracy-only component (e.g., exact-match flag rate) and discuss the extent to which the gain is attributable to procedural compliance versus genuine problem-solving.

minor comments (5)

[Section 3.1] The sentence 'The distribution of subtype difficulties is balanced with at least 16 challenges from all three difficulty levels in the 50 subtypes' is ambiguous; it should clarify whether '16' refers to subtypes or challenge instances.
[Figure 1 and Figure 2 captions] The captions say 'One data point shows the average of all rewards given out in a training step to all benchmarks,' which is unclear; this should probably read 'to all training examples' or 'to all challenges in the batch'.
[Abstract and Section 4.3] There is a formatting issue in the abstract: ':picoCTF' and ':AICrypto MCQ' are missing spaces after the colon. Similar spacing issues appear in Section 4.3.
[Section 3.2 / Table 1] Table 1 lists positive rewards only, but the text describes penalties for hallucinated flags and for outputs not actually sent to the MCP server; these penalties should be included in the table or clearly defined in the reward description.
[Section 4.2] The claim that the curriculum and hinted models 'both gained a significant 0.80 improvement' uses the word 'significant' without a statistical test; please replace with a descriptive term or add a significance analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central RL improvement claim is checked against a held-out Random-Crypto split and two external benchmarks, and no claimed result reduces by construction to its inputs.

full rationale

The derivation chain is not circular. The paper's central claim is that GRPO training on easy Random-Crypto tasks improves Pass@8 on previously unseen challenges, and this is measured on a 50-task, manually validated test split that is disjoint from the 5,000 training tasks, and on two external benchmarks (picoCTF and AICrypto MCQ) whose data the authors did not generate and which are not part of the RL reward. The only self-citation, HackSynth [6], supplies the evaluation harness and prompt format rather than the benchmark answers or the trained model's fitted values, so it is independent tooling rather than an imported conclusion. The Random-Crypto generator is authored by the same team, which limits the strength of the within-distribution generalization claim, but the train/test split and manual validation make the comparison real; the reported Pass@8 increase is an empirical outcome, not a quantity forced by the reward function. A separate reporting concern is that the Abstract's 'from 0.35 to 0.88' is not a matched same-model before/after comparison (Table 2 reports 0.10 for Llama-3.1-8B with tool use and no hint, while Table 3 is restricted to the easy subset), and the reward's format/tool-call components could inflate Pass@8 if the base model merely lacked format familiarity. These are correctness or statistics issues, not circularity, because the Pass@8 metric still requires the true flag and the final results are not definitionally equal to the training rewards.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or formal entities are postulated. The Random-Crypto dataset is an artifact, not an invented theoretical entity; its external checkability rests on the GitHub release (no commit hash given).

free parameters (4)

reward_weights = accuracy=1.0, execution=0.5, tool_call=0.2, format=0.1
Hand-specified in Table 1 to guide training; no tuning procedure reported, so they are free choices that shape the learned behavior.
max_tool_interactions = 4
Interface allows up to four Python calls per trajectory (Section 3.2); limits expressed depth of reasoning.
evaluation_sampling = temperature=0.7, nucleus p=0.95, 8 generations
Chosen for Pass@8/Maj@8 evaluation (Section 4.3); variance across seeds is not reported.
training_steps = 251
Computed budget choice (Section 3.2); no stated convergence criterion.

assumptions (4)

domain assumption The 50 Random-Crypto subtypes implement the stated cryptographic schemes correctly (RSA, ECC, AES-GCM, etc.).
Challenge solving assumes standard mathematical properties of the ciphers; only one instance per subtype was manually validated (Section 3.1).
domain assumption Pass@8 with eight independent generations is a stable estimator of agent capability.
No confidence intervals or repeated-seed variance are reported (Sections 3.3 and 4), so the metric may be noisy on 50 or fewer test tasks.
domain assumption The MCP Python execution environment behaves equivalently during training and evaluation.
Tool use is the core interface; any distribution shift (e.g., different server state, memory limits) could change measured performance (Section 3.2 and Discussion).
domain assumption The base model has not already memorized the generated tasks.
If the generator's distributions overlap with pretraining data, the 'unseen' claim weakens; no contamination analysis is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges." pith.science (2026). https://pith.science/paper/OPJ54KWQ

@misc{pith2026250602048,
  author       = {Pith},
  title        = {Pith review of: Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OPJ54KWQ}},
  note         = {Machine review of arXiv:2506.02048}
}

read the original abstract

We present 'Random-Crypto', a procedurally generated cryptographic Capture The Flag (CTF) dataset designed to unlock the potential of Reinforcement Learning (RL) for LLM-based agents in security-sensitive domains. Cryptographic reasoning offers an ideal RL testbed: it combines precise validation, structured multi-step inference, and reliance on reliable computational tool use. Leveraging these properties, we fine-tune a Python tool-augmented Llama-3.1-8B via Group Relative Policy Optimization (GRPO) in a secure execution environment. The resulting agent achieves a significant improvement in Pass@8 on previously unseen challenges. Moreover, the improvements generalize to two external benchmarks: 'picoCTF', spanning both crypto and non-crypto tasks, and 'AICrypto MCQ', a multiple-choice benchmark of 135 cryptography questions. Ablation studies attribute the gains to enhanced tool usage and procedural reasoning. These findings position 'Random-Crypto' as a rich training ground for building intelligent, adaptable LLM agents capable of handling complex cybersecurity tasks.

Figures

Figures reproduced from arXiv: 2506.02048 by the authors.

**Figure 1.** Reward gained during training. The bright lines mark the average, while the shaded lines mark the actual data points. One data point shows the average of all rewards given out in a training step to all benchmarks. slightly behind the o3 model. The training setup that included no hints achieved a slightly lower Pass@8 score. On the contrary, it produced a significantly higher Maj@8 score, indicating that this trainin… view at source ↗

**Figure 2.** Reward types gained during training. The bright lines mark the average, while the shaded lines mark the actual data points. One data point shows the average of all rewards given out in a training step to all benchmarks. We can observe the biggest threefold improvement in the accuracy of the model, indicating successful challenge resolution. surface. This choice is deliberate: success on picoCTF requires the agent to… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 22 canonical work pages

[1]

L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, et al., A survey on large language model based autonomous agents, Frontiers of Computer Science 18 (2024) 186345

2024
[2]

R. Fang, R. Bindu, A. Gupta, Q. Zhan, D. Kang, LLM agents can autonomously hack websites, arXiv preprint arXiv:2402.06664 (2024)

arXiv 2024
[3]

R. Fang, R. Bindu, A. Gupta, D. Kang, LLM agents can autonomously exploit one-day vulnerabilities, arXiv preprint arXiv:2404.08144 13 (2024) 14

arXiv 2024
[4]

Y. Zhu, A. Kellermann, A. Gupta, P. Li, R. Fang, R. Bindu, D. Kang, Teams of LLM agents can exploit zero-day vulnerabilities, arXiv preprint arXiv:2406.01637 (2024)

arXiv 2024
[5]

J. Yang, A. Prabhakar, K. Narasimhan, S. Yao, Intercode: Standardizing and benchmarking interactive coding with execution feedback, Advances in Neural Information Processing Systems 36 (2024)

work page 2024
[6]

Muzsai, D

L. Muzsai, D. Imolai, A. Lukács, HackSynth: LLM agent and evaluation framework for autonomous penetration testing, arXiv preprint arXiv:2412.01778 (2024)

arXiv 2024
[7]

M. Shao, S. Jancheska, M. Udeshi, B. Dolan-Gavitt, H. Xi, K. Milner, B. Chen, M. Yin, S. Garg, P. Krishnamurthy, et al., NYU CTF Dataset: A scalable open-source benchmark dataset for evaluating LLMs in offensive security, arXiv preprint arXiv:2406.05590 (2024)

arXiv 2024
[8]

A. K. Zhang, N. Perry, R. Dulepet, E. Jones, J. W. Lin, J. Ji, C. Menders, G. Hussein, S. Liu, D. Jasper, et al., Cybench: A framework for evaluating cybersecurity capabilities and risk of language models, arXiv preprint arXiv:2408.08926 (2024)

arXiv 2024

Show all 35 references

[9]

Y. Cao, H. Zhao, Y. Cheng, T. Shu, Y. Chen, G. Liu, G. Liang, J. Zhao, J. Yan, Y. Li, Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods, IEEE Transactions on Neural Networks and Learning Systems (2024)

2024
[10]

Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al., Deepseekmath: Pushing the limits of mathematical reasoning in open language models, arXiv preprint arXiv:2402.03300 (2024)

2024 arXiv
[11]

D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al., Deepseek-r1: Incentivizing reasoning capability in LLMs via reinforcement learning, arXiv preprint arXiv:2501.12948 (2025)

2025 arXiv
[12]

1-8B-Instruct, 2024

Meta AI, Meta LLaMA 3.1 8B Models, https://huggingface.co/meta-llama/Meta-Llama-3. 1-8B-Instruct, 2024. Accessed: 2025-05-28

2024
[13]

W. Tann, Y. Liu, J. H. Sim, C. M. Seah, E.-C. Chang, Using large language models for cybersecurity capture-the-flag challenges and certification questions, arXiv preprint arXiv:2308.10443 (2023)

2023 arXiv
[14]

G. Deng, Y. Liu, V. Mayoral-Vilches, P. Liu, Y. Li, Y. Xu, T. Zhang, Y. Liu, M. Pinzger, S. Rass, PentestGPT: An LLM-empowered automatic penetration testing tool, arXiv preprint arXiv:2308.06782 (2023)

2023 arXiv
[15]

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, Y. Cao, React: Synergiz- ing reasoning and acting in language models, in: International Conference on Learning Representations (ICLR), 2023

2023
[16]

URL: https://platform.openai.com/ docs/guides/function-calling, accessed: 2025-05-28

OpenAI, Function calling | OpenAI Platform, 2024. URL: https://platform.openai.com/ docs/guides/function-calling, accessed: 2025-05-28

2024
[17]

URL: https://www.anthropic.com/news/model-context-protocol, accessed: 2025-05-28

Anthropic, Model Context Protocol: An open standard for connecting AI models to tools and data, 2024. URL: https://www.anthropic.com/news/model-context-protocol, accessed: 2025-05-28

2024
[18]

X. Hou, Y. Zhao, S. Wang, H. Wang, Model context protocol (MCP): Landscape, security threats, and future research directions, arXiv preprint arXiv:2503.23278 (2025)

2025 arXiv
[19]

Schulman, F

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms, arXiv preprint arXiv:1707.06347 (2017)

2017 arXiv
[20]

Ouyang, J

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., Training language models to follow instructions with human feedback, Advances in Neural Information Processing Systems 35 (2022) 27730–27744

2022
[21]

Dettmers, A

T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer, Qlora: Efficient finetuning of quantized LLMs, Advances in Neural Information Processing Systems 36 (2023) 10088– 10115

2023
[22]

Y. Wang, Y. Liu, L. Ji, H. Luo, W. Li, X. Zhou, C. Feng, P. Wang, Y. Cao, G. Zhang, et al., Aicrypto: A comprehensive benchmark for evaluating cryptography capabilities of large language models, arXiv preprint arXiv:2507.09580 (2025)

2025 arXiv
[23]

Accessed: 2025-05-28

Meta AI, Meta LLaMA-4-Scout-17B-16E Model, https://huggingface.co/meta-llama/ Llama-4-Scout-17B-16E-Instruct, 2025. Accessed: 2025-05-28

2025
[24]

1-70B-Instruct, 2024

Meta AI, Meta LLaMA 3.1 70B Model, https://huggingface.co/meta-llama/Meta-Llama-3. 1-70B-Instruct, 2024. Accessed: 2025-05-28

2024
[25]

Accessed: 2025-05-28

OpenAI, GPT-4.1 Model, https://openai.com/index/gpt-4-1/, 2025. Accessed: 2025-05-28

2025
[26]

Accessed: 2025-05-28

OpenAI, Openai o3 model, https://openai.com/index/introducing-o3-and-o4-mini/, 2025. Accessed: 2025-05-28

2025
[27]

H. D. Research, MCP-Python: A Python REPL server for the Model Context Protocol, https://github.com/hdresearch/mcp-python, 2025. Accessed: 2025-05-29

2025
[28]

Schick, J

T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, T. Scialom, Toolformer: Language models can teach themselves to use tools, Advances in Neural Information Processing Systems 36 (2023) 68539–68551

2023
[29]

P. Rai, S. Sood, V. K. Madisetti, A. Bahga, Guardian: A multi-tiered defense architecture for thwarting prompt injection attacks on LLMs, Journal of Software Engineering and Applications 17 (2024) 43–68

2024
[30]

Greshake, S

K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, M. Fritz, Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection, in: Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, 2023, pp. 79...

2023
[31]

</reasoning> tags

Do your chain-of-thought inside<reasoning> ... </reasoning> tags
[32]

name": "execute_python

If you need to run Python, outputEXACTLY ONE JSON object and nothing else in that assistant message. The object must have the shape { "name": "execute_python", "inputs": { "code": "print(’hello’)", "reset": false } } – no prose before or after, no Markdown formatting
[33]

Wrap the final flag in\boxed{flag{...}}

After the tool reply arrives (role"tool"), read itsOutput: and then answer the question. Wrap the final flag in\boxed{flag{...}}. Never guess the flag before you have seen it in a tool response
[34]

Only write the boxed flagafter the tool’s reply (role="tool") has appeared

Never include \boxed{flag{...}} in the SAME assistant message that contains a JSON tool call. Only write the boxed flagafter the tool’s reply (role="tool") has appeared
[35]

A vailable tools MCP_TOOL_LIST Question: QUESTION

Every assistant message must obey these rules. A vailable tools MCP_TOOL_LIST Question: QUESTION

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, et al., A survey on large language model based autonomous agents, Frontiers of Computer Science 18 (2024) 186345

2024

[2] [2]

R. Fang, R. Bindu, A. Gupta, Q. Zhan, D. Kang, LLM agents can autonomously hack websites, arXiv preprint arXiv:2402.06664 (2024)

arXiv 2024

[3] [3]

R. Fang, R. Bindu, A. Gupta, D. Kang, LLM agents can autonomously exploit one-day vulnerabilities, arXiv preprint arXiv:2404.08144 13 (2024) 14

arXiv 2024

[4] [4]

Y. Zhu, A. Kellermann, A. Gupta, P. Li, R. Fang, R. Bindu, D. Kang, Teams of LLM agents can exploit zero-day vulnerabilities, arXiv preprint arXiv:2406.01637 (2024)

arXiv 2024

[5] [5]

J. Yang, A. Prabhakar, K. Narasimhan, S. Yao, Intercode: Standardizing and benchmarking interactive coding with execution feedback, Advances in Neural Information Processing Systems 36 (2024)

work page 2024

[6] [6]

Muzsai, D

L. Muzsai, D. Imolai, A. Lukács, HackSynth: LLM agent and evaluation framework for autonomous penetration testing, arXiv preprint arXiv:2412.01778 (2024)

arXiv 2024

[7] [7]

M. Shao, S. Jancheska, M. Udeshi, B. Dolan-Gavitt, H. Xi, K. Milner, B. Chen, M. Yin, S. Garg, P. Krishnamurthy, et al., NYU CTF Dataset: A scalable open-source benchmark dataset for evaluating LLMs in offensive security, arXiv preprint arXiv:2406.05590 (2024)

arXiv 2024

[8] [8]

A. K. Zhang, N. Perry, R. Dulepet, E. Jones, J. W. Lin, J. Ji, C. Menders, G. Hussein, S. Liu, D. Jasper, et al., Cybench: A framework for evaluating cybersecurity capabilities and risk of language models, arXiv preprint arXiv:2408.08926 (2024)

arXiv 2024