{"id":"a1892ee6-4bac-472c-bdd3-4198e4b0e6f9","arxiv_id":"2506.02048","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A tool-augmented 8B LLM fine-tuned with GRPO on a new procedurally generated crypto CTF dataset reaches 0.88 Pass@8 on unseen easy tasks, up from 0.10 in the body's tables.","lead":"The authors generate thousands of randomized cryptographic CTF puzzles and use reinforcement learning to fine-tune an 8-billion-parameter language model to solve unseen versions of them. The trained model also improves on two external security benchmarks, though the headline improvement numbers contain internal inconsistencies.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline gain is not backed by a matched pre/post baseline: the Abstract's 0.35 matches GPT-4.1 in Table 2, not Llama-3.1-8B, and Table 3's 0.88 is on the easy subset while Table 2's 0.10 is on the full 50-task set.","rationale":"The reader's weakest assumption flags the prompt-format confound, and that is real: the base model was not rewarded for the JSON tool-call format that the RL model was explicitly trained to emit, so Pass@8 gains conflate protocol compliance with cryptanalysis. However, the more decisive and easily checkable problem is that the paper never presents a matched baseline: the Abstract's 0.35 is not the base model's score in any table, and Table 3's post-training 0.88 is computed on the easy subset while Table 2's baseline 0.10 is for the full 50-task set. This internal inconsistency directly undermines the central numerical claim. The dataset and code release are real assets and the direction is sensible, so the correct disposition is conditional acceptance pending a corrected, matched baseline and variance reporting; this is exactly the reader's CONDITIONAL verdict, so I do not move it.","tokens_in":9890,"tokens_out":9285,"duration_ms":84786,"concrete_test":"Run a single matched re-evaluation: apply the base Llama-3.1-8B-Instruct and the GRPO w/o-hints checkpoint to the exact easy subset used for Table 3, with the same tool-augmented no-hint prompt (Appendix B, Prompt 2), and report Pass@8/Maj@8 with bootstrap 95% CIs over the 50 tasks. Trace the provenance of the Abstract's 0.35: if it is GPT-4.1's full-set Pass@8 (0.34 in Table 2), rewrite the headline as a cross-model comparison; if it is an easy-subset baseline, report it in the table. If the easy-subset base Pass@8 is near 0.35 instead of 0.10, the claimed improvement is smaller than stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (Abstract; Section 4.2) is that GRPO raises Pass@8 from 0.35 to 0.88 on previously unseen crypto challenges. The body does not support that before/after comparison. For Llama-3.1-8B with tool use and no hint, Table 2 reports Pass@8 = 0.10 on the full 50-task Random-Crypto set; Table 3 reports post-training Pass@8 = 0.88, but the text of Section 4.2 says Table 3 is restricted to the easy subset. The 0.35 in the Abstract is close to GPT-4.1's Table 2 Pass@8 (0.34 with tool use, no hint), suggesting the headline compares the trained Llama model against a different model rather than the same model before training, or uses an unreported easy-subset baseline. If 0.10 is the intended baseline, the comparison is confounded by task difficulty (full set vs easy subset). If 0.35 is the easy-subset baseline, it is never reported in the tables and the true improvement is +0.53, not the stated +0.78. A further confound is that the base model was never trained on the JSON tool-call/answer-format protocol (Section 3.2 reward design), so part of the measured gain may be format compliance; the low Maj@8 of 0.24 for the w/o-hints model shows the Pass@8 gain is not robust across the eight samples. The result may still be real, but the paper currently does not provide the controlled comparison needed to establish it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Random-Crypto, a procedurally generated cryptographic CTF dataset with 50 subtypes and 5,000+ training tasks, and uses GRPO to fine-tune a Python-tool-augmented Llama-3.1-8B model. The authors report that the trained agent improves Pass@8 on previously unseen crypto challenges from 0.35 to 0.88, and that the gains transfer to picoCTF and AICrypto MCQ. They attribute the improvement to enhanced tool usage and procedural reasoning, and release the benchmark and training code.","tokens_in":10253,"tokens_out":4010,"duration_ms":36071,"significance":"If the central quantitative claim were properly supported, this would be a valuable contribution: a procedurally generated, verifiable environment for RL in a security domain, with a modest-compute recipe (QLoRA on one A100, 250 steps) and credible transfer experiments. The release of dataset and code is a concrete strength. The RL-to-tool-use training recipe and the discussion of sandboxing risks are also useful. However, the headline result is currently not backed by a matched baseline, and the evidence is confined to an easy test subset with no statistical confidence information, so the significance of the claimed improvement is not yet established.","major_comments":[{"comment":"The abstract states that Pass@8 improves 'from 0.35 to 0.88' on previously unseen challenges, but this comparison does not appear anywhere in the body for Llama-3.1-8B. Table 2 reports Llama-3.1-8B with tool use and no hint at Pass@8 = 0.10 on the full 50-task test set, while Table 3 reports post-training Pass@8 = 0.88 on the easy subset, with an implied baseline of 0.10. The value 0.35 is close to GPT-4.1's Table 2 score (0.34) and is not a same-model baseline. The abstract must be reworded to state the exact baseline model, task subset, and metric, or the missing matched baseline must be reported.","section":"Abstract and Section 4.2 / Table 3"},{"comment":"The reported RL gain is computed on the easy subset of Random-Crypto, whereas the baselines in Table 2 are on the full 50-task set. The paper never reports post-training Pass@8 on the full test set, so the claim that RL improves performance on 'previously unseen challenges' in general is unsupported. The authors should either report full-set before/after results or explicitly restrict the claim to the easy subset, and provide the easy-subset baseline separately.","section":"Section 4.2 / Table 3 vs Table 2"},{"comment":"Table 4 is internally inconsistent: the text says the hinted and curriculum models 'increase Pass@8 from 0.10 to 0.18', but the improvement column (+0.11) implies a baseline of 0.07, and the w/o-hints row implies the same 0.07 baseline. The reader cannot tell whether the true picoCTF baseline is 0.10 or 0.07. This inconsistency directly affects the generalization claim and must be corrected.","section":"Table 4 and Section 4.3"},{"comment":"No error bars, confidence intervals, or significance tests are reported anywhere. The test set has only 50 tasks, and the easy subset is smaller; Pass@8 with eight samples can be highly variable. The low Maj@8 (0.24 for the w/o-hints model despite Pass@8 = 0.88) shows that most tasks are solved by only one of eight samples, so the headline Pass@8 may overstate consistent capability. The paper should report per-sample accuracy, confidence intervals, or a bootstrap analysis.","section":"Section 4.2 / Table 3"},{"comment":"The reward design gives substantial weight to execution success (0.5) and format/tool-call compliance (0.2 + 0.1) relative to the accuracy reward (1.0). The paper itself reports that format adherence rose from ~40% to ~80% and execution success from ~39% to ~56%, while the accuracy reward rose only from 10% to 30% during training. This means a large part of the Pass@8 gain may reflect learned format and tool-call compliance rather than cryptanalytic skill. The authors should isolate the accuracy-only component (e.g., exact-match flag rate) and discuss the extent to which the gain is attributable to procedural compliance versus genuine problem-solving.","section":"Section 3.2 / Table 1"}],"minor_comments":[{"comment":"The sentence 'The distribution of subtype difficulties is balanced with at least 16 challenges from all three difficulty levels in the 50 subtypes' is ambiguous; it should clarify whether '16' refers to subtypes or challenge instances.","section":"Section 3.1"},{"comment":"The captions say 'One data point shows the average of all rewards given out in a training step to all benchmarks,' which is unclear; this should probably read 'to all training examples' or 'to all challenges in the batch'.","section":"Figure 1 and Figure 2 captions"},{"comment":"There is a formatting issue in the abstract: ':picoCTF' and ':AICrypto MCQ' are missing spaces after the colon. Similar spacing issues appear in Section 4.3.","section":"Abstract and Section 4.3"},{"comment":"Table 1 lists positive rewards only, but the text describes penalties for hallucinated flags and for outputs not actually sent to the MCP server; these penalties should be included in the table or clearly defined in the reward description.","section":"Section 3.2 / Table 1"},{"comment":"The claim that the curriculum and hinted models 'both gained a significant 0.80 improvement' uses the word 'significant' without a statistical test; please replace with a descriptive term or add a significance analysis.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the dataset/code release is a strength, but the abstract and Section 4.2 do not currently provide a same-model, same-subset baseline for the headline Pass@8 gain. The authors should also address the internal inconsistency in Table 4 and the absence of uncertainty quantification. I would not reject, because the problems are fixable with re-analysis and rephrasing, but the paper is not publishable in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is Random-Crypto: a procedural generator for crypto CTF tasks, 50 families, 5,000 training instances, plus code and data on GitHub. That is a real contribution. Using GRPO plus QLoRA to fine-tune an 8B model on this kind of verifiable tool-use environment is a sensible and practical direction, and the authors report honest external transfer to picoCTF and AICrypto MCQ. The paper deserves credit for releasing the benchmark and for documenting the execution-safety issues they hit.\n\nThe soft spots are real, and they mostly sit on the headline claim. The abstract says Pass@8 goes from 0.35 to 0.88. In the body, the Llama-3.1-8B baseline with tools and no hint on the full Random-Crypto test set is 0.10 (Table 2), while the 0.88 post-training number is on the easy subset (Table 3). The 0.35 in the abstract is close to GPT-4.1's score, not to the Llama baseline. So the before/after comparison is not matched: it mixes task difficulty and possibly models. That is a load-bearing inconsistency, not a typo.\n\nThere are smaller issues. Table 4's text says picoCTF Pass@8 goes from 0.10 to 0.18, but the improvement column implies a baseline of 0.07. No error bars appear anywhere, and the Maj@8 values (0.14 to 0.24) suggest high variance across the eight samples. The reward function gives 0.5 for successful execution and 0.2 for a valid tool call, so a model can collect most of the reward without solving anything; the paper's own Figure 2 shows format and execution rewards jumping early while solve rate climbs slowly. Part of the gain is likely prompt-format compliance. That does not kill the result, but it means the current tables overstate how much is cryptanalytic.\n\nI do not think this is circular. The Random-Crypto test split is a held-out set from the generator, and the gains on picoCTF and AICrypto MCQ are external. The direction is sound. The paper is just not ready as-is: the baseline comparison needs to be corrected, results need variance reporting, and the easy-subset vs. full-set distinction needs to be explicit in the abstract.\n\nWho is this for? Anyone working on LLM agents for security or procedural RL environments. I would want to see a corrected version before citing the performance numbers, but the dataset itself is worth citing.\n\nRecommendation: send to peer review. It deserves a serious referee, and the revisions are manageable.","headline":"A useful dataset and a plausible RL result, but the headline Pass@8 comparison mixes task subsets and models, and needs a corrected baseline before it can be taken at face value.","tokens_in":10816,"tokens_out":1588,"would_cite":true,"duration_ms":16458,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that GRPO reinforcement learning on a procedurally generated cryptographic CTF dataset can raise a tool-augmented 8B model's Pass@8 on unseen crypto challenges from 0.35 to 0.88, with transfer to external benchmarks.","keywords":["Reinforcement learning","LLM agents","Cryptography","Capture The Flag","GRPO","Tool-augmented reasoning","Cybersecurity","Procedural dataset"],"falsifier":"Train the identical GRPO setup with the accuracy reward set to zero, or replaced with a reward for any boxed flag, while keeping all tool-format and execution rewards unchanged, and evaluate Pass@8 on the same easy test split. If it remains near 0.88, the gain is format compliance, not cryptanalysis; if it collapses to near base-model levels, the accuracy signal is doing the work. A complementary check is to evaluate the base model with few-shot demonstrations of the exact tool-call and boxed-answer format; if its Pass@8 jumps from 0.10 to near 0.88 without RL, format alone explains the result.","tokens_in":9676,"feed_emoji":"🔐","tokens_out":8026,"duration_ms":70576,"temperature":0.7,"pith_summary":"This paper claims that GRPO-style reinforcement learning on a procedurally generated cryptographic CTF dataset can turn a tool-augmented 8B language model into a much stronger solver of unseen crypto challenges. Pass@8, the fraction of tasks solved in at least one of eight attempts, rises from 0.35 to 0.88 on previously unseen Random-Crypto tasks, with the detailed easy-subset table showing the same final level from a 0.10 baseline. The training also transfers to external benchmarks: picoCTF Pass@8 rises from 0.10 to 0.18, and the curriculum-trained agent reaches 0.19 on AICrypto MCQ, a multiple-choice benchmark never seen in training. If true, this matters because it offers a scalable, verifiable training recipe for security agents that does not depend on huge proprietary models or manually labeled examples.","feed_headline":"RL lifts an 8B agent's crypto CTF solve rate from 0.35 to 0.88","feed_subtitle":"Group-relative policy optimization on procedurally generated puzzles also transfers to picoCTF and an MCQ benchmark.","key_machinery":"The two coupled mechanisms are Random-Crypto and GRPO with a shaped reward. Random-Crypto generates over five thousand training tasks from fifty randomized cryptographic scheme families, each with a unique flag and an LLM-written narrative, so the agent practices on infinitely varying verifiable challenges. GRPO, a PPO variant that ranks candidate outputs within a group, compares eight trajectories per prompt and updates the policy by relative reward, while the reward function gives 1.0 for a correct flag and smaller rewards for boxed-answer formatting, valid JSON tool calls, and error-free Python execution, with explicit penalties for hallucinated flags. The agent interacts with a persistent Python REPL server, allowing up to four tool calls per challenge, and training is made compute-efficient with QLoRA on a single A100 GPU.","core_discovery":"The core claim is that cryptographic CTF challenges form an ideal reinforcement learning environment—precise flag validation, structured multi-step reasoning, and deterministic tool execution—and that a procedurally generated supply of such challenges makes RL training practical. Fine-tuning Llama-3.1-8B with GRPO on easy Random-Crypto tasks, using a Python execution server and a composite reward for accuracy, answer format, valid tool calls, and clean execution, produces an agent that solves previously unseen easy crypto tasks at Pass@8 0.88–0.90, where the tool-augmented base model scored 0.10. The authors attribute the gains to improved procedural reasoning and tool use, shown by cleaner code execution and format adherence, and report that hint-free or curriculum training generalizes to picoCTF and to a multiple-choice cryptography benchmark the model never encountered during training.","pith_inferences":["The 0.35-to-0.88 Pass@8 headline may substantially reflect the base model's unfamiliarity with the tool-call prompt format; a controlled ablation that removes the accuracy reward would reveal how much of the gain is format compliance versus cryptanalytic skill.","If format compliance is the main driver, the same GRPO recipe could transfer to any domain with a verifiable final answer and a tool interface, making Random-Crypto a testbed for tool-use RL rather than for cryptography specifically.","The easy-only training choice hints that reward density, not raw task difficulty, governs RL success; procedurally generating tasks with calibrated difficulty could extend this approach to harder crypto and non-crypto CTF categories.","The authors' own sandboxing warning suggests a natural extension: make safe execution, through timeouts and memory limits, part of the reward so agents learn to avoid resource-exhausting code paths."],"forward_implications":["A small 8B open model trained this way can approach the Pass@8 of far larger proprietary models on unseen easy crypto challenges, suggesting a compute-efficient route to security-agent competence.","RL on procedural crypto tasks generalizes beyond the training distribution, improving performance on picoCTF's heterogeneous challenges and on a multiple-choice cryptography format the model never saw during training.","The early training gains consist mostly of learning to follow the tool protocol—code that runs without errors and flags that are not hallucinated—so format compliance is a learnable and necessary component of agent performance.","Curriculum training with hints first and no hints later gives the best transfer to new formats, while pure no-hint training gives higher Maj@8 consistency on the training distribution.","The observed memory-exhaustion failures when the agent generates heavy enumeration code imply that RL tool-use environments need resource sandboxing alongside accuracy rewards."],"supporting_citations":[{"why":"Introduces GRPO, the optimization algorithm used for training the agent.","marker":"[10]"},{"why":"Demonstrates GRPO on reasoning benchmarks, motivating its application here.","marker":"[11]"},{"why":"The base model that is fine-tuned and also serves as the main baseline.","marker":"[12]"},{"why":"Provides the picoCTF benchmark and evaluation framework used for transfer testing.","marker":"[6]"},{"why":"The multiple-choice cryptography benchmark used for generalization evaluation.","marker":"[22]"},{"why":"Enables the compute-efficient quantized fine-tuning on a single GPU.","marker":"[21]"},{"why":"The Python REPL execution server that provides the tool interface and whose memory failures motivate sandboxing.","marker":"[27]"}],"fun_headline_variants":["RL on crypto CTFs lifts 8B agent's unseen solve rate to 0.88","Procedural crypto CTFs make RL for LLM agents practical","GRPO on CTF puzzles transfers to picoCTF and MCQ benchmarks","Crypto CTF RL: 8B agent hits 0.88 Pass@8 on unseen puzzles","RL improves tool-augmented LLM agents via crypto CTF training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline result rests on the assumption that a held-out challenge sharing the same prompt format, tool interface, and procedural generation style as the training tasks is solved because of genuine cryptographic reasoning, rather than because the model learned to produce the expected tool-call and answer format.","fun_headline_variants_meta":{"raw":{"variants":["RL on crypto CTFs lifts 8B agent's unseen solve rate to 0.88","Procedural crypto CTFs make RL for LLM agents practical","GRPO on CTF puzzles transfers to picoCTF and MCQ benchmarks","Crypto CTF RL: 8B agent hits 0.88 Pass@8 on unseen puzzles","RL improves tool-augmented LLM agents via crypto CTF training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000821,"raw_usage":{"total_tokens":3576,"prompt_tokens":908,"completion_tokens":2668,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":2560}},"tokens_in":524,"tokens_out":2668,"duration_ms":17011,"temperature":1.0,"reasoning_tokens":2560,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:57:50.519541+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical GRPO setup with the accuracy reward set to zero, or replaced with a reward for any boxed flag, while keeping all tool-format and execution rewards unchanged, and evaluate Pass@8 on the same easy test split. If it remains near 0.88, the gain is format compliance, not cryptanalysis; if it collapses to near base-model levels, the accuracy signal is doing the work. A complementary check is to evaluate the base model with few-shot demonstrations of the exact tool-call and boxed-answer format; if its Pass@8 jumps from 0.10 to near 0.88 without RL, format alone explains the result.","supporting_citations":[{"cited_title":"1-8B-Instruct, 2024","cited_arxiv_id":null,"evidence_quote":"The base model that is fine-tuned and also serves as the main baseline."},{"cited_title":"AICrypto: Evaluating Cryptography Capabilities of Large Language Models","cited_arxiv_id":"2507.09580","evidence_quote":"The multiple-choice cryptography benchmark used for generalization evaluation."},{"cited_title":"Dettmers, A","cited_arxiv_id":null,"evidence_quote":"Enables the compute-efficient quantized fine-tuning on a single GPU."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Python REPL execution server that provides the tool interface and whose memory failures motivate sandboxing."}],"review_version":1}