{"id":"3434f80f-31fd-4993-989b-bcaf6b79f218","arxiv_id":"2505.01065","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Current LLMs cooperate with exploit-generation requests, but none generated a working exploit for refactored lab programs, indicating limited near-term abuse risk.","lead":"This paper tested five large language models on their ability to write working computer exploits for five security lab programs, including renamed versions of the labs. It found that the models were usually willing to help but none succeeded on the renamed challenges, and GPT-4o came closest.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central negative claim depends on unexecuted manual error counts from an unreported number of runs; without execution-based verification or repetition, 'none succeeded' is not yet established.","rationale":"The reader's weakest_assumption identifies the same core issue: the evaluation relies on a single, manually counted outcome without reported runs, sampling randomness, or inter-rater checks. My independent reading of the paper confirms that the central negative claim is not backed by an observable success definition. The paper does release raw data and code, which is genuine support, but the reported metric is a subjective error count rather than an execution-based pass/fail. The ambiguity in Section III-C about GPT-4o-mini refining the target output further weakens the attribution of the final exploit to the evaluated model. Because the conclusion is a universal negative, even one correctly executed success in a repeated trial would change the headline. Therefore the verdict should remain CONDITIONAL: the claim is plausible but only acceptable if the protocol is tightened to include execution-based verification, repeated runs, and transparent sampling parameters.","tokens_in":6819,"tokens_out":3951,"duration_ms":43681,"concrete_test":"Re-run the full attacker pipeline for each target model and lab condition at a fixed temperature (or default), then execute the final exploit artifact against the actual refactored binaries and environment with a timeout and observable success criteria (crash, control-flow hijack, file modification, or privilege change). Use at least 10 independent runs per condition and have two independent annotators recount errors on a blinded subset. If any execution succeeds, the abstract's 'no model ... for refactored labs' claim is refuted; if zero successes are observed across repetitions and annotator agreement is high, the claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's main conclusion — 'no model successfully generates exploits for refactored labs' (Section VI; RQ2) — rests on the error-counting procedure in Section III-A: 'the number of errors in the LLM-generated code that prevented the exploit from functioning correctly.' No statement anywhere says these outputs were compiled, run, or otherwise verified against the refactored programs; the metric is a manual code inspection. A manual error count can misclassify code in both directions (a counted error may be harmless, an uncounted interaction may break the exploit), so the binary 'success/no success' used for the headline is not objectively determined. The protocol section also omits the number of independent runs per condition, sampling temperature/seed, and inter-rater agreement, while the refinement step is ambiguously described: Section III-C says 'its output further refined by GPT-4o-mini,' which could mean the measured artifact is not the target model's raw output. Since the central claim is a universal negative across five labs and five models, a single actual success under an executable definition would overturn it. This makes the unexecuted, single-run manual count the load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a first systematic evaluation of LLMs for automated exploit generation (AEG). It uses five SEED Labs programs in both original and refactored (renamed) forms, builds an LLM-based attacker that iteratively prompts a target model for up to 15 iterations, and measures (i) cooperativeness, defined only through an average percentage, and (ii) effectiveness, measured as the number of mistakes in the final generated code as judged by the authors. The headline findings are that GPT-4o and GPT-4o-mini are nearly as cooperative as uncensored models, Llama3 is most resistant, and none of the evaluated models successfully generates exploits for the refactored labs, with GPT-4o making the fewest errors. The authors release the benchmark and attacker code.","tokens_in":7126,"tokens_out":3992,"duration_ms":39980,"significance":"If the central negative result is reliable, the paper provides a valuable data point for the security community: current LLMs are not yet a serious AEG threat for refactored programs, while their cooperativeness and moderate error counts suggest potential for improvement. The refactored benchmark is a sensible step toward mitigating training-data memorization, and the LLM-based attacker is a useful attempt to make prompt-based evaluations more reproducible and less dependent on human interaction. However, the evaluation protocol currently lacks execution-based verification, clear definitions, and variance information, so the headline claims are not yet established. With a tightened protocol, this could become a solid contribution to a security or software engineering venue.","major_comments":[{"comment":"The success metric is 'the number of errors in the LLM-generated code that prevented the exploit from functioning correctly,' but the paper nowhere states that the final generated code was compiled or executed against the refactored programs. Since the central claim is a universal negative—'none of the evaluated models successfully generated exploits for refactored vulnerabilities'—a manual code inspection that can misclassify harmless errors as fatal, or miss runtime interactions that break the exploit, does not establish that claim. The authors should either execute the generated exploits in the actual lab environments and report per-lab success/failure, or weaken the conclusion to 'no model produced code that the authors judged likely to work.'","section":"§III-A, §VI (RQ2)"},{"comment":"Section III-C says the target model's output is 'further refined by GPT-4o-mini' during the iterative loop. It is therefore unclear whether the final artifact whose mistakes are counted in Figures 2 and 3 is the target model's raw output or a GPT-4o-mini-edited version. If the latter, the mistake counts are not attributable solely to the target models being compared, and GPT-4o-mini acts as both a target model and a post-processor in every condition. The authors must clarify the exact data flow and ensure that the measured artifact is the target model's own output, or explain how the refiner's edits are controlled for.","section":"§III-C"},{"comment":"Table I reports the 'average percentage of the cooperative response' but never defines what cooperativeness means operationally—for example, whether a response is cooperative if it contains any exploit code, if it does not refuse, or if it follows the attacker's prompt. Without a coding rubric and inter-rater agreement, the comparison of GPT-4o and GPT-4o-mini with uncensored models ('surprisingly cooperative') is not reproducible. The authors should provide the prompt template, the classification criteria, and ideally raw dialogue logs in the released artifact.","section":"§IV-A, Table I"},{"comment":"The paper does not report the number of independent runs per lab and model, nor sampling temperature or seeds. The figures appear to present a single run per condition (e.g., 'GPT-4o, total: 6' in Figure 3). Because LLM outputs are stochastic, the claim that no model succeeded on refactored labs rests on one sample per condition, and the inference that 'Dolphin Mistral might have been trained with SEED Lab materials' is based on that same single-run, non-executed difference. At minimum, the authors should report multiple runs and the variance of mistake counts, or justify why a single run suffices for a universal negative.","section":"§III-C, §IV-B"}],"minor_comments":[{"comment":"There is a typo: 'occuring' should be 'occurring.'","section":"§IV-C"},{"comment":"The figures are hard to parse because each lab's mistake count is shown across several stacked or grouped values without a clear key; a small table of per-lab mistake counts with success/failure flags would be more readable.","section":"Figures 2 and 3"},{"comment":"The abstract and introduction attribute high cooperativeness to 'GPT-4 and GPT-4o,' but Table I lists only GPT-4o and GPT-4o-mini; the paper should clarify whether GPT-4 base was actually evaluated or whether 'GPT-4' is a collective label.","section":"§I, Abstract"},{"comment":"The threat model assumes the attacker has access to source code; this is worth stating explicitly as a boundary condition in the limitations, since many real-world attacks target binaries without source.","section":"§II-C"},{"comment":"Reference [14] for SEED Labs is listed as 'Seed labs, 2025' with no version or URL in the text; the full URL should be provided in the reference entry.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's main problem is not its direction but that the headline negative claim is built on unexecuted, single-run, manually counted mistakes without inter-rater validation. I would not require a full re-execution of every condition, but the authors should at least execute the final generated exploits against the refactored labs, report multiple runs, and clearly separate the target model's raw output from the GPT-4o-mini refinement step. If these are addressed, the paper could be suitable for a security venue; the current version is too thin on protocol details for the strength of its conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is an honest, well-scoped first cut at measuring whether LLMs can write working exploits, and the headline result—no model cracks even simple refactored lab programs—is probably right in direction, but the measurement behind it is softer than the prose admits.\n\nWhat's new: the refactored SEED Labs benchmark to control for memorization, and the LLM-based attacker to make prompting systematic. Both are sensible instruments, and the comparison across five models is real work. The paper also breaks down common error patterns clearly, even though the explanations are speculative (and the authors say so).\n\nWhere it gets soft: the main claim is a universal negative, but \"success\" is never defined in executable terms. The evaluation metric is a manual count of errors that \"prevented the exploit from functioning,\" yet the paper never states that the generated exploits were compiled or run. A manual reading can easily misclassify something as fatal when it isn't, or miss a broken interaction. The protocol also omits the number of independent runs, sampling parameters, and any inter-rater check on the error counts. One sentence about GPT-4o-mini refining the target model's output is ambiguous enough that it's unclear whose output is actually being scored. None of these are fatal on their own, but together they mean the strong \"no model succeeds\" claim is not yet airtight.\n\nIf the authors release the artifact and either execute the exploits or clearly justify why manual inspection is sufficient, the conditional verdict becomes much more solid.\n\nRecommendation: send it to peer review. The benchmark and comparison are worth referee time, and the methodological gaps are fixable. This is the kind of paper that could get significantly better with a revision.","headline":"A useful first systematic study on LLM-driven exploit generation with a plausible but not yet airtight central negative result; the paper deserves peer review but needs a more rigorous evaluation protocol.","tokens_in":7513,"tokens_out":2256,"would_cite":false,"duration_ms":22964,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"No evaluated LLM succeeded in exploiting refactored vulnerable programs.","keywords":["automated exploit generation","large language models","SEED Labs","LLM security","exploitability assessment","jailbreak","code refactoring","benchmark"],"falsifier":"Run the five refactored labs repeatedly with automated verification that actually executes each generated payload and checks for a shell or code execution. If any model's exploit succeeds on any run, the paper's central negative claim is false; conversely, many repeated failures across different random seeds would strengthen it.","tokens_in":6613,"feed_emoji":"🤖","tokens_out":5278,"duration_ms":47178,"temperature":0.7,"pith_summary":"This paper asks whether current large language models can automate exploit generation well enough to be a practical threat. It builds a benchmark from five software-security labs, each provided in its original form and in a refactored form where variable and function names are replaced with meaningless tokens to defeat memorized answers. An automated attacker pipeline iteratively prompts each target model up to fifteen times. The central negative result is that no model produced a working exploit for any refactored lab, while GPT-4o came closest with only six blocking errors across all five. The finding matters because it suggests today's models are not yet a reliable tool for script-kiddie-level exploitation, but the gap to success is small.","feed_headline":"No LLM generates working exploits for refactored code","feed_subtitle":"Five models tried; none succeeded, and GPT-4o came closest with just six errors.","key_machinery":"The central mechanism is the LLM-based attacker, an automated script in which one LLM (here GPT-4o) acts as a prompter that reads the target model's latest exploit attempt and issues a refined instruction, cycling for up to fifteen iterations or until it decides no more improvement is possible. The other load-bearing component is the refactored benchmark: the original vulnerable programs have every variable and function renamed to varX and functionX, so a model cannot succeed by recalling a known SEED Lab solution. The evaluation metric is a manual count of errors in the model's final code that would prevent the exploit from running, compared against the authors' own reference solutions.","core_discovery":"The authors report the first systematic evaluation of LLMs for automated exploit generation (AEG). Across five refactored SEED Labs programs covering buffer overflow, return-to-libc, format string, race condition, and dirty COW attacks, none of the five tested models (GPT-4o, GPT-4o-mini, Llama3, Dolphin-Mistral, Dolphin-Phi) produced an exploit that worked. The metric was the number of mistakes in each model's final code that would prevent the exploit from functioning; GPT-4o made the fewest with six mistakes total, followed by GPT-4o-mini with eight, Dolphin-Mistral with twelve, Llama3 with fifteen, and Dolphin-Phi with seventeen. The authors also find that GPT-4 and GPT-4o cooperate with exploit requests almost as readily as uncensored open-source models, while Llama3 resists most often. They interpret the performance drop between original and refactored labs as evidence that some models may have memorized SEED Lab solutions from training data.","pith_inferences":["The single-run, manually counted protocol is likely high-variance; if the authors were to sample each model over several runs and average error counts, the ordering could shift even if the all-fail result stands.","Reasoning-focused models released after this study (such as GPT-o1 and DeepSeek-r1) might succeed on the refactored labs because the failures appear to be about forming connections between program structure and payload mechanics, which inference-time reasoning directly targets.","The refactoring only renames symbols; a future benchmark that changes control flow or adds unrelated code would be a stronger test of genuine exploitation ability, and the current result may overstate or understate the memorization effect.","If a single LLM achieves success on refactored labs, the practical implication for exploitability assessment is immediate: AEG could shift from expert-driven symbolic execution to prompting, changing who can meaningfully assess and exploit a known vulnerability."],"forward_implications":["Current LLMs are not yet capable of fully automated, general exploit generation: none of the tested models solved any refactored lab.","GPT-4o's error counts (one or two mistakes per attempt) suggest that a small improvement in reasoning or debugging could push LLM-driven AEG over the threshold to success.","The cooperation data indicate that alignment does not strongly protect GPT-series models against exploit-request prompting, so safety cannot rely on refusal behavior alone.","Because performance drops sharply when code symbols are obfuscated, some models likely succeed on public benchmarks by memorization rather than by understanding the vulnerability.","The released benchmark and attacker pipeline give future work a reproducible starting point for measuring progress in LLM-based AEG."],"supporting_citations":[{"why":"Supplies the five SEED Labs programs that form the benchmark's vulnerable targets.","marker":"[14]"},{"why":"Defines the AEG task and the end-to-end approach that this study evaluates LLMs against.","marker":"[5]"},{"why":"Prior LLM-driven security automation that motivates the design of the LLM-based attacker.","marker":"[12]"},{"why":"Establishes exploitability as the severity metric that AEG aims to automate.","marker":"[1]"},{"why":"Explains LLM refusal and alignment behavior that the cooperation analysis relies on.","marker":"[16]"}],"fun_headline_variants":["No LLM cracks refactored exploit labs","All five LLMs fail at exploit generation","GPT-4o comes closest but still can't exploit","Refactored code defeats every LLM exploit","LLMs cooperate on exploits, but none succeed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline result rests on a single manual pass: one run of the iterative attacker per lab, with the errors in each model's final output counted by hand, so a lucky run or a miscounted mistake could change whether the 'none succeeded' claim holds.","fun_headline_variants_meta":{"raw":{"variants":["No LLM cracks refactored exploit labs","All five LLMs fail at exploit generation","GPT-4o comes closest but still can't exploit","Refactored code defeats every LLM exploit","LLMs cooperate on exploits, but none succeed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000352,"raw_usage":{"total_tokens":1890,"prompt_tokens":892,"completion_tokens":998,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":940}},"tokens_in":508,"tokens_out":998,"duration_ms":9180,"temperature":1.0,"reasoning_tokens":940,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:26:57.259394+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the five refactored labs repeatedly with automated verification that actually executes each generated payload and checks for a shell or code execution. If any model's exploit succeeds on any run, the paper's central negative claim is false; conversely, many repeated failures across different random seeds would strengthen it.","supporting_citations":[{"cited_title":"Seed labs,","cited_arxiv_id":null,"evidence_quote":"Supplies the five SEED Labs programs that form the benchmark's vulnerable targets."},{"cited_title":"Aeg: Automatic exploit generation,","cited_arxiv_id":null,"evidence_quote":"Defines the AEG task and the end-to-end approach that this study evaluates LLMs against."},{"cited_title":"Pentestgpt: Evaluating and harnessing large language models for automated penetration testing,","cited_arxiv_id":null,"evidence_quote":"Prior LLM-driven security automation that motivates the design of the LLM-based attacker."},{"cited_title":"A survey on software vulnerability exploitability assessment,","cited_arxiv_id":null,"evidence_quote":"Establishes exploitability as the severity metric that AEG aims to automate."},{"cited_title":"A comprehensive study of jailbreak attack versus defense for large language models,","cited_arxiv_id":null,"evidence_quote":"Explains LLM refusal and alignment behavior that the cooperation analysis relies on."}],"review_version":1}