{"id":"a28a887d-c09a-437a-be9a-b3907c0eb6a6","arxiv_id":"2507.09329","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"About 21% of OpenHands agent trajectories on SetupBench contained at least one insecure action, CWE-200 information exposure was the most common weakness, and GPT-4.1 showed the highest mitigation success at 96.8%.","lead":"Researchers tested five large language models acting as autonomous coding agents on 93 real-world software setup tasks and found that about 21% of agent sessions included at least one insecure action, such as hardcoding passwords or using overly broad file permissions. They also found that adding security reminders or automated feedback can correct many of these lapses, but the effect varies strongly by model, with GPT-4.1 responding best.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 21% trajectory rate and the 16–27% model ranking rest on an LLM detector that was tuned and evaluated on the same 500 labeled actions; with 61.11% recall and a 'prefer SECURE' prompt, these headline numbers are not independently established.","rationale":"I agree with the reader's weakest_assumption: the detector is the linchpin, since every headline number in Tables 2 and 4 is a product of that detector. The training-set evaluation is a textbook case of circular performance estimation, and the prompt's explicit SECURE bias makes false negatives systematic rather than random. Consequently, the true trajectory rate is likely higher than 21%, which would strengthen the safety warning but would invalidate the precise figures and the cross-model ranking. The remediation numbers are even more sensitive because success is scored by the same detector; the paper's self-reported offline, post-hoc design in §4.2 and the missing denominators further weaken Table 4. These issues do not demand rejection—the manual labels and concrete examples provide real evidence of insecure behaviors—but they do require conditional acceptance with held-out validation, variance reporting, and released artifacts. The reader's proposed conditions align with this conclusion, so the verdict remains CONDITIONAL.","tokens_in":11674,"tokens_out":6727,"duration_ms":75153,"concrete_test":"Hold out a fresh, stratified sample of ~300 actions from the full 12,000-action corpus (never seen during prompt tuning), have two independent security experts label each action, and run the final detection prompt on this held-out set. Report per-model precision and recall with 95% confidence intervals, plus label variance across 5 repeated runs at temperature=0. If held-out recall is materially below 61% or per-model recall varies by more than ~10 percentage points, the 21% trajectory rate and the model ranking in Table 2 are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claims—21% of trajectories with insecure steps, the per-model ranking in Table 2, and the 73.3%/96.8% remediation success rates in Table 4—all flow through the detection prompt described in §2.2. That prompt was iteratively optimized on the same 500 manually labeled steps that are then used for its performance evaluation, so the reported 98.6% accuracy, 100% precision, and 61.11% recall are training-set metrics, not independent estimates. The prompt itself instructs the model to treat task-appropriate actions as SECURE and to prefer SECURE when unsure (Appendix A), creating a strong prior against flagging. With recall at 61%, roughly 39% of expert-identified insecure actions are missed; if missed cases cluster by model or action type, the trajectory-level percentages and the remediation rates (which are scored by the same detector) can shift materially. The paper reports no repeated-run variance for the detector despite acknowledging LLM classifier nondeterminism in §4.2. This does not undermine the qualitative conclusion that agents sometimes take insecure actions—the 100% precision and concrete examples support that—but it does mean the precise headline numbers and the model comparison are not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an empirical study of insecure behaviors in LLM-based coding agents. The authors run the OpenHands agent on 93 SetupBench configuration tasks with five LLM backends (GPT-4o, GPT-4.1, Claude 3.5, Claude 3.7, Claude 4 Sonnet), use an LLM-based classifier to label agent actions as secure or insecure, and report that 21% of trajectories contain at least one insecure action, with per-model trajectory rates from 16.13% to 26.88%. They categorize insecure actions into four CWE-based classes, propose two mitigation strategies (security reminders and feedback), and report remediation success rates, with GPT-4.1 achieving 96.8% average remediation success. The paper also reports a negative association between insecure steps and task success.","tokens_in":11929,"tokens_out":4842,"duration_ms":49836,"significance":"If the quantitative results were independently validated, this would be a useful first systematic measurement of how often coding agents take security-relevant actions during routine setup tasks, with practical implications for model selection and guardrail design. The paper is transparent in publishing the detection and categorization prompts, providing concrete examples of flagged behaviors, and acknowledging the post-hoc nature of the mitigation evaluation. Its main weakness is that the detector that drives all reported rates is validated only on the same 500-step subset used to optimize it, with 61.11% recall and an explicit bias toward SECURE classifications; until this is addressed, the precise headline numbers and model rankings should be treated as provisional.","major_comments":[{"comment":"The performance metrics in Section 2.2 are computed on the same 500 manually labeled steps used to iteratively optimize the detection prompt, so 98.6% accuracy, 100% precision, and 61.11% recall are training-set estimates rather than independent validation. Because every downstream result in Tables 2 and 4 is produced by this detector, the headline trajectory rates and remediation rates inherit any bias in the detector. Please re-validate on a held-out set (or with cross-validation), report repeated-run variance given the nondeterminism acknowledged in Section 4.2, and provide a full confusion matrix.","section":"2.2"},{"comment":"The detection prompt in Appendix A explicitly instructs the classifier to prefer SECURE when unsure and to treat task-aligned actions as SECURE. Combined with the reported 61.11% recall, this means roughly 39% of expert-identified insecure actions are missed, and the paper does not analyze whether false negatives are distributed uniformly across models, action types, or CWE categories. If they cluster, the per-model ranking in Table 2 and the secure-versus-insecure success comparison in Figure 2 could shift materially. Please report false-negative distributions and a sensitivity analysis.","section":"2.2 / Appendix A"},{"comment":"Table 4 reports remediation success rates without denominators; with only 21 baseline insecure steps for GPT-4.1 (Table 2), 96.8% could correspond to a very small sample, and the same issue affects other cells and the trajectory percentages in Table 2 (e.g., GPT-4o 16.13% = 15/93). Please report raw counts, per-model and per-strategy denominators, and confidence intervals for every percentage. In addition, because Section 4.2 states that mitigation was evaluated offline and post-hoc, the term 'remediation success' should be qualified to reflect that no live agent interaction was tested.","section":"3.2 / Table 4"},{"comment":"The claim that secure trajectories consistently achieve higher success rates rests on very small per-model counts: for GPT-4.1, 31.2% of insecure trajectories corresponds to 5 of 16 trajectories, and one additional success changes the rate by more than six percentage points. Please report exact trajectory counts, event counts, and a statistical test (e.g., Fisher's exact test) or confidence intervals before framing this as a consistent security-performance trade-off.","section":"3.1.1 / Figure 2"}],"minor_comments":[{"comment":"The phrase 'fine-tuned detection prompt' is misleading; the paper optimizes a prompt, not model weights, so 'prompt-engineered' would be more accurate.","section":"2.2"},{"comment":"The statement that 56.61% of first insecure steps occur in the second half of trajectories lacks a per-model breakdown; please provide the supporting counts.","section":"3.1.1"},{"comment":"Percentages are reported without counts or confidence intervals; given the small totals (e.g., 21 for GPT-4.1), a single misclassification can change a category share substantially.","section":"3.1.2 / Table 3"},{"comment":"The caption says 'distribution of collected agent's actions' but does not state what the y-axis represents; clarify whether bars are action counts and add the task-success information referenced in the text.","section":"Figure 1"},{"comment":"Reference [18] lists authors 'Gabriel Costa, John Smith, and Alice Johnson,' which appear generic; verify that this is the correct bibliographic entry.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"To the editor: I believe the paper addresses a timely and publishable question, but the detector-validation gap is substantial and must be fixed before the quantitative claims can be taken at face value. The authors should be asked to provide an independent validation set, repeated-run variance estimates, and denominator reporting; if they cannot, the paper should be reframed as a qualitative exploration rather than a quantitative comparison. The manuscript may also benefit from a security-focused venue given the CWE framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the qualitative core is real and worth knowing: all five coding agents, running ordinary setup tasks with no security guardrails, do things like embed credentials in scripts, chmod 777 files, and pipe curl to sh. The examples in Appendix D are concrete, and the manual labeling found insecure steps, so this phenomenon is not an artifact of the detector. Second, the headline numbers—21% of trajectories, the per-model ranking, the 96.8% remediation rate—are not independently established. The detector was iteratively optimized on the same 500 labeled actions used to report its accuracy, precision, and recall, so those metrics are training-set metrics. With 61% recall and a prompt that instructs the model to prefer SECURE when unsure, the measured rates are likely lower bounds with unknown bias across models.\n\nWhat's actually new: this is the first comparative measurement I know of looking at default, non-adversarial agent behavior across multiple LLM backends on a common benchmark, with a CWE-based taxonomy and mitigation experiments. The taxonomy (CWE-200, 284, 494, 693) is sensible, and the observation that insecure trajectories are less likely to succeed is interesting and consistent across models. The authors are transparent about their method and honestly list limitations in Section 4.2.\n\nSoft spots: besides the evaluation leakage, Table 4 gives no denominators, so GPT-4.1's 96.8% could rest on a handful of flagged steps. There is no held-out validation, no repeated-run variance for the detector, and no artifacts or data released, which makes it impossible to verify the CWE categorization or the trajectory rates. The paper acknowledges LLM detector nondeterminism but doesn't quantify it. None of this undercuts the qualitative conclusion, but it does mean the precise numbers should be read as illustrative, not as security benchmarks for model-selection decisions.\n\nWho this is for: people building guardrails for coding agents, security teams evaluating LLM backends, and researchers studying evaluation methodology for agent safety. The paper is a useful wake-up call that agent behavior in normal operation deserves security review.\n\nRecommendation: worth sending to peer review. A serious referee should require release of trajectories and labels, a held-out validation of the detector, and denominators for the mitigation table. With those changes the quantitative claims could be made solid; without them, the paper is a useful qualitative study with unreliable point estimates.","headline":"The qualitative finding is credible and useful—agents do take insecure actions during routine setup tasks—but the headline rates rest on a detector validated on its own tuning set, so the specific numbers should be treated as provisional, not as benchmarks.","tokens_in":12465,"tokens_out":2612,"would_cite":true,"duration_ms":30512,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A systematic evaluation of five leading coding agents on 93 real-world setup tasks finds that 21% of trajectories contain insecure actions, that secure trajectories complete tasks more often, and that feedback-based remediation fixes…","keywords":["LLM coding agents","agent security evaluation","insecure actions","CWE taxonomy","information exposure","access control","feedback mitigation","software setup tasks"],"falsifier":"Re-score the same 12,000-plus actions with a deterministic rule-based detector for the four named patterns (hardcoded credentials, chmod 666/777, curl-pipe-to-shell, and disabled SSH host-key or database auth) and compare trajectory-level rates against the reported 21.0%. If deterministic scoring finds the rate below about 15% across models, the paper's headline claim overstates the problem; if it matches or exceeds 21%, the claim holds. Running the LLM detector repeatedly on identical trajectories would also settle whether the model ranking is stable.","tokens_in":11463,"feed_emoji":"🔐","tokens_out":6629,"duration_ms":75331,"temperature":0.7,"pith_summary":"This paper tries to establish that LLM-based coding agents, left to their default settings on routine software setup tasks, regularly take actions that introduce exploitable security weaknesses, and that these lapses are not rare edge cases. Analyzing more than 12,000 agent actions across five leading language models on 93 real-world setup tasks, it reports that 21% of task trajectories contained at least one insecure action, with model-level trajectory rates between 16.13% and 26.88%. It also claims a consistent link between security and task success: trajectories free of insecure steps finished the task more often than those containing insecure steps for every model. Finally, it evaluates two mitigation strategies and reports that real-time feedback corrected 73.3% of flagged actions on average, with the best model reaching 96.8%.","feed_headline":"One in five coding-agent tasks includes an insecure action","feed_subtitle":"Across 93 setup tasks and five models, hardcoded credentials dominate; feedback fixes 73% of flagged steps.","key_machinery":"The load-bearing mechanism is a two-stage, prompt-based detection pipeline rather than a formal analyzer. A custom classification prompt, optimized on 500 human-labeled steps, takes each agent action together with the task assignment and the agent's own reasoning and labels the action secure or insecure, reaching 98.6% accuracy, 100% precision, and 61.11% recall on that same labeled set; a second categorization prompt maps flagged actions to four CWE classes. The pipeline is what converts raw trajectories into every reported rate, so the entire argument runs through it.","core_discovery":"The paper's central discovery is that insecure behavior is a normal byproduct of current coding agents rather than an adversarial trigger. On a standardized benchmark of 93 software setup tasks, an open-source coding agent backed by five state-of-the-art LLMs produced at least one action classified as insecure in 21% of trajectories; the most common failure was CWE-200, exposure of sensitive information such as hardcoded credentials, followed by improper access control (CWE-284) and downloading code without integrity checks (CWE-494). The same data show a security-success correlation: for all five models, trajectories without insecure steps had higher task completion rates than trajectories with insecure steps, with the largest gap being 24.1 percentage points. The paper further claims that mitigation works unevenly: feeding the agent the detector's reasoning as feedback remediated 73.3% of insecure actions on average, while one model reached 96.8% and another only 54.8%, and that proactive security reminders were less effective than targeted feedback.","pith_inferences":["Inference: Because the detector's recall is only 61% and it is biased toward 'secure' in ambiguous cases, the 21% trajectory rate is more likely an undercount than an overcount; a more complete detector would probably push the rate higher while possibly changing the model ordering.","Inference: The consistent success gap between secure and insecure trajectories suggests insecurity is a marker of task difficulty or poor trajectory quality, so a runtime security check could double as a progress signal, not just a guardrail.","Inference: The mitigation numbers come from offline post-hoc re-scoring, not live intervention; in a real agent loop, interrupting an action would likely change both remediation success and task completion, so the 73.3% average is an upper-bound estimate.","Inference: The same method could be extended to longer code-editing tasks and to agent-visible tests; a testable prediction is that the 56.61% late-trajectory concentration of first insecure steps grows with context length, since the paper cites long-context degradation as a candidate cause."],"forward_implications":["Without added security guardrails, an enterprise deploying such agents should expect roughly one in five setup tasks to include at least one security-relevant mistake.","Insecure trajectories are less likely to complete their task, so security failures are not merely hypothetical risks; they co-occur with functional failure.","Hardcoded credentials and overly broad permissions are the dominant failure classes, so credential-management tooling and least-privilege defaults would address most observed issues.","Feedback that explains why an action is insecure is more effective than generic security reminders, supporting runtime guardrails over prompt-only guidance.","Mitigation effectiveness varies widely by model, so model choice determines how much benefit guardrails will deliver."],"supporting_citations":[{"why":"Supplies the open-source agent platform whose trajectories are the study's data.","marker":"[5]"},{"why":"Supplies the 93 real-world software setup tasks the agent was run on.","marker":"[6]"},{"why":"Documents that LLMs cannot reliably identify security vulnerabilities, supporting the paper's detector-nondeterminism caveat.","marker":"[13]"},{"why":"Shows LLMs err on easy problems, used to motivate caution about repeated classifier runs.","marker":"[14]"},{"why":"Establishes the broader agent-safety evaluation paradigm this work extends to coding agents.","marker":"[15]"},{"why":"Supplies a benchmark for risky code execution in code agents, a comparison point for the insecurity taxonomy.","marker":"[21]"},{"why":"Prior evidence that LLM code generation produces vulnerable code, the baseline this work extends to agent actions.","marker":"[22]"},{"why":"Shows LLMs can be guided to fix their own code flaws, motivating the feedback mitigation strategy.","marker":"[9]"}],"fun_headline_variants":["21% of coding agent tasks include an insecure action","Hardcoded credentials are the top coding agent security leak","Feedback fixes 73% of insecure coding agent actions","Coding agents' security gaps vary by model, from 55% to 97%","Even the best coding agent model still has security flaws"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported rates and rankings depend on the LLM-based detector being a valid and sufficiently complete measure of what counts as insecure; the detector was tuned and evaluated on the same 500 labeled steps, has 61% recall, and was explicitly instructed to prefer 'secure' when unsure, so undiscovered false negatives could shift every headline number.","fun_headline_variants_meta":{"raw":{"variants":["21% of coding agent tasks include an insecure action","Hardcoded credentials are the top coding agent security leak","Feedback fixes 73% of insecure coding agent actions","Coding agents' security gaps vary by model, from 55% to 97%","Even the best coding agent model still has security flaws"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000317,"raw_usage":{"total_tokens":1779,"prompt_tokens":914,"completion_tokens":865,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":780}},"tokens_in":530,"tokens_out":865,"duration_ms":10943,"temperature":1.0,"reasoning_tokens":780,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:57:52.691311+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the same 12,000-plus actions with a deterministic rule-based detector for the four named patterns (hardcoded credentials, chmod 666/777, curl-pipe-to-shell, and disabled SSH host-key or database auth) and compare trajectory-level rates against the reported 21.0%. If deterministic scoring finds the rate below about 15% across models, the paper's headline claim overstates the problem; if it matches or exceeds 21%, the claim holds. Running the LLM detector repeatedly on identical trajectories would also settle whether the model ranking is stable.","supporting_citations":[{"cited_title":"Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H","cited_arxiv_id":null,"evidence_quote":"Supplies the open-source agent platform whose trajectories are the study's data."},{"cited_title":"Setupbench: Assessing software engineering agents’ ability to bootstrap development environments","cited_arxiv_id":null,"evidence_quote":"Supplies the 93 real-world software setup tasks the agent was run on."},{"cited_title":"Llms cannot reliably identify and reason about security vulnerabilities (yet?): A compre- hensive evaluation, framework, and benchmarks","cited_arxiv_id":null,"evidence_quote":"Documents that LLMs cannot reliably identify security vulnerabilities, supporting the paper's detector-nondeterminism caveat."},{"cited_title":"Easy problems that llms get wrong","cited_arxiv_id":null,"evidence_quote":"Shows LLMs err on easy problems, used to motivate caution about repeated classifier runs."},{"cited_title":"Agent-safetybench: Evaluating the safety of llm agents","cited_arxiv_id":null,"evidence_quote":"Establishes the broader agent-safety evaluation paradigm this work extends to coding agents."},{"cited_title":"Redcode: Risky code execution and generation benchmark for code agents","cited_arxiv_id":null,"evidence_quote":"Supplies a benchmark for risky code execution in code agents, a comparison point for the insecurity taxonomy."},{"cited_title":"thought","cited_arxiv_id":null,"evidence_quote":"Prior evidence that LLM code generation produces vulnerable code, the baseline this work extends to agent actions."},{"cited_title":"Guiding ai to fix its own flaws: An empirical study on llm-driven secure code generation","cited_arxiv_id":null,"evidence_quote":"Shows LLMs can be guided to fix their own code flaws, motivating the feedback mitigation strategy."}],"review_version":1}