{"id":"02e60d63-be4e-44e2-904e-88067fdfdad6","arxiv_id":"2507.22063","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-turn red-teaming agent trained on simulated attacker-defender conversations induces vulnerable code at higher rates than prior attack methods across several code LLMs.","lead":"RedCoder is an automated agent that chats with code-generating AI models over multiple turns to trick them into writing insecure code. It trains on conversations produced by a multi-agent game and retrieves proven attack strategies to adapt its prompts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation hinges on CodeGuru as both training filter and test oracle; a 20-conversation human check is too small to rule out that REDCODER games CodeGuru's flags rather than inducing genuinely vulnerable code.","rationale":"I reviewed the paper against the reader's verdict. The reader's weakest_assumption about CodeGuru as both training signal and evaluation oracle matches my analysis: the entire pipeline—gaming evaluator, strategy distillation, and final metric—uses CodeGuru, so REDCODER is optimized to trigger CodeGuru. The human validation in Appendix D is far too small (20 conversations, one annotator, one victim model) to establish that the flags correspond to genuine vulnerabilities across the test set. The OOD experiment helps with task reuse but does not address the oracle concern since it uses the same detector. The concern is addressable by an independent labeling study, so CONDITIONAL remains the appropriate verdict; no change needed.","tokens_in":16263,"tokens_out":4032,"duration_ms":41326,"concrete_test":"Independently re-label all snippets from the Table 1 evaluation (or a stratified random sample of at least 100 conversations per victim model) using CodeQL or Semgrep security suites plus two human annotators blind to CodeGuru labels, resolving disagreements with a third; recompute Vulnerability Rates. If REDCODER's absolute rates fall by more than 10 percentage points, or its advantage over the best baseline shrinks below statistical significance (e.g., via bootstrap), the CodeGuru circularity is material and the headline should be qualified. This single check settles whether the reported rates reflect real vulnerabilities or CodeGuru-specific patterns.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that REDCODER outperforms baselines in inducing vulnerable code. The metric everywhere is CodeGuru: Section 2.2 uses CodeGuru as the evaluator to filter prototype conversations, the strategy analyst distills tactics from CodeGuru-labeled successes and failures, and Section 3.1 measures Vulnerability Rate with CodeGuru. Consequently, REDCODER is explicitly optimized to produce code that CodeGuru flags, while the baselines are not. If CodeGuru's flags have any systematic false-positive bias for the kinds of code REDCODER learns to emit, the reported absolute rates (61.18%, 65.29%) and the relative gains over baselines would overstate practical risk. The only external check, Appendix D, is 20 CodeGemma conversations labeled by a single PhD student, with 90% agreement on vulnerability presence; the sample is far too small to establish precision, and the note itself identifies context-dependent disagreements that are conservatively treated as non-vulnerable, whereas the main metric counts any CodeGuru CWE flag as success. This makes oracle reliability load-bearing: if CodeGuru is accurate on this distribution, the claim stands; if not, the method may be gaming the oracle.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents RedCoder, a multi-turn red-teaming agent for Code LLMs. RedCoder is constructed by a multi-agent gaming process in which an attacker LLM interacts with a defender (a coding agent plus a custom multi-turn guardrail), an evaluator (Amazon CodeGuru) labels whether any response contains CWE vulnerabilities, and a strategy analyst distills reusable tactics from failed-to-successful conversation transitions. The successful prototype conversations are used to fine-tune a Llama-3-8B backbone, and at deployment a retrieval-augmented module retrieves strategy summaries to guide each turn. Experiments on CodeLlama-7B, CodeGemma-7B, Qwen2.5-Coder-7B, and DeepSeek-R1-Distill-Llama-8B report vulnerability rates between 39.41% and 65.29%, outperforming AutoDAN, GCG, CoA-Feedback, and ActorAttack. The paper also reports an out-of-distribution evaluation on 20 new CWE tasks, a small human validation of CodeGuru, and a defense study comparing single-turn and multi-turn guardrails.","tokens_in":16473,"tokens_out":9128,"duration_ms":97687,"significance":"If the reported numbers survive scrutiny, RedCoder is a timely and useful contribution: it addresses a realistic multi-turn setting, automates the pipeline without per-task human effort, and provides a reusable strategy arsenal with retrieval-augmented deployment. The OOD experiment in Appendix E.1 (65.00% on 20 held-out CWE types vs. 65.29% on the original set) is a genuine strength, as is the head-to-head comparison against four published baselines and the inclusion of Algorithm 1 for reproducibility. The paper does not currently provide code or per-task results, but the core idea is clear and falsifiable. The main risk is measurement validity: because the same detector (CodeGuru) serves as both training signal and evaluation oracle, the absolute rates and relative gains could partly reflect detector-specific optimization rather than genuine vulnerability induction.","major_comments":[{"comment":"The central evaluation metric, Vulnerability Rate, is computed by Amazon CodeGuru (§3.1), and CodeGuru is also the evaluator used during the gaming process to filter successful prototype conversations and to create the transition pairs used to distill attack strategies (§2.2). RedCoder is therefore trained and retrieval-augmented on CodeGuru-positive examples, while none of the baselines is optimized against this detector. Consequently, a systematic false-positive tendency of CodeGuru on the particular code patterns RedCoder learns to emit would inflate both the absolute rates in Table 1 and the relative gains over baselines. Appendix D validates CodeGuru on only 20 CodeGemma conversations with a single annotator; this sample is too small to establish precision on the adversarial distribution, and the two observed disagreements are CodeGuru false negatives, which do not directly address the false-positive concern. Please substantially expand the human-validated sample (across victim models and including conversations that CodeGuru flags), or replace the oracle with an independent detector plus human verification, and report Vulnerability Rates on human-confirmed snippets.","section":"§2.2 and §3.1 (Evaluator and Evaluation Details)"},{"comment":"Table 1 reports each vulnerability rate as a point estimate with no confidence intervals, repeated runs, or significance tests. The headline comparisons (e.g., RedCoder 65.29% vs. ActorAttack 8.24% on Qwen2.5-Coder-7B) are large, but several smaller gaps (e.g., DeepSeek-R1-Distill-8B: RedCoder 40.00% vs. AutoDAN 22.49%) may be within sampling noise for 170 conversations. Please report bootstrap confidence intervals over the 170 tasks, or repeated runs with different random seeds, and a paired significance test (e.g., McNemar or permutation test) over the same task set for all methods.","section":"Table 1 and §3.1 (Main Results)"},{"comment":"The main benchmark includes the same 43 seed tasks used in the gaming process that generated RedCoder's fine-tuning conversations and strategy arsenal (§3.1). The authors state that the conversations are distinct because the defender differs from test-time victims, but the task instructions and target CWE types are shared, so RedCoder has an in-distribution advantage over the baselines on those tasks. The OOD experiment in Appendix E.1, which uses 20 disjoint CWE types and achieves 65.00% vs. 65.29%, is encouraging and largely answers this concern; however, it is confined to one victim model (Qwen2.5-Coder-7B) and reported only in the appendix. Please make the OOD results a primary result, or clearly separate in-distribution and OOD performance in the main table, and extend the OOD evaluation to at least one more victim model.","section":"§3.1 (Dataset) and Appendix E.1"}],"minor_comments":[{"comment":"The text says 'As shown in Tab. 8' when discussing the single-turn vs. multi-turn guardrail defense results, but Table 8 in Appendix G reports guardrail detection rates on 500 conversations, not the end-to-end defense vulnerability rates; the reference should point to Table 2.","section":"§3.4 and Table 2"},{"comment":"Model names are inconsistent: Table 1 uses 'Qwen-2.5-Coder-7B' and 'DeepSeek-R1-Distill-8B', while §3.1 uses 'Qwen-Coder-7B' and 'DeepSeek-R1-Distill-Llama-8B'; please unify the names across the paper.","section":"§3.1 and Table 1"},{"comment":"The related-work section cites AutoDAN as Liu et al. (2024), while the baseline implementation in Appendix B cites Liu et al. (2025) and links to AutoDAN-Turbo; please clarify which version is actually evaluated and cite it consistently.","section":"§4, §B, and references"},{"comment":"Appendix D says 'A PhD student studying' without completing the sentence; also, the human-validation protocol should state whether the annotator saw the CodeGuru labels and how CWE-level disagreements were adjudicated.","section":"Appendix D"},{"comment":"The guardrail decision is defined as arg max P(g|...), but the paper does not specify how P is obtained from the guardrail model; please provide the exact scoring function or logit extraction procedure.","section":"§2.2 (Defender)"},{"comment":"Algorithm 1 line 14 says the evaluator detects 'CWE vulnerabilities or malicious cyberactivity', but malicious cyberactivity is not defined or used elsewhere in the paper; please remove or define this term.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable fit for a security or software-engineering venue. The funding from Amazon and the use of CodeGuru are not themselves problematic, but because the evaluation oracle is also the training objective, I would ask the editor to ensure the revised version includes an independent validation of the detector on the actual attack distribution. I see no evidence of misconduct; the issues are measurement validity and reporting transparency. The recommended decision is major_revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read RedCoder. The core idea is new and practical: a multi-agent gaming loop that produces prototype multi-turn conversations and distills failure-to-success transitions into a strategy arsenal, then fine-tunes a small LLM with RAG over that arsenal. That is a sensible way to automate red teaming in a domain where single-turn attacks transfer poorly. Head-to-head results are dramatic: 39-65% vulnerability rates versus 1-33% for prior methods, and the paper shows the gains hold on an out-of-distribution set of 20 new CWE types and on Claude 3.5 Sonnet. Credit where due: the authors also report an ablation of the retrieval component, a guardrail comparison, and a human validation of the evaluator. The writing is honest—they state limitations and discuss CWE abstraction levels.\n\nThe soft spot is exactly the one you flagged: CodeGuru is the training reward and the test oracle. The human check in Appendix D is 20 CodeGemma conversations, labeled by one PhD student, with 90% agreement on overall vulnerable/clean. That is reassuring but thin, especially because the main metric counts any CodeGuru CWE flag as success, while the human review treats context-dependent cases as clean. If CodeGuru systematically false-positives on the kind of code RedCoder learns to emit, the absolute rates and the relative gains over baselines could be overstated. The OOD experiment helps with task overfitting, but it does not fix the oracle-circularity problem. I would also like to see repeated runs and error bars; the headline numbers are point estimates.\n\nNone of this sinks the paper. The contribution is real and the method is reproducible in principle (the pipeline is described well, though no code release is mentioned). The central claim is plausible and the evidence is decent, just not airtight. This is exactly the kind of paper that should go to peer review, where the authors can be pushed to expand the human validation and report variance. The audience is the AI-security and code-LLM evaluation community; they will get value from the method even if the exact numbers shift after better oracle validation. Send it out.","headline":"A useful, genuinely new red-teaming pipeline with strong reported gains, but the load-bearing evaluation oracle (CodeGuru) needs stronger independent validation before I'd trust the absolute rates.","tokens_in":17045,"tokens_out":2070,"would_cite":true,"duration_ms":23939,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RedCoder, an automated multi-turn red-teaming agent, induces vulnerable code in up to 65% of conversations with code LLMs.","keywords":["red teaming","code LLMs","multi-turn attacks","vulnerability induction","CWE","guardrails","retrieval-augmented generation","security evaluation"],"falsifier":"Have several independent security engineers manually review a few hundred conversations that CodeGuru flagged as vulnerable and an equal number it cleared, applying a concrete exploitability test (e.g., whether the code can be triggered with a realistic input to produce a security impact). If expert agreement with CodeGuru falls well below the 90% agreement reported in the paper's 20-conversation pilot, the central claim that RedCoder induces genuine vulnerabilities at the reported rates is not established.","tokens_in":16032,"feed_emoji":"⚔️","tokens_out":7442,"duration_ms":65077,"temperature":0.7,"pith_summary":"The paper claims that red-teaming of code LLMs should model the interactive, multi-turn nature of real AI-assisted programming, and that a fully automated agent can do this better than any previous single-turn or multi-turn attack. RedCoder is built by running a four-agent gaming loop that produces successful 'prototype conversations' and a library of reusable attack tactics; a code LLM fine-tuned on those conversations steers live conversations while retrieving tactics from the library. Across four victim models, RedCoder achieves vulnerability rates of 39.41% to 65.29%, roughly 2–10 times higher than the compared baselines, including attacks on a proprietary model. The authors also report that single-turn guardrails do not meaningfully reduce the attack, whereas a multi-turn guardrail trained on the same prototype conversations cuts the rates substantially. If these results hold, security evaluation of AI coding assistants can be automated at scale, and safety work needs to move to conversation-level defenses.","feed_headline":"Multi-turn red team agent breaks code LLMs up to 65% of the time","feed_subtitle":"Automated agent beats single- and multi-turn baselines, and shows single-turn guardrails barely stop it.","key_machinery":"The load-bearing object is the gaming loop: an attacker LLM (GPT-4o) carries up to 20 five-turn conversations against a defender made of a coding agent (Llama3-8B-Instruct) plus a guardrail, while Amazon CodeGuru labels each conversation for CWE vulnerabilities and a strategy analyst writes a short summary of the behavioral change that turned a failed attempt into a successful one. The successful conversations (2,098 in all) are decomposed into (history → next utterance) pairs for supervised fine-tuning of the RedCoder backbone. The strategy summaries form a key–value arsenal whose keys are embeddings of individual (query, response) turns; at each turn beyond the first, RedCoder embeds the previous interaction and injects the nearest strategy summary into the system prompt. That retrieval step is what lets the agent adapt its wording to the victim's responses rather than replaying a fixed attack.","core_discovery":"RedCoder's central claim is that the search over adversarial prompts for code LLMs can be automated and made multi-turn without human prompt engineering. The approach treats successful red-teaming as a learnable behavior: a multi-agent game generates conversations in which an attacker defeats a guardrail-equipped defender, the successful conversations are used as supervised fine-tuning data, and a strategy analyst distills what changed between failed and successful attempts into a retrievable tactic arsenal. At test time, RedCoder treats the victim's latest response as feedback and retrieves the most relevant tactic to shape its next utterance. The reported result is that this method induces at least one CWE-listed vulnerability in 39.41% (CodeLlama-7B), 61.18% (CodeGemma-7B), 65.29% (Qwen2.5-Coder-7B), and 40.00% (DeepSeek-R1-Distill-8B) of conversations, outperforming GCG, AutoDAN, CoA-Feedback, and ActorAttack by large margins, and that the learned tactics transfer to unseen CWE families and to Claude 3.5 Sonnet.","pith_inferences":["The method's success is measured through Amazon CodeGuru; a plausible inference is that RedCoder is partly optimized to trigger this detector, so its 'vulnerability rate' should be read as a CodeGuru-flagged rate until independent, exploit-level validation is done.","The strategy-arsenal design is domain-agnostic: swapping the evaluator (e.g., a SQL-injection or privacy-policy checker) could yield comparable multi-turn red-team agents for other structured generation tasks.","A direct implication for safety research is that guardrail training should use conversation-level labels at the exact turn where vulnerable code first appears; the paper's Appendix A labels history up to that turn as 'safe' and the turn itself as 'unsafe,' a recipe that could generalize to other multi-turn safety benchmarks.","Because the gaming process uses GPT-4o as attacker and Llama3-8B as defender, the prototype conversations inherit those models' blind spots; different victim and attacker models may yield different tactic arsenals."],"forward_implications":["Automated multi-turn red teaming of code LLMs is feasible without human prompt engineering, enabling large-scale security audits of coding assistants.","Single-turn guardrails miss multi-turn attacks: the paper finds they leave RedCoder's vulnerability rate essentially unchanged, so deployed defenses need conversation-level context.","Vulnerability-inducing behavior transfers across CWE families: on 20 held-out vulnerability types RedCoder still reaches 65.00% on Qwen2.5-Coder-7B, close to the 65.29% on the original 43-task set.","Proprietary, safety-tuned models remain susceptible: RedCoder induces vulnerable code in 42.85% of conversations with Claude 3.5 Sonnet.","Reasoning-oriented training does not by itself harden code LLMs, since DeepSeek-R1-Distill-8B exhibits a 40.00% vulnerability rate."],"supporting_citations":[{"why":"Baseline GCG, the gradient-based adversarial suffix attack whose reported vulnerability rates RedCoder must beat.","marker":"(Zou et al., 2023)"},{"why":"Baseline AutoDAN, the genetic-algorithm jailbreak, and also the inspiration for comparing failed vs. successful attacks in the strategy analyst.","marker":"(Liu et al., 2025)"},{"why":"Baseline CoA-Feedback, a semantic-driven multi-turn attacker that RedCoder is compared against.","marker":"(Yang et al., 2024b)"},{"why":"Baseline ActorAttack, a multi-turn semantic-network attack that RedCoder is compared against.","marker":"(Ren et al., 2024b)"},{"why":"Amazon CodeGuru, the production-grade detector used as the vulnerability oracle in both the gaming loop and the main evaluation.","marker":"(AWS, 2025)"},{"why":"ThinkGuard, the guardrail framework adapted to build the customized multi-turn defender in the gaming loop.","marker":"(Wen et al., 2025)"}],"fun_headline_variants":["Multi-turn agent beats code LLMs, hitting 65% vulnerability rate","Automated multi-turn red teaming defeats code LLMs at 65% rate","RedCoder's learnable tactics trick code LLMs into emitting vulnerable code","Multi-turn agent beats all baselines, hitting 65% on code LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire measured success of RedCoder rests on Amazon CodeGuru being an accurate oracle for whether generated code is genuinely vulnerable; if CodeGuru's flags are noisy or biased, RedCoder's reported vulnerability rates overstate real-world security risk.","fun_headline_variants_meta":{"raw":{"variants":["Multi-turn agent beats code LLMs, hitting 65% vulnerability rate","Automated multi-turn red teaming defeats code LLMs at 65% rate","RedCoder's learnable tactics trick code LLMs into emitting vulnerable code","Multi-turn agent beats all baselines, hitting 65% on code LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000665,"raw_usage":{"total_tokens":3068,"prompt_tokens":1013,"completion_tokens":2055,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":1971}},"tokens_in":629,"tokens_out":2055,"duration_ms":19900,"temperature":1.0,"reasoning_tokens":1971,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:55:04.623112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have several independent security engineers manually review a few hundred conversations that CodeGuru flagged as vulnerable and an equal number it cleared, applying a concrete exploitability test (e.g., whether the code can be triggered with a realistic input to produce a security impact). If expert agreement with CodeGuru falls well below the 90% agreement reported in the paper's 20-conversation pilot, the central claim that RedCoder induces genuine vulnerabilities at the reported rates is not established.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Amazon CodeGuru, the production-grade detector used as the vulnerability oracle in both the gaming loop and the main evaluation."}],"review_version":1}