{"id":"22e8ee6c-477e-480e-9070-29af977c3498","arxiv_id":"2504.15867","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A short string hidden inside a code comment can make LLM coding assistants generate attacker-chosen vulnerable code when they retrieve that comment from the web.","lead":"Attackers can post a correct-looking code solution on Q&A sites with a hidden gibberish 'attack sequence' in a comment, and LLM coding assistants that retrieve that solution will often copy it into code containing security flaws. The paper shows this works on four open models and on a retrieval-based coding assistant, with success rates above 75%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 84.29% ASR is optimization-run success, not held-out attack success; frozen-sequence per-input ASR in Table III is only 57.97%.","rationale":"The reader's weakest_assumption concerns organic retrieval, which is a real limitation: the real-world experiment hosted crafted answers on local pages, so StackOverflow ranking, moderation, and search behavior were bypassed. However, the more load-bearing issue for the quantified claim is the meaning of the 84.29% ASR. The attack generation loop in Algorithm 1 stops as soon as the target model produces tVul; Table II therefore counts optimization successes, not the probability that a fixed, pre-planted sequence succeeds in a fresh interaction. The paper's own Table III gives the needed held-out estimate: 57.97% per fresh assembled input, and only 37.85% of problems pass all five fresh inputs. This gap directly affects how readers should interpret the central claim, and it also casts light on the real-world number: that experiment uses the same models that generated the sequences, so 75.92% is best understood as transfer across prompt templates on known models, not independent deployment success. The concern is not that the phenomenon is absent; the examples and the transfer/quantization results still support the existence of a real threat. The issue is scientific reporting of the headline number. A frozen-sequence evaluation with a benign control would resolve whether the headline should be 84%, roughly 58%, or somewhere in between.","tokens_in":20545,"tokens_out":9886,"duration_ms":99425,"concrete_test":"Freeze the final attack sequence for each of the 35 problems after Algorithm 1 returns, then run a fresh evaluation: for each of the four LLMs, sample 10 assembled inputs not used in generation (new PT/IN/Q), generate one response per input with no further gradient updates, and compute per-input ASR = fraction of responses containing tVul. Compare this with Table II (84.29%), Table III (57.97%), and a benign-reference control (same correct code without Seq). If frozen-sequence per-input ASR is close to 57.97% rather than 84.29%, the headline ASR reflects optimization success, not deployed attack success; if it exceeds 80%, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline 84.29% ASR (Table II) is not an independent measure of how often a fixed planted answer makes a target LLM emit tVul. In Algorithm 1 the sequence is iteratively mutated until the target model responds with tVul; the loop exits on success (lines 6-7, 15-16), so Table II records the fraction of optimization runs that terminated successfully, not the success rate of a finalized Seq on fresh inputs. The paper's own held-out transfer evaluation (Table III) gives a per-fresh-input average ASR of 57.97% and only 37.85% of problems passing all five fresh inputs. The real-world 75.92% (Table V) was obtained with attack sequences generated for the same models that power the tested ChatChat assistant (Mistral-7b, CodeLlama-7b), so it likewise measures transfer to the application's prompt template, not success against an unseen model. Thus the central claim's headline number overstates the deployed success rate: for a fixed attack sequence, the relevant probability of inducing vulnerable code is closer to the held-out transfer numbers. A no-attack control is also absent, so the causal contribution of the sequence is not separately quantified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HACKODE, an attack against LLM coding assistants that incorporate external references. The attacker posts a correct-looking code solution containing a short, optimized \"attack sequence\" hidden in a comment; when a retrieval-augmented assistant feeds this reference into an LLM, the sequence steers the LLM to generate code containing a predetermined vulnerability. The attack sequences are produced by a two-phase gradient-guided token-mutation algorithm (Algorithm 1) over a set of assembled inputs built from varied prompt templates, instructions, and user queries. Evaluation on 35 StackOverflow-derived problems across four 7B-15B open models reports an average 84.29% ASR (Table II), 57.97% average per-input transfer to fresh assembled inputs (Table VI), 50.76% on quantized models (Table IV), and 75.92% ASR on a ChatChat-based assistant with locally hosted references (Table V). The authors conclude that HACKODE poses a realistic real-world threat.","tokens_in":20788,"tokens_out":6259,"duration_ms":55833,"significance":"The attack surface is real and increasingly relevant: retrieval-augmented coding assistants are deployed, and poisoning external references with imperceptible content is a plausible supply-chain risk. The paper's core mechanism--optimizing a short token sequence in a comment to condition the model toward a target vulnerability--is interesting, and the ablations (progressive generation, insertion position) are useful. The strongest parts are the transfer evaluation on held-out assembled inputs and quantized models, and the release of source code. However, the significance of the headline numbers is currently overstated: the 84.29% figure is an optimization-run success rate rather than a deployment-time success rate, and the causal contribution of the sequence is not isolated from baseline LLM error. With a proper control condition and honest reporting of the held-out transfer ASR, the contribution would be solid empirical evidence for a new class of attack.","major_comments":[{"comment":"The headline 84.29% ASR in Table II is an optimization-run success rate, not a held-out success rate. In Algorithm 1, a run terminates when tVul appears in the output on the fixed input used during optimization (lines 6-7 for the preliminary phase and lines 15-16 for the enhancement phase), so Table II records the fraction of runs that found a sequence, not the probability that a fixed sequence succeeds on a fresh input. The paper's own held-out transfer evaluation gives a per-input average ASR of 57.97% (Table VI) and only 37.85% of problems pass all five fresh inputs (Table III). The abstract and conclusion should be reframed around the transfer success rate, and Table II should be labeled accordingly.","section":"§V-A and Algorithm 1 (Tables II, III, VI)"},{"comment":"There is no no-attack control. The experiments never measure the frequency with which the target LLMs generate the target vulnerability from the correct reference without the attack sequence. Since the target vulnerability is a small modification of the correct code (e.g., changing `histogramBinCount` to `histogramBinCount + 1` in Figure 5), part of the reported ASR could be ordinary LLM error on the referenced code rather than an effect of the sequence. A control condition (reference only, or reference with a benign comment of similar length) is required to attribute the observed vulnerable generation to the attack sequence. This concern applies to Tables II, III, and V.","section":"§V-A, §V-C (evaluation design)"},{"comment":"The real-world experiment does not exercise organic retrieval and ranking. The assistant was forced to retrieve from local web pages containing the crafted answers, and the tested models (Mistral-7b and CodeLlama-7b) are the same models on which the attack sequences were generated. The 75.92% ASR therefore measures transfer to one application's prompt template under forced retrieval, not success against real search ranking or against unseen models. The Discussion argues that such crafted answers would likely be prioritized, but this is not tested; the abstract's \"real-world impact\" claim is stronger than the experiment supports. The authors should present this as a controlled simulation and temper the corresponding claim.","section":"§V-C and §VI (real-world experiment)"},{"comment":"The vulnerability-matching procedure is underspecified. In the implementation description, the success check is only described as \"comparing the differences between the code lines in the response and the correct code,\" and §V-C states that responses were \"checked if the responses ... contained vulnerabilities\" without specifying the exact matching rule, the handling of semantically equivalent but textually different code, or the criteria for distinguishing target vulnerabilities from similar benign patterns. Since every reported ASR depends on this check, the paper should provide the precise detection rule or release the verification script as part of the artifact.","section":"§V-A and Algorithm 1 (vulnerability detection)"}],"minor_comments":[{"comment":"The answer to RQ2 states an \"average ASR of 57.97%\" but Table III presents cumulative percentages of problems passing at least N tests; the relationship between the table and the reported average should be stated explicitly, or the per-input average should be shown directly in the table.","section":"§V-B (Table III)"},{"comment":"The notation `pSeq m := (pSeq[i]← pSeq[i] + grad[i])m` is confusing; the authors should clarify that m variants are generated and how the index i is selected.","section":"Algorithm 1, line 9"},{"comment":"The research question uses \"quantified LLMs\" where \"quantized LLMs\" is intended; likewise, the limitation section uses \"quantified\" in one place. These typos should be corrected.","section":"§V-B (RQ2 wording)"},{"comment":"The description of how generated queries are verified on StackOverflow does not specify which LLM is used for rephrasing or the verification criteria, which limits reproducibility; please provide the prompts and filtering rules.","section":"§IV-A (User Query derivation)"},{"comment":"The statement that GPT-4 can solve only 19 of the 50 most recently answered StackOverflow problems should include the evaluation date, model version, and sampling methodology, since this claim is used to motivate the need for external reference retrieval.","section":"§I (Motivating measurement)"}],"recommendation":"major_revision","confidential_remarks":"The contribution is potentially publishable in a security or software-engineering venue, but the current framing overstates the empirical results. The two most important fixes are (1) reporting the held-out transfer ASR as the primary effectiveness measure and (2) adding a no-attack control. I would also ask for a precise description of the vulnerability-checking procedure before accepting any revision. The small dataset (35 problems) and lack of error bars further weaken the per-type claims, though these could be addressed with confidence intervals or additional data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one if you care about the security of retrieval-augmented code generation. The paper shows that a short token sequence hidden in a code comment of an otherwise correct reference can make several open LLMs emit a chosen vulnerability, and it transfers surprisingly well across prompt templates, instructions, and query phrasings. That is a real result and worth a serious referee.\n\nWhat's actually new: the attack surface is the external reference itself, and the two-phase mutation (fixed input then random assembled inputs) is a sensible way to build transferability. The code is public, and the ablation against HACKODE- shows the enhancement step adds roughly 25 points. The quantized-model transfer is also a nice practical check.\n\nThe soft spots are real but not fatal. The stress-test note is right: the headline 84.29% ASR is not what it looks like. Algorithm 1 mutates until the target model emits tVul and returns on success, so Table II is the fraction of optimization runs that terminated, not the success rate of a finalized sequence on fresh inputs. The paper's own held-out numbers are much more modest: only 37.85% of problems pass all five held-out assembled inputs, and the per-input transfer sits near 58%. The real-world 75.92% was generated with the same models that power the ChatChat assistant, so it measures template transfer, not unseen-model transfer. There is also no no-attack control, so part of the effect could be ordinary LLM error on correct references. And the \"real-world\" test forces retrieval from local pages; organic search ranking is only argued, not tested.\n\nNone of this sinks the paper. The attack demonstrably works on the lab setup, the transfer results are decent, and the threat model is plausible. But the abstract and conclusions sell the optimization-run number as the operational success rate, and that is misleading. A revision should report held-out ASR as the headline, add the control, and either test organic retrieval or soften the real-world claim.\n\nWho's it for: security researchers working on LLM code generation and poisoning attacks, and folks building RAG-based coding assistants. It's a solid conditional accept, not a desk reject. I'd engage with it, and I'd expect the authors to fix the reporting before publication.","headline":"The headline 84.29% ASR is an optimization-run success rate, not a held-out attack success rate; the paper still demonstrates a real and transferable poisoning attack on retrieval-augmented code generation, but it needs honest reporting before publication.","tokens_in":21280,"tokens_out":3614,"would_cite":true,"duration_ms":30606,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Attackers can plant a short, meaningless sequence in an otherwise correct code answer and cause LLM coding assistants to generate attacker-chosen vulnerable code, with reported success rates over 80%.","keywords":["LLM security","code generation","retrieval-augmented coding assistants","attack sequence","prompt injection","StackOverflow poisoning","transferable adversarial attacks","vulnerable code"],"falsifier":"Conduct a field test in which crafted but harmless answers are posted to a live Q&A forum and a retrieval-augmented coding assistant searches the open web without local-page forcing; if such pages are rarely retrieved or ranked below legitimate answers, the reported 75.92% real-world attack success rate would not transfer to actual deployments. A second check: measure whether removing the comment-inserted attack sequence while keeping the code identical eliminates the vulnerable output, confirming that the sequence itself is the causal agent.","tokens_in":20381,"feed_emoji":"🐛","tokens_out":5912,"duration_ms":48110,"temperature":0.7,"pith_summary":"This paper claims that an attacker who posts a correct-looking answer to a programming question can embed a short meaningless string—an attack sequence—that, when retrieved by an LLM coding assistant, steers the model to emit code containing a chosen vulnerability such as a buffer overflow or missing validation. The claim matters because popular coding assistants routinely fetch external solutions from the web, so a poisoned answer could reach developers without any compromise of the model itself. The reported prototype, HACKODE, achieves an average 84.29% attack success rate across four open LLMs and 75.92% on a retrieval-based assistant in a local-web experiment, with attack sequences averaging about 33 tokens.","feed_headline":"Short planted comments flip AI coding assistants to vulnerable code","feed_subtitle":"An 84% attack success rate means attacker-posted answers can poison retrieval-based coding tools.","key_machinery":"The load-bearing object is the attack sequence (Seq), a short string of tokens generated by a two-phase gradient-guided mutation procedure and inserted into the referenced code as a comment. In the first phase, HACKODE optimizes a preliminary sequence against a fixed assembled input—query, instruction, prompt template, and reference—by computing cross-entropy loss between the model's logits and the target vulnerable code and replacing tokens with gradient-scored alternatives. In the second phase, it mutates the sequence further against randomly assembled inputs so the sequence generalizes across variations the attacker cannot control. The procedure treats the target vulnerability as a token-level goal and the attack sequence as the only manipulable variable.","core_discovery":"The central discovery is that LLMs generating code from external references can be induced to produce vulnerable code even when the referenced example is functionally correct and human-readable. The vulnerability appears because the attack sequence, inserted as an innocuous code comment near the code to be altered, biases the model's token predictions; the paper measures this by checking whether the target vulnerable code appears in responses. Across 35 StackOverflow-derived problems and five vulnerability types, the attack succeeded on average in 84.29% of cases over two general LLMs (Llama2-7b, Mistral-7b) and two code LLMs (CodeLlama-7b, StarChat2-15b), and it transferred to randomly assembled prompts and to GPTQ and BitsAndBytes quantized models with lower but substantial success rates.","pith_inferences":["Extension: if search engines and Q&A forums rank human-plausible answers highly, this attack could combine with SEO to poison coding assistants at scale, turning every retrieved snippet into a potential exploit delivery vehicle.","Extension: a concrete defense to test is post-generation static analysis of LLM output; the paper notes high false-positive rates, so a calibrated experiment comparing detection of naturally vulnerable versus HACKODE-induced code would tell whether such tools can blunt the threat.","Extension: the mechanism suggests a broader class of reference poisoning beyond code—documentation pages, API examples, and config snippets could be tested for the same comment-insertion effect.","Extension: larger or safety-tuned closed models may resist better; a transfer test against a frontier model's retrieval pipeline, without access to model weights, would clarify how far the success rates generalize."],"forward_implications":["Retrieval-based coding assistants can be attacked without access to model weights: attackers only need to publish a correct-looking answer containing the attack sequence.","Because attack sequences average 33 tokens, roughly 3.4% of the assembled input length, the manipulation is hard for a human reader to spot in a code answer.","The generated attack sequences transfer to unseen prompt templates, instructions, and user queries, and to GPTQ and BitsAndBytes quantized models, with average success rates of 48.07% and 53.45%.","Comment insertion is markedly more effective than variable renaming as an embedding position, raising average attack success rate from 11.43% to 80.00% on Mistral-7b.","Five vulnerability classes are inducible (CWE-125, CWE-787/CWE-120/CWE-122, CWE-457/CWE-190, CWE-20/CWE-570, CWE-835), so the threat is not limited to a single flaw type."],"supporting_citations":[{"why":"Supplies the retrieval-based coding assistant (ChatChat) used in the real-world evaluation.","marker":"[4]"},{"why":"Documents a prior AI poisoning incident, supporting the claim that attacker-posted external content can be referenced by LLM applications.","marker":"[6]"},{"why":"Mistral-7b is one of the four open models on which the 84.29% average attack success rate is measured.","marker":"[15]"},{"why":"Provides the StackExchange API/library used to build the 35-problem dataset from StackOverflow.","marker":"[17]"},{"why":"StarChat2-15b is one of the code LLMs evaluated for attack effectiveness and transferability.","marker":"[22]"},{"why":"Earlier finding that Copilot generates vulnerable code, motivating the security risk studied here.","marker":"[32]"},{"why":"CodeLlama-7b is one of the code LLMs evaluated across the five vulnerability types.","marker":"[35]"},{"why":"Llama2-7b is one of the general LLMs evaluated, including quantized-model transfer tests.","marker":"[46]"}],"fun_headline_variants":["One comment is all it takes to make LLMs write vulnerable code","Comment attack: 84% of the time, LLMs produce vulnerable code","Plant a comment, break a code assistant: 84% success rate","Hidden in plain sight: comments weaponize coding AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The real-world impact claim rests on the assumption that attacker-posted, correct-looking answers will actually be retrieved and ranked highly by coding assistants during organic web searches; the real-world experiment forced retrieval by hosting crafted pages locally rather than demonstrating organic ranking.","fun_headline_variants_meta":{"raw":{"variants":["One comment is all it takes to make LLMs write vulnerable code","Comment attack: 84% of the time, LLMs produce vulnerable code","Plant a comment, break a code assistant: 84% success rate","Hidden in plain sight: comments weaponize coding AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001419,"raw_usage":{"total_tokens":5675,"prompt_tokens":837,"completion_tokens":4838,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":4763}},"tokens_in":453,"tokens_out":4838,"duration_ms":29142,"temperature":1.0,"reasoning_tokens":4763,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:15:29.171168+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Conduct a field test in which crafted but harmless answers are posted to a live Q&A forum and a retrieval-augmented coding assistant searches the open web without local-page forcing; if such pages are rarely retrieved or ranked below legitimate answers, the reported 75.92% real-world attack success rate would not transfer to actual deployments. A second check: measure whether removing the comment-inserted attack sequence while keeping the code identical eliminates the vulnerable output, confirming that the sequence itself is the causal agent.","supporting_citations":[{"cited_title":"Chatchat","cited_arxiv_id":null,"evidence_quote":"Supplies the retrieval-based coding assistant (ChatChat) used in the real-world evaluation."},{"cited_title":"User solana wallet exploited in first case of ai poisoning attack, 2024","cited_arxiv_id":null,"evidence_quote":"Documents a prior AI poisoning incident, supporting the claim that attacker-posted external content can be referenced by LLM applications."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Mistral-7b is one of the four open models on which the 84.29% average attack success rate is measured."},{"cited_title":"Stackexchange, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the StackExchange API/library used to build the 35-problem dataset from StackOverflow."},{"cited_title":"Starcoder 2 and the stack v2: The next generation, 2024","cited_arxiv_id":null,"evidence_quote":"StarChat2-15b is one of the code LLMs evaluated for attack effectiveness and transferability."},{"cited_title":"Asleep at the keyboard? assessing the security of github copilot’s code contributions","cited_arxiv_id":null,"evidence_quote":"Earlier finding that Copilot generates vulnerable code, motivating the security risk studied here."},{"cited_title":"Llama: Open and efficient foundation language models, 2023","cited_arxiv_id":null,"evidence_quote":"Llama2-7b is one of the general LLMs evaluated, including quantized-model transfer tests."}],"review_version":1}