{"id":"88d063c9-1a1f-4a3c-a67f-f7b273cc9e9d","arxiv_id":"2507.16773","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Hidden strings in code exploit a reasoning model's tendency to copy tokens into its own thinking, enabling output length and result manipulation.","lead":"Attackers can hide trigger strings inside code that cause step-by-step reasoning AI models to loop forever, stop early, or wrongly judge whether the code is safe. This security paper maps a new attack surface for AI code-analysis assistants and shows the trick works in limited settings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RO(I) results validate steering only when the trigger is already in context; the paper never measures whether optimized triggers are spontaneously copied from a payload-only prompt, so the central attack premise is unverified.","rationale":"The reader's weakest assumption identifies exactly the load-bearing gap: the attack materializes only if the model spontaneously copies the optimized trigger into its reasoning prefix. The paper conditions its objectives on the trigger being present after a prefix p (RO(I)) or in both prompt and output (RO(II)), but never measures the natural copy rate from a payload-only prompt. My independent reading of Section IV-A and Tables II-III confirms that Table II's successes are optimization successes under a relaxed objective, not deployment successes under the threat model. Figure 1 provides a single illustrative case with a hand-chosen trigger, not systematic evidence for optimized triggers. The paper's own Open Questions section acknowledges scalability and transferability limits but does not flag this missing premise, making the abstract's 'reliably induces' overclaim. I therefore agree with the reader's conditional verdict: the contribution is plausible and worth publishing if the copy-rate measurement and a payload-only attack evaluation are added. If the proposed test fails, the central claim would need to be downgraded to 'context-injected trigger biasing' rather than copy-guided attack. Because the reader has already conditioned on this gap, no verdict change is needed.","tokens_in":8339,"tokens_out":4523,"duration_ms":51463,"concrete_test":"Run each optimized trigger from Table II in the actual attack configuration (payload-only input i ⊕ bt ⊕ t ⊕ at, with no p/t appended to the prompt) under the same greedy decoding used in the paper, for at least 100 runs per AM on DeepSeek-R1-Distill-Qwen-1.5B. Record (a) the fraction of runs in which the optimized trigger or a substring of length at least half the trigger length appears in the generated rationale before any target behavior, and (b) the AM success rate. Also run the same with random triggers as baseline. If the copy rate is not substantially above baseline and the AM success rate drops to near zero, the RO(I) results do not support the claimed copy-guided attack.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that an attacker controlling only the external code can induce copying of embedded triggers and thereby steer the reasoning trace. The optimization in Section IV-A does not enforce this premise. RO(I) is defined as LI(t) = -log P(y | i ⊕ bt ⊕ t* ⊕ at ⊕ p ⊕ t): at optimization time the trigger t is already present in the conditioning context immediately before the target, so the model is never required to copy it out of the payload. Table II therefore demonstrates that a context-injected trigger can bias generation, not that a payload-only trigger will be copied. RO(II) optimizes the trigger in both prompt and output positions, but even there the output occurrence is placed by the optimizer during training; deployment requires the model to emit that same trigger spontaneously. The paper reports no measurement of the rate at which payload-only optimized triggers appear in the rationale under greedy decoding, and Figure 1 is an anecdotal hand-picked case rather than a systematic count. Unless that copy rate is high, RO(I) results are an artifact of explicit context injection and the threat model in Section III-A is not realized. This is the load-bearing gap behind the abstract's claim that CGA 'reliably induces' the listed failures.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Copy-Guided Attacks (CGA), a class of indirect prompt injection attacks that exploit the tendency of reasoning LLMs to copy tokens from the input into their rationale. The authors define five attack manifestations (repetition, premature end-of-sequence, false refusal, premature end-of-thought, and semantic distortion), formalize trigger construction as an optimization problem with four relaxations (RO(I) through RO(IV)), and adapt Greedy Coordinate Gradient (GCG) to synthesize triggers. Experiments on DeepSeek-R1-Distill-Qwen-1.5B with a single 35-line Solidity contract show that RO(I) can be optimized for all five manifestations, RO(II) succeeds only for three, and RO(III)/(IV) are estimated to be computationally infeasible. The paper claims this demonstrates a new vulnerability in LLM-powered code analysis, though the authors also acknowledge that generalization remains an open problem.","tokens_in":8591,"tokens_out":6225,"duration_ms":61416,"significance":"If the central premise were verified—that a trigger embedded in external code is spontaneously copied into the reasoning trace and then steers the model—the attack would be significant: it would enable denial-of-service, false refusals, and adversary-chosen vulnerability judgments in code-analysis pipelines. The taxonomy of five manifestations and the formal optimization framework are useful conceptual contributions, and the paper is commendably transparent about the RO(II) failures and the prohibitive estimated cost of RO(III)/(IV). The release of code (if made available) and the reproducible optimization setup are additional strengths. However, the current experiments support only a conditional statement—'when a trigger is already present in the decoding context, it can steer the output'—which is materially weaker than the paper's stated claim that embedded triggers are copied during normal decoding and thereby manipulate inference. The missing copy-rate measurement is the decisive gap.","major_comments":[{"comment":"The central claim of the paper is that an attacker who controls only the external code can induce the model to copy an embedded trigger and thereby steer the reasoning trace. This premise is not tested. The RO(I) objective is defined as LI(t) = -log P(y | i ⊕ bt ⊕ t* ⊕ at ⊕ p ⊕ t), so the optimized trigger t is already present in the conditioning context immediately before the target y; the model never has to copy it out of the payload during optimization or evaluation. Table II therefore demonstrates that an explicitly injected trigger can bias generation, not that a payload-only trigger will be spontaneously copied. No systematic copy-rate measurement for payload-only prompts is reported, and Figure 1 is a single hand-picked case. Without such a measurement, the threat model in §III-A is not realized, and the abstract's claim that CGA 'reliably induces' the listed failures is unsupported. The authors should add a copy-rate measurement (the fraction of runs in which the optimized trigger appears in the rationale under greedy decoding for prompts of the form i ⊕ bt ⊕ t ⊕ at) and report it per AM, or explicitly restrict the claims to the conditional setting.","section":"§IV-A, RO(I) and §V-B, Table II"},{"comment":"The empirical evaluation is example-based rather than statistical. Only one model (DeepSeek-R1-Distill-Qwen-1.5B) and one 35-line Solidity contract are used, and each row of Table II shows a single selected output with no number of trials, success rate, variance, or error bars. The abstract's phrase 'reliably induces' is therefore not supported by the reported evidence. Please report success rates over multiple seeds and, ideally, over additional instructions, contracts, and model checkpoints; at a minimum, temper the abstract and Section I to say 'can induce' in the tested examples, with the explicit caveat that generalization was not measured.","section":"§V-A, §V-B, Table II"},{"comment":"The relaxed objectives progressively build the copy event into the optimization. For RO(II), Multi-Pos GCG optimizes the same trigger tokens in both the prompt and the output positions, so the output occurrence is placed by the optimizer during training; the experiments do not measure whether the model would emit that trigger spontaneously from a payload-only prompt. RO(II) also fails for AM-2 and AM-4, and RO(III)/(IV) are estimated at 80 and 8,000 days and were not run. Consequently, the statement in Section VI that 'multi-prompt CGA is an observable phenomenon' rests on the single anecdotal example in Figure 1 rather than on a measured copy probability or a multi-prompt success rate. This should be either supported with data or removed.","section":"§IV-A, RO(II) and §V-B, Table III"}],"minor_comments":[{"comment":"The notation is inconsistent: RO(I) uses t* in the prompt and t as the optimized suffix, while Figure 2 labels both occurrences as t. Please define t* and clarify that the optimized variable in RO(I) is the suffix after p, not the payload trigger.","section":"§IV-A"},{"comment":"The caption should state the generation protocol: whether the trigger was appended after p (the RO(I) setup) and whether the displayed outputs were selected from multiple runs. Currently the reader cannot tell if these are successful examples or representative outcomes.","section":"Table II"},{"comment":"The figure is difficult to read, and the notation 'LOOPk' and the relation between the three task variants are not explained. Please enlarge and clarify.","section":"Figure 1"},{"comment":"The sentence 'We release our code and initial results' should include a repository link or be removed.","section":"Section I"}],"recommendation":"major_revision","confidential_remarks":"The paper is currently a preliminary feasibility study, and the missing copy-rate measurement is the decisive issue. In my view, the central claim can be defended only if the authors either verify the copy premise with systematic data or substantially scope down the claims to the conditional setting. I would be willing to see a revised version; the paper may also be well suited to a security workshop or a short-paper track in its present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the framing: naming copy-guided attacks as a class, and splitting them into inference-length and inference-result manipulations. That taxonomy is useful and goes beyond the usual prompt-injection literature. The formalization of the attack objective and the extension of GCG to multi-position optimization are also real, if incremental, technical steps. Credit where due: the paper does not hide its failures. RO(II) failing for AM-2 and AM-4, and RO(III)/(IV) being estimated at 80 and 8,000 days, are reported plainly. That honesty makes the work read like an early exploration, not a polished claim.\n\nThe soft spot is exactly where the stress-test lands. The central threat model says an attacker controls only the external code payload, and the attack works when the model spontaneously copies the trigger into its reasoning. But RO(I) conditions on the trigger already being present before the target in the conditioning context. That never tests the copying premise. RO(II) places the trigger in the output during optimization, but deployment requires the model to emit that trigger on its own. The paper never measures how often optimized triggers are naturally copied from a payload-only prompt under greedy decoding. Figure 1 is a hand-picked anecdote, not a rate. So the abstract's 'reliably induces' is not supported. Table II shows single examples, not success rates or error bars. The generalization claim in the introduction—that the attack works 'regardless of the user instruction'—is based on three instructions on one model, and the experiments only use one 35-line Solidity contract and a 1.5B model. The missing code/data links also matter; the paper says it will release artifacts but provides none.\n\nThese are significant gaps, but not fatal to the idea. The copying behavior of reasoning models is real and plausibly exploitable; the paper just does not demonstrate the exploitation. It should be treated as a positioned hypothesis with a preliminary proof-of-concept, not as a measured attack.\n\nRecommendation: send it to peer review, but require major revision: measure the natural copy rate, report success rates with multiple seeds, clarify the RO(I) setup so it matches the threat model, tone down the abstract, and release the code and data. The taxonomy alone is worth referee attention.","headline":"A credible new attack taxonomy built on an unverified copying premise; the paper is honest about its limits but the abstract oversells the evidence.","tokens_in":9117,"tokens_out":1474,"would_cite":false,"duration_ms":18288,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reasoning LLMs that copy prompt tokens into their chain of thought can be steered by hidden triggers toward infinite loops, early termination, false refusals, and adversary-chosen vulnerability verdicts.","keywords":["copy-guided attacks","reasoning LLMs","chain-of-thought","prompt injection","vulnerability detection","Greedy Coordinate Gradient","inference manipulation","LLM security"],"falsifier":"Run the Table II triggers through the same model with ordinary user instructions on the same Solidity contract and count how often the trigger appears in the reasoning prefix before the claimed behavior. If the copy rate is near zero, or if the target behavior occurs at the same rate with a random trigger in place of the optimized one, the copy-guided mechanism is not what produces the effect.","tokens_in":8148,"feed_emoji":"🧠","tokens_out":6394,"duration_ms":65432,"temperature":0.7,"pith_summary":"This paper argues that a behavior normally seen as harmless, a reasoning LLM copying tokens from the prompt into its chain-of-thought, is itself an attack surface. If an adversary hides a crafted trigger inside an external code snippet, the model will likely reproduce that trigger during reasoning, and the copied tokens then act as anchors that bias all subsequent generation. The authors formalize trigger construction as an optimization problem, adapt the Greedy Coordinate Gradient method to solve relaxed versions of it, and show on DeepSeek-R1-Distill-Qwen-1.5B that the attack can produce infinite repetition, premature end-of-sequence, false refusal, premature end-of-thought, and flipped vulnerability judgments. The reported success is confined to a targeted, single-instruction setting; generalizing across prompts is left as an open research problem because the multi-prompt objectives are estimated to require 80 to more than 8,000 days of computation with current methods. This matters because LLM-based code auditors are entering development pipelines, where a malicious code snippet could turn the auditor into a denial-of-service source or a tool that returns adversary-chosen verdicts.","feed_headline":"Copied triggers let attackers steer reasoning LLMs","feed_subtitle":"A hidden string in code becomes an anchor that makes the model loop, refuse, or flip verdicts.","key_machinery":"The load-bearing mechanism is the copied trigger as an autoregressive anchor: an adversarial string $t$ placed in the payload is selected so that, once the model echoes $t$ in its reasoning prefix $p$, the conditional probability of the attacker's target sequence $y$ rises. To find such triggers the paper defines a ladder of relaxed objectives: the original loss sums over all instructions $i$ and decoding prefixes $p$; RO(IV) restricts those sets; RO(III) fixes one instruction; RO(II) fixes one prefix and forces the same trigger into prompt and output; RO(I) drops the dependence of decoding on the prompt and trigger. The optimization machinery is Greedy Coordinate Gradient (GCG), extended to Multi-Position GCG to co-optimize both occurrences of the trigger, and combined with Universal Prompt Optimization for the multi-prompt objectives.","core_discovery":"The paper's central claim is that a reasoning LLM's tendency to copy prompt tokens into its rationale is a distinct and exploitable vulnerability, separate from instruction hijacking. The attacker decomposes the adversarial payload into before-trigger, trigger, and after-trigger parts; when the model copies the trigger into its reasoning prefix, the recent-token bias of autoregressive decoding makes that trigger steer the next tokens. The authors demonstrate five manifestations: token repetition until the maximum length is reached, early emission of the end-of-sequence token, false triggering of the safety refusal mechanism, early termination of the think block, and semantic distortion that flips vulnerability judgments. Their optimization results are mixed: all five targets are reachable under the simplest relaxed objective RO(I), three remain feasible when the same trigger must appear in both prompt and output, and none are feasible with current methods once multiple instructions or prefixes are required.","pith_inferences":["A cheap, testable defense follows from the paper's own mechanism: monitor the reasoning trace for verbatim echoes of payload tokens and flag them as potential anchors; the paper does not propose such a detector, but its mechanism implies it should work.","The failure of the end-of-sequence and end-of-thought targets under RO(II) suggests those two manifestations may depend less on copy-anchoring and more on direct pressure on special-token probabilities; comparing the probability of <eos> and </think> with and without a copied trigger would separate the two paths.","If the copying tendency is model-wide rather than task-specific, CGA should transfer to other code tasks that make the model echo identifiers, such as summarization or repair; the paper's three-instruction denial-of-service case study points in that direction but does not establish the boundary."],"forward_implications":["A user who asks an LLM to audit or summarize untrusted code can be forced into a repetition loop that consumes the model's maximum token budget, even though the payload contains no hidden instruction.","The same trigger can make the auditor stop before answering, refuse harmless requests, or terminate its reasoning early, degrading accuracy on code that needs multiple inference steps.","Semantic distortion lets the attacker choose the vulnerability verdict: safe code is reported as vulnerable, or flawed code as safe, creating false positives and false negatives in the pipeline.","Because the payload contains no readable instruction, CGA is stealthier than ordinary prompt injection: the user sees only code, and the model appears to be malfunctioning rather than obeying a hidden command.","Under current methods the attack is practical only for a fixed prompt; extending it across instructions or prefixes is computationally out of reach, so multi-prompt CGA remains an open problem."],"supporting_citations":[{"why":"Supplies a mechanistic account of step-by-step reasoning, supporting the claim that reasoning traces copy content from the context.","marker":"[4]"},{"why":"Supports the premise that reasoning LLMs exhibit copying and repetition behaviors inside their rationales.","marker":"[5]"},{"why":"Frames indirect prompt injection and defines the threat model of malicious content hidden in external data.","marker":"[7]"},{"why":"Grounds the key assumption that recent tokens dominate next-token prediction in transformer models.","marker":"[15]"},{"why":"Provides the contrast of direct decoding manipulation, motivating why an attacker limited to the prompt must exploit copying instead.","marker":"[18]"},{"why":"Supplies the Greedy Coordinate Gradient method and the claim that triggers optimized on a limited prompt set can generalize to other prompts.","marker":"[19]"}],"fun_headline_variants":["Copied tokens hijack reasoning LLMs","Copy-and-steer: new attack on reasoning LLMs","LLMs tricked into loops by copied triggers","Attack forces LLMs to copy and misreason","Copy bias lets attackers manipulate LLM output"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack only exists if the model spontaneously copies the optimized trigger into its own reasoning during ordinary decoding, and the paper never measures how often that copy actually happens.","fun_headline_variants_meta":{"raw":{"variants":["Copied tokens hijack reasoning LLMs","Copy-and-steer: new attack on reasoning LLMs","LLMs tricked into loops by copied triggers","Attack forces LLMs to copy and misreason","Copy bias lets attackers manipulate LLM output"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1329,"prompt_tokens":941,"completion_tokens":388,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":317}},"tokens_in":557,"tokens_out":388,"duration_ms":4617,"temperature":1.0,"reasoning_tokens":317,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:02:43.159102+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Table II triggers through the same model with ordinary user instructions on the same Solidity contract and count how often the trigger appears in the reasoning prefix before the claimed behavior. If the copy rate is near zero, or if the target behavior occurs at the same rate with a random trigger in place of the optimized one, the copy-guided mechanism is not what produces the effect.","supporting_citations":[{"cited_title":"Not what you’ve signed up for: Com- promising real-world llm-integrated applications with indirect prompt injection","cited_arxiv_id":null,"evidence_quote":"Frames indirect prompt injection and defines the threat model of malicious content hidden in external data."},{"cited_title":"On large language models’ resilience to coercive interro- gation","cited_arxiv_id":null,"evidence_quote":"Provides the contrast of direct decoding manipulation, motivating why an attacker limited to the prompt must exploit copying instead."}],"review_version":1}