{"id":"85abba2a-2e35-4b62-9be7-d30dc80f7766","arxiv_id":"2607.05587","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Semantic conflicts between code and cues cut execution-grounded correctness; residual-stream patching localizes the causal signal to the edit site, sparse carriers, and readout.","lead":"LLMs often follow misleading comments or names over actual code when the two conflict, and activation patching shows the conflict signal lives in a few residual-stream sites. This matters because AI coding tools can silently encode the wrong intent into tests, repairs, and agent actions.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"Stimulus selection that requires a cue-consistent error for at least one LLM risks circularly inflating both the behavioral effect and the staged recovery pattern.","rationale":"The reader's weakest_assumption correctly identifies the load-bearing point: the 45 triplets are not a random or naturalistic sample of semantic conflicts but a contrast-engineered set filtered so that at least one model already exhibits the desired failure mode. That filter is methodologically convenient for patching (it guarantees a non-zero source-destination margin) yet directly underwrites both the headline behavioral claim (large, directed cue-consistent errors) and the mechanistic claim (staged residual recovery). Within the filtered set the experiments are carefully executed—paired tests, bidirectional patching, margin-gap robustness, replication package—so the within-scope results stand. The concern does not warrant REJECT; it keeps the verdict CONDITIONAL pending an unfiltered stimulus check that would show whether the same effects and staging appear when the selection rule is removed. No stronger internal inconsistency was found.","tokens_in":18699,"tokens_out":517,"duration_ms":4833,"concrete_test":"Rebuild the behavioral tables and recovery heatmaps on a held-out set of 20+ token-aligned triplets that satisfy only syntactic/token constraints and produce a true execution-vs-cue contrast, without any pre-filter requiring cue-consistent error on any of the four models. If the average correctness drop falls below ~15 pp or the staged recovery (changed early, sparse carriers mid, readout late) disappears for a majority of pairs, the load-bearing selection assumption fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim rests on 45 hand-curated, token-aligned triplets that, by construction (Section IV-A and VII-B), must already produce a cue-consistent error on the conflicting variant for at least one studied LLM (aligned correct, conflicting cue-consistent). This selection rule makes the large McNemar/Wilcoxon drops and the early-changed / middle-carrier / late-readout staging partly a property of the filtered stimulus set rather than an independent discovery about how LLMs resolve arbitrary semantic conflicts. Residual patching then localizes information that was deliberately made behaviorally decisive, so the recoverable causal pattern may not hold for milder, multi-site, or non-discriminating real-world conflicts that never trigger the same output flip. Bidirectional patching and multi-model reporting reduce but do not remove this circularity.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper studies how open-weight LLMs behave when executable code and non-executable semantic cues (comments/identifiers) disagree. The authors construct 45 token-aligned Python triplets (aligned, cue-varied, implementation-varied), evaluate four ~7–8B models on final-output prediction and unit-test generation, and use residual-stream activation patching to localize token–layer states that causally shift outputs between paired conditions. Behaviorally, conflicts reduce execution-grounded correctness and often yield cue-consistent errors; mechanistically, recoverable conflict signal concentrates at the edited region (early layers), sparse intermediate “carrier” tokens (middle layers), and the readout (late layers), with an analogous pattern extending into generated assertion prefixes for tests.","tokens_in":18914,"tokens_out":1294,"duration_ms":29479,"significance":"If the results hold under clearer stimulus accounting and broader settings, this is a useful methodological contribution at the SE–mechanistic-interpretability boundary: it shows how controlled, token-aligned code contrasts plus residual-stream patching can turn ambiguous cue/code reliance into a causal localization problem rather than only a behavioral failure mode. Strengths include a paired experimental design, McNemar/Wilcoxon tests with Holm correction, bidirectional patching, recovery-threshold and margin-gap sensitivity checks, dual tasks (comprehension and test generation), and a replication package with the 45 triplets. The staged localization pattern and the observation that conflict information is recoverable at generated assertion sites are concrete, falsifiable findings that can guide follow-up circuit work and conflict-aware tooling.","major_comments":[{"comment":"Section IV-A and VII-B state that each pair must produce a meaningful conflict for at least one studied LLM, and VII-B further requires aligned-correct plus cue-consistent error on the conflicting variant for that model. This selection rule is load-bearing for RQ1.1/RQ2.1: large correctness drops and high cue-consistent error rates are partly expected by construction for selector models, so the claim that conflicts “significantly reduce” correctness and are “frequently directed toward the misleading cue” is not fully independent of filtering. Please report the pre-filter candidate pool, discard rates, which model(s) licensed each pair, and either (i) behavioral results restricted to non-selector model–pair cells or (ii) a clearly scoped claim that findings characterize this contrastive set rather than arbitrary real-world conflicts.","section":null},{"comment":"Relatedly, the abstract and RQ1.1/RQ2.1 framing present the behavioral effect as a general discovery about LLM program comprehension under semantic conflict, while the mechanistic staging (RQ1.2/RQ2.2) is measured on pairs deliberately chosen so the conflict is behaviorally decisive. Activation patching then localizes information that was selected to flip outputs. The staging result can still be informative, but the manuscript should separate (a) existence of a recoverable staged pathway on decisive micro-conflicts from (b) prevalence/severity of cue-following in unfiltered or multi-site conflicts, and temper generalization language in the abstract, §V, and §VIII accordingly (cf. VII-C).","section":null},{"comment":"Table I / Figure 2 and Table II / Figure 5 report strong effects on 45 hand-curated minimal snippets. For the central SE claim—that such conflicts matter for downstream artifacts—the unit-test results are important, yet assertion inputs are model-chosen and many tests are both/non-discriminating (Fig. 5). The paper notes this, but does not quantify how much of the pass-rate drop is cue-following versus input avoidance versus brittle formatting. A breakdown of pass-rate change attributable to cue-consistent vs non-discriminating vs neither assertions (per model/conflict) is needed so RQ2.1 is not over-read as pure semantic-cue dominance.","section":null}],"minor_comments":[{"comment":"Define the recovery score formula in the main text (currently deferred to the replication package). Even a short equation for recovery and Δ = m_S − m_D in §IV-E would make the 0.3 threshold and τ filters interpretable without external files.","section":null},{"comment":"Figure 3 is a single representative heatmap; a small multi-example appendix figure (one cue-varied, one implementation-varied, two models) would better support the claim of a “consistent pattern.”","section":null},{"comment":"Clarify tokenization of multi-token edits: §IV-E says consecutive changed tokens are one patch unit—state whether partial-token or subword splits ever forced redesign of a stimulus and how often.","section":null},{"comment":"“Carrier tokens” are a useful operational category; add an explicit decision rule (max recovery ≥ 0.3 over layers, excluding changed/readout sites) in §IV-E so the term is not only introduced in Results.","section":null},{"comment":"Minor prose: abstract says “first controlled, mechanistic study” while intro says “among the first”—align the priority claim. Also fix occasional grammar (“a models’s algorithm,” “incite a cue–implementation conflict”).","section":null},{"comment":"Report decoding details for unit-test generation beyond temp=0 (max tokens, stop sequences, whether few-shot examples were used), since assertion form affects RQ2.2 prompt construction.","section":null}],"recommendation":"major_revision","confidential_remarks":"The selection-for-cue-consistent-error issue is the main reason I chose major_revision rather than minor_revision; if the authors can show that non-selector model–pair cells still exhibit large directed effects and the same staging, this could become a strong accept after revision. Scope fit for a serious SE journal is good if claims stay about method + controlled evidence rather than broad prevalence of cue dominance in the wild. Novelty is real as an SE application of residual-stream patching, not as a new patching method per se."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a careful empirical + residual-patching study that documents a practical failure mode (LLMs follow misleading comments/names) and localizes a staged residual pattern on controlled Python micro-examples. Worth reading if you care about LLM-for-SE reliability or bringing mech-interp into code tasks.\n\nWhat is actually new is the stimulus design and dual-task framing, not activation patching itself. They build 45 token-aligned triplets (aligned / cue-varied / implementation-varied), measure execution-grounded correctness on output prediction and pytest-style tests across four open-weight models, then patch residual states bidirectionally. The behavioral result is clear: large paired drops (McNemar / Wilcoxon + Holm), with many cue-consistent errors and some non-discriminating tests that hide the conflict. Mechanistically they recover a consistent staging—changed region early, sparse intermediate “carrier” tokens mid-network, readout late—and for tests the signal also appears on generated assertion prefixes. Stats hygiene, sensitivity checks on recovery thresholds and margin gaps, both patch directions, and a replication package are real strengths. Circularity burden is low: candidates are defined from execution/cue oracles, not from the patch metric.\n\nThe soft spot the stress-test flags is real but overstated as circularity. Requiring that at least one model already flip to a cue-consistent error makes the set a filtered contrast set, so effect sizes and the clean staging partly reflect stimuli chosen to be behaviorally decisive. That is standard for patching and does not invent the recovery pattern from free parameters; it does limit how far you can generalize to mild, multi-site, or non-flipping real conflicts. Other limits they mostly own: tiny snippets, residual-only (no heads/MLPs/paths), 7–8B open models. Those are scope limits, not load-bearing cracks in the within-scope claims.\n\nWho it is for: people working on code LLMs, test generation, or SE-facing interpretability. A serious editor should send it to referees. I would engage, cite the framework and the dual-task behavioral numbers, and treat the staging as a hypothesis to stress on broader code.","headline":"Clean first causal study of cue-vs-code conflicts in LLMs; the behavioral drop and residual staging are real on their stimuli, with external validity—not circular math—as the main limit.","tokens_in":19519,"tokens_out":559,"would_cite":true,"duration_ms":10675,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"When comments and code disagree, language models often follow the misleading cue—and the conflict is recoverable from a small set of internal residual-stream states.","keywords":["large language models","mechanistic interpretability","semantic conflicts","program comprehension","activation patching","unit-test generation","code comments","software engineering"],"falsifier":"Run the same residual-stream patching protocol on a larger set of naturally occurring code-comment mismatches (not constructed micro-triplets) and check whether the early-changed-region / sparse-carrier / late-readout recovery staging still appears with comparable strength and sparsity.","tokens_in":19614,"feed_emoji":"🔧","tokens_out":670,"duration_ms":9192,"temperature":0.7,"pith_summary":"This paper asks what large language models do when executable code and non-executable semantic cues (comments, names) point to different behavior. Using 45 carefully paired Python snippet triplets and four open-weight models, it shows that such conflicts sharply cut execution-grounded correctness on both final-output prediction and unit-test generation, with many errors matching the misleading cue rather than the running code. The authors then intervene inside the models with residual-stream activation patching: swapping internal states between aligned and conflicting prompts reveals a staged pattern in which the edited cue or code region, a sparse set of intermediate carrier tokens, and the late readout site carry most of the recoverable causal signal. For generated tests, that pattern continues into the assertion prefix before the expected value is written. The practical stake is clear for AI-assisted development: stale docs and misleading names are not just noise—they can steer models toward wrong but plausible behavior, and the paper offers a causal way to localize where that decision takes shape.","feed_headline":"Code LLMs follow misleading comments when code conflicts","feed_subtitle":"Activation patching shows the conflict concentrates in a few residual-stream states before output.","key_machinery":"Residual-stream activation patching on token-aligned aligned/conflicting Python snippet pairs: residual activations at matched token-layer sites are swapped between prompts, and a recovery score measures how much the destination model’s preference shifts toward the source behavior, localizing which states causally carry the conflict.","core_discovery":"Semantic conflicts between cues and implementation significantly reduce execution-grounded correctness on output prediction and unit-test generation across four open-weight LLMs, and models frequently produce cue-consistent errors. Residual-stream activation patching recovers a consistent multi-stage causal pattern: early recovery at the changed cue or code region, middle-layer recovery at sparse intermediate carrier tokens (including generated assertion-prefix sites for tests), and late aggregation at the readout where the final answer is produced.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["LLMs follow misleading comments over conflicting code","Semantic conflicts cut code LLM accuracy on outputs and tests","Activation patching traces cue-code conflicts to few residual states","Code LLMs often trust wrong cues when comments conflict with code","Conflict signals recover early at cue/code then sparse carriers"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The claim rests on 45 hand-built, minimal, token-aligned Python micro-examples that were selected so at least one model already shows a cue-consistent error—so the staged internal pattern may not hold for larger, messier, real-world conflicts.","fun_headline_variants_meta":{"raw":{"variants":["LLMs follow misleading comments over conflicting code","Semantic conflicts cut code LLM accuracy on outputs and tests","Activation patching traces cue-code conflicts to few residual states","Code LLMs often trust wrong cues when comments conflict with code","Conflict signals recover early at cue/code then sparse carriers"]},"model":"grok-4.5","effort":"low","cost_usd":0.003046,"raw_usage":{"total_tokens":1141,"prompt_tokens":854,"num_sources_used":0,"completion_tokens":62,"cost_in_usd_ticks":30460000,"prompt_tokens_details":{"text_tokens":854,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":225,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":854,"tokens_out":62,"duration_ms":2473,"temperature":1.0,"reasoning_tokens":225,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T05:22:09.423273+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the same residual-stream patching protocol on a larger set of naturally occurring code-comment mismatches (not constructed micro-triplets) and check whether the early-changed-region / sparse-carrier / late-readout recovery staging still appears with comparable strength and sparsity.","supporting_citations":[],"review_version":1}