{"id":"ea6840f7-a127-48c8-a1e0-f112f337d592","arxiv_id":"2607.24964","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Adaptive adversarial source-code comments evade four LLM vulnerability detectors at >90% success on 125 real null-pointer bugs without altering program behavior.","lead":"Adversarial comments alone flip LLM vulnerability detectors from “vulnerable” to “safe” over 90% of the time without changing code. The work shows current detectors over-trust natural-language claims and that sanitization beats prompt defenses.","discovery_kind":"new_method","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The benchmark’s ground truth may be the weakest link: most synthesized NPDs are certified by an LLM judge rather than manual or executable validation, so some reported “true-positive flips” could instead be corrections of non-vulnerabilities.","rationale":"I partially agree with the reader’s weakest-assumption choice. Access to detector reasoning and repeated adaptive queries is an important deployment condition, especially given the much weaker non-adaptive baseline. However, the paper states that assumption explicitly and evaluates the attack under it, so it mainly limits transfer to less informative production APIs rather than undermining the reported laboratory result.\n\nThe LLM-judge ground truth is more load-bearing for the paper’s central numerical claim because TPclean and ASR are defined relative to benchmark labels. The paper has useful safeguards—real CVE-derived tasks, repository builds, test execution where available, rejection of candidate implementations, and partial correspondence with original dereference sites—so this is not an accusation that the benchmark is invalid. It is a missing independent check on the exact property being measured: genuine reachable vulnerability.\n\nI would keep the verdict at CONDITIONAL rather than reject the paper. The qualitative phenomenon is independently supported by detailed detector traces, multiple architectures, presentation ablations, transfer to UAF, and defense results. But unconditional acceptance should wait for blinded human or executable validation of the benchmark labels, plus the already relevant qualification that adaptive ASRs assume detector feedback.","tokens_in":25883,"tokens_out":2671,"duration_ms":106380,"concrete_test":"Have two independent C/C++ security reviewers, blinded to the LLM-judge labels and detector outputs, audit all 125 accepted implementations with the full repository context and original CVE fix. Classify whether each contains a genuinely reachable attacker-introduced CWE-476; adjudicate disagreements. Recompute each detector’s TPclean and ASR using only confirmed vulnerabilities. If the confirmed subset preserves ASRs above 90%, the concern does not materially land; if many instances are rejected or ASR falls substantially, the headline claim weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim depends on every denominator instance being a genuine, reachable CWE-476 that the detector correctly flagged before the comment was added. In §4.1, functionality is checked by building and running available tests, but vulnerability validity is certified by an independent LLM judge (GPT-5-mini). Passing tests is explicitly only a compatibility check, and the judge’s criteria are not shown to include execution, manual audit, or inter-rater validation.\n\nThis matters most because the benchmark is dominated by difficult null sources: Appendix E reports that CppCheck, Infer, and CodeQL together detect only 13/125 bugs. The paper plausibly attributes this to structural limits of static analysis, but the converse also means that nearly 90% of ground-truth labels rest primarily on LLM judgment about interprocedural, field-sensitive, or call-site reachability—precisely the kind of reasoning that can be wrong when external constraints are not visible. If the judge and evaluated detectors share a tendency to infer plausible but invalid null paths, some TPclean instances may be false vulnerabilities; a comment that flips those verdicts would be a correction, not evasion.\n\nThe 42% independent recreation of the original dereference site, build validation, and rejection of 17 candidates provide meaningful support. They do not, however, establish the correctness of the remaining labels at the level required for ASRs as high as 91.2–100%. This is an internal-validity concern about the measured effect, distinct from the already-noted external-validity question of whether deployed detectors expose rationales.","agreement_with_reader":"partial"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The paper studies whether non-executable source-code comments can cause LLM-based vulnerability detectors to miss vulnerabilities they would otherwise catch. The authors build a benchmark of 125 coding tasks derived from real CWE-476 CVE-fix commits (MegaVul), have an attacker model synthesize functional-but-vulnerable implementations that build and pass tests, and then run ALIBI, a three-phase adaptive black-box attack that reads detector reasoning, seeds adversarial comments in 10 presentation styles, and refines them over 5 rounds. On four architecturally distinct detectors (OPENVUL, VULNLLM-R, VUL-RAG, VULTRIAL), ASR over each detector's clean true positives is 91.2–100%. Ablations cover presentation style, portfolio set-cover, refinement curves, and a non-adaptive baseline (2.6–17.3%). Four defenses are evaluated; pre-detector sanitization (D4) achieves <3% evasion. Generalization to CWE-416 (N=70) is shown. The central mechanism is a \"comment trust prior\": detectors accept plausible but unverifiable external safety claims.","tokens_in":26220,"tokens_out":5478,"duration_ms":42014,"significance":"If the results hold, this is a significant and timely security result: it identifies a persistent, in-artifact attack channel (developer-style comments) distinct from prior PR-context or code-perturbation attacks, and shows it defeats four very different detector architectures at high rates. Strengths worth naming: (i) ASR is restricted to TPclean, so flips measure evasion rather than baseline blindness — a methodologically careful choice; (ii) the non-adaptive baseline cleanly isolates the contribution of adaptivity; (iii) the presentation-style ablation with a shared-payload control and the greedy set-cover analysis are genuinely informative; (iv) defenses are evaluated against adaptive refinement, not just frozen payloads, and the D1–D4 hierarchy is a useful design lesson; (v) the benchmark construction (CVE-fix-to-coding-task, build/test validation, 42% independent recreation of original deref sites) is a reusable artifact; (vi) qualitative traces make the failure mechanism falsifiable and inspectable. The work is reproducible in principle and makes concrete, testable claims.","major_comments":[{"comment":"Ground truth for the benchmark is certified solely by an LLM judge (GPT-5-mini); there is no dynamic validation (e.g., a harness driving null to the deref site under ASan), no manual audit, and no inter-rater check. Since ASR is computed over TPclean, any non-vulnerability in the denominator turns a reported 'flip' into a correction of a false alarm, directly inflating the headline 91–100% ASRs. The 42% site-recreation statistic and build/test checks support plausibility but do not bound label error, and the 13/125 static-analysis recall (App. E/Table 9) is equally consistent with imperfect labels as with 'structural limits.' Concrete fix: manually audit a random subsample and report judge agreement, dynamically validate the intra-procedural categories where feasible, and report an estimated label-error rate with adjusted ASR bounds.","section":"§4.1 Benchmark Evaluation; Appendix E / Table 9"},{"comment":"The headline ASRs assume the attacker observes the detector's natural-language reasoning and obtains up to ~60 queries (10 strategies × 6 rounds) against a local copy. The paper's own non-adaptive baseline (Table 6: 2.6–17.3%) shows the result is driven by this feedback channel, and even round-0 initialization consumes one reasoning trace. An intermediate condition — adaptive refinement with verdict-only (binary) feedback — would quantify how much of the effect depends on reasoning disclosure versus query access alone, and would substantially strengthen external validity for deployments that return only a verdict. At minimum, the abstract and §4.2 takeaway should scope the headline numbers to the reasoning-visible, multi-query setting.","section":"§3.1 Threat Model; §4.4; Tables 2 and 6"},{"comment":"The TPclean denominators differ between tables for all four detectors (OPENVUL 115 vs 114; VULNLLM-R 113 vs 111; VUL-RAG 111 vs 115; VULTRIAL 105 vs 98) with no explanation. Presumably the clean verdicts are nondeterministic across runs (API models, sampling; OPENVUL uses n=8 majority vote). Because TPclean is the denominator of the central metric, the paper must state how nondeterminism is controlled (temperature, seeds, voting), which clean run defines TPclean in each experiment, and how stable the clean verdicts are; if the baseline fluctuates, ASR should be defined relative to a fixed clean run and variance across repetitions reported.","section":"Table 2 vs Table 6: TPclean mismatch"},{"comment":"D4 is evaluated only by sanitization-evasion rate on attack payloads (<3%); its collateral cost is not measured. An aggressive sanitizer can achieve low evasion by stripping most comments, yet §3.4 itself argues comments improve detection [58,63], so the interesting quantity is the false-removal rate on legitimate comments and the post-sanitization detection quality on clean, commented code. Neither is reported, and the sanitizer's model/prompt is unspecified (Appendix A details detectors but not D4). Please report benign-comment retention, clean-code detection impact, and the D4 configuration; otherwise 'D4 substantially improves resilience' is only half-evaluated.","section":"§4.5 / Table 5: D4 evaluation"}],"minor_comments":[{"comment":"No confidence intervals are reported for ASRs; with N≈105–115 per detector, binomial CIs are roughly ±3–5 points and should accompany Tables 2, 5, and 7.","section":"Tables 2, 5, 7"},{"comment":"The query cost of the headline ASR (up to 60 detector queries per instance across 10 strategies and 5 refinement rounds) is never stated alongside Table 2; Figure 4 reports cost only for the 5-type portfolio. Please state it explicitly.","section":"§4.2 / §4.4"},{"comment":"Results are pooled across detectors; since the persuasiveness ranking plausibly interacts with architecture (e.g., VUL-RAG's retrieval channel), a per-detector breakdown would be informative, at least in an appendix.","section":"Table 3"},{"comment":"Example 3's attack comment contains LaTeX artifacts ($\\in$, $\\forall$) inside what is presented as a C comment; if this is a rendering artifact please fix, and if the literal payloads contained these, note it.","section":"Example 3"},{"comment":"'slug' is used without definition; 'TPclean' vs 'TP clean' typesetting is inconsistent; the three instances 'excluded as unfixable during evaluation' (§4.1) should be explained.","section":"§4.1–4.2 notation"},{"comment":"The Gemma attacker is evaluated only on the 46 slugs it could synthesize; this is a success-conditioned subsample and the resulting ASRs (77–97.5%) are not comparable to Table 2 without noting the selection effect.","section":"Appendix F"},{"comment":"VULNLLM-R has TPclean=41/70 on UAF; ASR=100% on a 41-instance denominator deserves a CI and a remark on the low clean detection rate.","section":"Table 7"},{"comment":"Duplicate references: [24] and [25] are the same Flashboom paper, as are [33] and [34]; please deduplicate.","section":"References"},{"comment":"Axis/denominator caption is hard to parse ('All 125 TPclean slugs included' while denominators are per-detector); please clarify whether curves are per-detector over that detector's TPclean.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The empirical core is careful and the phenomenon is convincingly demonstrated qualitatively; my hesitation is quantitative. The 91–100% ASRs rest on LLM-judge-certified labels with no manual or dynamic spot-check, and on a reasoning-visible threat model whose weight is only partially bounded by the non-adaptive baseline. Both are addressable within scope (a manual audit subsample; a verdict-only adaptive condition), which is why I recommend major revision rather than rejection. No concerns about citation practice or dual-use handling; the ethics section and defense evaluation are responsible."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a solid systems-security paper showing that non-executable comments alone can systematically flip four different LLM vulnerability detectors from vulnerable to benign, without touching program behavior. The numbers are high (91–100% ASR on each detector’s clean true positives over 125 NPD instances) and the comparison to a weak non-adaptive baseline (roughly 3–17%) makes the adaptive piece load-bearing rather than decorative.\n\nWhat is actually new is the combination of (1) persistent developer-style safety claims inside the code artifact, (2) an adaptive black-box loop that mines detector reasoning, and (3) a CVE-fix-to-coding-task benchmark so the attack is evaluated on freshly synthesized implementations rather than historical vulnerable snippets. Prior work mostly obfuscates executable code or poisons PR text that disappears after merge. The presentation ablations, set-cover diversity result, refinement curves, defense hierarchy (prompt weak; isolation and sanitization stronger), and UAF transfer are the parts I would actually use.\n\nSoft spots, in proportion. The threat model assumes the attacker sees natural-language explanations and can query a local copy iteratively. That is honest in the paper, but end-to-end ASRs will drop if production gates return only a bit. Second, vulnerability labels after build/test rest heavily on an LLM judge; static tools only catch 13/125, which the authors read as “why we need LLMs,” but it also means most of the denominator is not independently executed or hand-audited. The 42% recreation of original dereference sites and the filter that dropped broken candidates help; they do not fully close the internal-validity gap the stress-test flags. I would not throw out the result over that, but I would want clearer judge criteria or a human sample audit in revision. Free parameters (budget, strategy set, attacker/judge models) are normal for this genre and mostly ablated.\n\nCitation pattern and related work look fair: static analysis, comment/NL channels, prompt injection, and prior detector attacks are in the right places. No circular math story—this is empirical measurement.\n\nWho it is for: people building or deploying LLM code auditors, and anyone working on artifact trust / indirect injection in agentic coding. Worth a serious referee. I would bring it to reading group and cite the attack surface and defense ranking. Send to peer review.","headline":"Real attack surface, careful measurement: adaptive in-source comments flip LLM vuln detectors at >90% ASR, with the main caveats being rationale access and LLM-judged ground truth.","tokens_in":27311,"tokens_out":592,"would_cite":true,"duration_ms":16473,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Adversarial code comments alone can flip LLM vulnerability detectors from vulnerable to safe without changing the program.","keywords":["LLM vulnerability detection","adversarial code comments","prompt injection","adaptive black-box attack","null-pointer dereference","comment trust prior","AI coding agents","defense by sanitization"],"falsifier":"Run the same 125-instance protocol with detectors that return only a binary verdict and allow no multi-round adaptive querying: if end-to-end flip rates then stay near the paper’s non-adaptive baseline (roughly 3–17%) rather than above 90%, the central adaptive-attack claim does not hold in that deployment setting.","tokens_in":27149,"feed_emoji":"💬","tokens_out":990,"duration_ms":21067,"temperature":0.7,"pith_summary":"This paper shows that large language models used as vulnerability detectors can be systematically misled by ordinary-looking source-code comments. An attacker-controlled coding agent can implement new functionality, leave a real bug in place, and insert comments that assert unverifiable safety claims—caller contracts, fake static-analysis results, or step-by-step “security analyses”—so the detector overrides its own correct reading of the code. On a benchmark built from real null-pointer-dereference fixes turned into coding tasks, the authors’ adaptive black-box framework, ALIBI, flips more than 90% of each detector’s clean true positives across four systems, including specialized open-weight models and multi-agent setups, and reaches 100% on one of them. The same pattern holds for use-after-free bugs. Prompt-only “don’t trust comments” instructions barely help against adaptive refinement; stripping or isolating unverifiable comments before the detector reasons does. The practical point is that current detectors overweight natural-language context relative to program evidence, and that imbalance is an attack surface in AI-assisted development workflows.","feed_headline":"Fake code comments fool LLM bug detectors over 90% of the time","feed_subtitle":"Unverifiable safety claims flip correct verdicts without changing program behavior","key_machinery":"ALIBI: a three-phase adaptive black-box attack that synthesizes a functional vulnerable implementation, seeds presentation-specific comments from the detector’s own reasoning, then iteratively refines those comments against detector feedback—especially strategies that steer reasoning (COT) or fabricate external tool authority.","core_discovery":"Non-executable adversarial comments are enough to make representative LLM-based vulnerability detectors change a correct VULNERABLE verdict to BENIGN while the code, tests, and real bug stay unchanged. Across 125 real-world null-pointer-dereference instances, adaptive comment attacks exceed 90% success on every system studied and hit 100% on one, by exploiting a shared comment trust prior rather than by hiding the bug in the executable code.","pith_inferences":["Any LLM reviewer that treats documentation, PR text, or in-code narrative as soft evidence faces a similar trust-calibration problem, not only dedicated vulnerability detectors.","Vendors that expose chain-of-thought or debate transcripts to users may be handing attackers the exact signal ALIBI refines against.","A split pipeline—strong model for code, cheap local model only for comment crafting—would still be enough once a vulnerable implementation exists, lowering the bar for the attack.","Human reviewers and classical static analyzers that ignore comments would still see these bugs; hybrid gates that require agreement with comment-blind tools would blunt the channel."],"forward_implications":["Security gates that feed full source—including comments—into LLM detectors can be bypassed by a malicious or compromised coding agent without failing builds or tests.","Presentation that launders claims as reasoning or tool output is more dangerous than plain developer notes at the same bug site.","Prompt-only distrust policies are insufficient against adaptive attackers who still shape the detector’s later reasoning.","Architectural isolation of a comment-free baseline and pre-detector removal of unverifiable comments are the mitigations the paper finds actually move the needle.","The failure mode is not limited to null-pointer bugs; the same unverifiable-outside-code claim pattern works on use-after-free."],"fun_headline_variants":["Adversarial comments flip LLM bug detectors over 90% of the time","Fake comments make LLM vulnerability scanners miss real bugs","Code comments alone fool LLM detectors on 90%+ of real bugs","LLM bug finders trust comments enough to miss null-pointer flaws","Adaptive comment attacks beat LLM vulnerability detectors at 90%+"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The attacker can see the detector’s natural-language explanation after each query and refine comments against a copy of the detector; without that feedback loop, success falls sharply toward a weak fixed-comment baseline.","fun_headline_variants_meta":{"raw":{"variants":["Adversarial comments flip LLM bug detectors over 90% of the time","Fake comments make LLM vulnerability scanners miss real bugs","Code comments alone fool LLM detectors on 90%+ of real bugs","LLM bug finders trust comments enough to miss null-pointer flaws","Adaptive comment attacks beat LLM vulnerability detectors at 90%+"]},"model":"grok-4.5","effort":"low","cost_usd":0.002915,"raw_usage":{"total_tokens":1111,"prompt_tokens":829,"num_sources_used":0,"completion_tokens":72,"cost_in_usd_ticks":29148000,"prompt_tokens_details":{"text_tokens":829,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":210,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":829,"tokens_out":72,"duration_ms":4400,"temperature":1.0,"reasoning_tokens":210,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T04:51:10.586308+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the same 125-instance protocol with detectors that return only a binary verdict and allow no multi-round adaptive querying: if end-to-end flip rates then stay near the paper’s non-adaptive baseline (roughly 3–17%) rather than above 90%, the central adaptive-attack claim does not hold in that deployment setting.","supporting_citations":[],"review_version":1}