{"id":"37de1249-c2a0-4d5b-bb4f-494b1f4b2d9f","arxiv_id":"2607.08011","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"CodeTracer attributes backdoored code completions to responsible fine-tuning examples via structured behavioral fingerprints, embedding retrieval, and LLM semantic reasoning under realistic post-deployment constraints.","lead":"CodeTracer traces a malicious code completion back to the specific poisoned fine-tuning examples that taught the model the unsafe behavior, using only the corpus and the reported event. This post-attack forensic capability lets providers clean data and audit pipelines after backdoors evade prevention.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the honest-report assumption already flagged by the reader.","rationale":"The reader correctly isolates the honest-report assumption as the primary caveat and rates the empirical claim as sound under that assumption. My re-examination of the full manuscript finds no additional technical soft spot that would move the verdict: the adaptive-attack results (Table 15), multi-hop and multi-attacker tables, and ablation (Table 11) all support the robustness claim, and the baselines are shown to fail for well-articulated reasons (Appendix F). The recommended concrete test is simply an independent reproduction of the core numbers; a successful reproduction would leave the CONDITIONAL ACCEPT intact, while a large discrepancy would surface an unreported implementation detail. No change to the reader’s verdict is warranted.","tokens_in":25638,"tokens_out":469,"duration_ms":73021,"concrete_test":"Independently re-implement the three-stage pipeline (fingerprint extraction with Prompt 1, UniXcoder top-K=500 retrieval, Prompt-2 attribution) on the public CodeGen-Multi + GitHub Python corpus used by Yan et al. (2024) for the jinja2 SIMPLE and CB-CGPT settings; if FNR exceeds 0.05 or FPR exceeds 0.001 the headline claim weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is an empirical performance result: under the stated threat model (honest miscompletion reports + access to the full fine-tuning corpus), CodeTracer recovers the responsible poisoned examples with FNR ≤ 0.03 across ten attacks and three vulnerability classes while remaining robust to the two adaptive attacks the authors themselves designed. The experimental design (Tables 1, 3–5, 15–20, ablations, multi-attacker and multi-hop settings) is extensive and internally consistent; no circular derivation or hidden mathematical assumption undermines the numbers. The honest-report premise (Section 3 + Limitations) is a genuine scope limitation, but it is already the reader’s weakest_assumption and is standard in the poisoning-forensics literature the paper cites. No stronger load-bearing flaw is present.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces CodeTracer, a three-stage post-deployment forensic framework that attributes a reported malicious code completion (miscompletion event) back to the responsible poisoned fine-tuning examples. Given only the fine-tuning corpus and the prompt+completion pair, it (1) extracts a structured behavioral fingerprint via an external LLM, (2) narrows the search space by function-level embedding retrieval with UniXcoder and max cosine similarity (Eq. 1), and (3) uses LLM-based semantic comparison against the fingerprint to label candidates. Experiments cover three vulnerability classes (jinja2, requests, socket), eight standard plus two adaptive backdoor attacks, sixteen baselines, ablations, multi-attacker/multi-hop/noisy-report settings, and scale-up to 8 M clean examples. Reported results show FNR ≤ 0.03, near-zero FPR, large ASR reductions after removal of traced examples, and robustness under the authors’ own adaptive attacks.","tokens_in":25881,"tokens_out":991,"duration_ms":9081,"significance":"If the empirical claims hold, the work supplies a practical, gradient-free forensic capability that is currently missing for code-completion models. Preventive defenses are shown to be weak against stealthy payloads; a reliable post-hoc attribution tool would let providers audit and clean fine-tuning corpora without full retraining. Strengths include the breadth of the evaluation (ten attacks, three cases, sixteen baselines, adaptive and multi-hop settings), explicit ablations of each pipeline stage, and the realistic threat model that assumes only the corpus and a reported miscompletion. The honest-report premise is a genuine scope limit already acknowledged in the Limitations section and shared with the cited forensics literature; it does not invalidate the measured performance under that premise.","major_comments":[{"comment":"Section 3 and Limitations: the entire pipeline is conditioned on honest, true-positive miscompletion reports. Fabricated reports would cause CodeTracer to attribute blame to clean data. While the authors correctly flag this and note that the same assumption appears in prior forensics work, the manuscript should quantify the risk more carefully—e.g., by measuring how many clean examples are flagged when a deliberately false or noisy report is supplied, and by discussing concrete human-in-the-loop filters beyond a one-sentence suggestion.","section":null},{"comment":"Section 4.2 / Eq. (1) and Table 9: the claim that function-level max-similarity retrieval recovers nearly all responsible examples rests on an empirical axiom that is only partially stress-tested. Table 9 shows FNR rising to 0.05–0.09 when top-K drops to 250; the adaptive embedding-perturbation attack (Appendix E, Table 15) is the only direct attempt to break the embedding stage. A short additional experiment that systematically varies function length, padding volume, or multi-function payloads would make the recall guarantee more convincing.","section":null}],"minor_comments":[{"comment":"Algorithm 1 appears twice in the appendix (identical copies). Remove the duplicate.","section":null},{"comment":"Tables 3 and 4 report FPR/DACC as fractions over 80 000; the absolute counts are useful but the tables become hard to scan. Consider moving the raw counts to an appendix or using scientific notation for the denominators.","section":null},{"comment":"Figure 5 caption and axis labels are readable, yet the TROJAN panel uses a different N scale (280–2240) without an explicit note in the main text; a one-sentence clarification would help.","section":null},{"comment":"Prompt 1 and Prompt 2 are given in full; a short note on temperature / decoding settings used for the external LLM would improve reproducibility.","section":null},{"comment":"A few typographical inconsistencies remain (e.g., “CodeTracer” sometimes rendered without the space after the period in running text; “jinja2” vs. “Jinja2”). A light copy-edit pass would clean these.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The experimental design is unusually thorough for this sub-area and the central empirical claim appears solid under the stated threat model. The honest-report limitation is real but already standard; I do not view it as grounds for rejection. Minor revision is appropriate so the authors can tighten the discussion of false-flag risk and add a brief extra stress test of the retrieval stage."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a clean engineering paper that fills a real gap. Prior poisoning forensics either need gradients (unavailable for Codex/CodeLlama-style pipelines) or were built for RAG/multi-agent settings and simply do not transfer. CodeTracer is the first end-to-end method that starts from a single reported miscompletion, builds a structured behavioral fingerprint (issue type, canonical form, variants, etc.), does function-level code-to-code retrieval with UniXcoder, then uses an external LLM for semantic attribution. That combination is new and the numbers are strong: FNR ≤ 0.03 across ten attacks (including two adaptive ones the authors designed against themselves), three vulnerability classes, near-zero FPR, and ASR collapsing after removal of the traced samples. Ablations, multi-attacker, multi-hop, noisy-report, and 8 M-scale experiments are all present and consistent. The baselines (All-at-Once, Context-Cite, PoiF, TracLLM, etc.) fail for understandable reasons the authors diagnose, so the comparison is fair rather than straw-man.\n\nThe soft spots are real but proportionate. The honest-report assumption is a genuine scope limit (false-flag reports would poison the attribution), yet it is standard in the literature they cite and they flag it in the limitations. Dependence on proprietary LLMs and the absence of released code/data hurt independent reproduction; top-K and the external model are free parameters, though the sensitivity tables show the method is not brittle. None of these undermine the central empirical claim under the stated threat model.\n\nThis is for people working on AI supply-chain security, code-model defenses, and post-deployment forensics. It is not theory, but it is careful, useful, and better-executed than most of the related forensic work. I would send it to referees; the experimental package is already strong enough that a serious review can focus on scope and artifacts rather than basic soundness.","headline":"Solid, practical forensic pipeline for code-completion backdoors that actually works under the no-gradient constraint and beats a large baseline suite.","tokens_in":26433,"tokens_out":496,"would_cite":true,"duration_ms":5891,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"CodeTracer traces a malicious code completion back to the poisoned fine-tuning examples that caused it, using only the fine-tuning corpus and the reported miscompletion.","keywords":["code completion","backdoor attacks","poisoning forensics","LLM attribution","behavioral fingerprint","fine-tuning data","code security"],"falsifier":"Run the full CodeTracer pipeline on a held-out fine-tuning set that contains known backdoored examples for the jinja2, requests, and socket cases; if the false-negative rate rises above 0.10 for any of the ten attacks or the adaptive variants, the central accuracy claim is falsified.","tokens_in":26558,"feed_emoji":"🔍","tokens_out":616,"duration_ms":8955,"temperature":0.7,"pith_summary":"Code completion models can be quietly poisoned during fine-tuning so that, under specific triggers, they emit unsafe code. Existing defenses often miss sophisticated backdoors, leaving operators without a way to find the root cause after the fact. This paper establishes that a forensic pipeline called CodeTracer can attribute a reported malicious completion to the responsible fine-tuning examples by extracting a structured behavioral fingerprint, retrieving a shortlist of semantically similar training snippets, and using an external LLM to decide which of them encode the same unsafe logic. The method needs no gradients, no attacker knowledge, and no full retraining. If it works, developers and service providers can audit and clean poisoned datasets after a backdoor has already manifested, restoring trust without discarding the entire model.","feed_headline":"CodeTracer finds the poisoned examples behind a bad autocomplete","feed_subtitle":"A three-stage forensic pipeline attributes malicious code completions to their fine-tuning sources without gradients","key_machinery":"CodeTracer’s three-stage pipeline: (1) an external LLM extracts a structured behavioral fingerprint (issue type, canonical form, normalized semantics, variants, transformations) from the miscompletion; (2) a code encoder retrieves the top-K most similar function-level snippets from the fine-tuning corpus; (3) the same LLM performs fine-grained semantic comparison of the fingerprint against each candidate and labels matches as responsible backdoors.","core_discovery":"Under realistic post-deployment constraints—access only to the fine-tuning corpus and a user-reported miscompletion event—CodeTracer recovers the backdoored training examples responsible for a malicious completion with consistently low false-negative rates (at most 0.03) across three vulnerability classes and ten poisoning strategies, while keeping false-positive rates near zero and remaining effective against adaptive attacks that deliberately target its own stages.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["CodeTracer traces bad autocomplete to its poisoned training data","Forensic tool links malicious code completions to backdoor examples","CodeTracer attributes unsafe suggestions to fine-tuning poisons","Recovering the data that planted backdoors in code autocomplete","Pinpointing fine-tuning samples behind a reported miscompletion"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"Users honestly report genuine miscompletion events and do not submit fabricated or false-flag reports that would cause the pipeline to blame clean data.","fun_headline_variants_meta":{"raw":{"variants":["CodeTracer traces bad autocomplete to its poisoned training data","Forensic tool links malicious code completions to backdoor examples","CodeTracer attributes unsafe suggestions to fine-tuning poisons","Recovering the data that planted backdoors in code autocomplete","Pinpointing fine-tuning samples behind a reported miscompletion"]},"model":"grok-4.5","effort":"low","cost_usd":0.004936,"raw_usage":{"total_tokens":1366,"prompt_tokens":719,"num_sources_used":0,"completion_tokens":64,"cost_in_usd_ticks":49360000,"prompt_tokens_details":{"text_tokens":719,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":583,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":719,"tokens_out":64,"duration_ms":5405,"temperature":1.0,"reasoning_tokens":583,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T13:45:29.990570+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the full CodeTracer pipeline on a held-out fine-tuning set that contains known backdoored examples for the jinja2, requests, and socket cases; if the false-negative rate rises above 0.10 for any of the ten attacks or the adaptive variants, the central accuracy claim is falsified.","supporting_citations":[],"review_version":1}