{"id":"bb7a3bca-d171-4772-98cd-02f020e4c55e","arxiv_id":"2602.10778","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Retraining only the neurons most responsive to security judgments raises a code LLM's default safe-code rate to 87.5% on C++ (average) with about 1.9 million trainable parameters.","lead":"GoodVibe isolates the small set of neurons inside a code-writing AI that handle security and retrains only those neurons, so the model produces safer code by default. It claims security gains matching full fine-tuning with vastly fewer trainable parameters, but the safety scores come from an AI judge whose reliability is not independently verified.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The security gains may be an artifact of an overlapping learned judge: the evaluator's training data is undisclosed and plausibly drawn from the same CyberNative distribution used to select and fine-tune security neurons, so Table 1 lacks independent ground-truth support.","rationale":"The reader's weakest assumption is exactly the load-bearing issue: the safety metric is a learned judge whose training data may overlap the CyberNative dataset used to select and fine-tune security neurons. The paper does not disclose the judge's training data; Section 7 only says the judge is 'trained independently from all generation models,' which is not the same as being trained on a disjoint dataset. Since evaluation prompts are also drawn from CyberNative, there is a plausible closed loop: the same distribution is used to identify security neurons, fine-tune them, and score the result. This would not be fatal if independent ground-truth validation existed, but none is provided. I considered other potential issues — e.g., the mismatch between Eq. (1), which defines importance via parameter gradients, and Section 4.1, which computes output-gradient magnitudes — but that is an implementation-level ambiguity rather than a direct threat to the central empirical claim. The parameter-efficiency and utility-preservation results are structurally credible and do not depend on the judge; however, the paper's core security contribution is entirely metric-dependent. Therefore, the reader's conditional verdict is appropriate: the method is plausible, but the central security claim should not be accepted until the judge's data is disclosed and independent validation is performed. I recommend no change to the verdict.","tokens_in":20155,"tokens_out":4648,"duration_ms":45401,"concrete_test":"Require the authors to disclose the judge's training corpus and all dataset splits. Then re-run the Table 1 evaluation with a judge trained strictly on data disjoint from CyberNative, and additionally validate a random sample of 100 generated code snippets per model/language against human expert labels or a vetted static analyzer (e.g., CodeQL or Semgrep). Compute GoodVibe-minus-baseline deltas under this independent judge. If the deltas shrink to within noise or reverse, the metric-circularity concern is confirmed and the central security claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that neuron-selective fine-tuning makes code LLMs secure by default (Section 3.2, Section 5.2) — is measured exclusively by a fine-tuned Qwen3-0.6B judge (Section 4.3). The judge's training data is never disclosed, and Section 7 only says it is 'trained independently from all generation models,' which does not rule out overlap with the CyberNative Code Vulnerability and Security Dataset. Section 5.1 states that CyberNative supplies the 424 secure/insecure pairs used both for neuron identification and for supervised fine-tuning; evaluation prompts are 'held-out' but 'drawn from the dataset.' If the judge was fine-tuned on CyberNative or a derivative, then GoodVibe's improvements in safe-response rate may reflect better matching of the judge's learned patterns rather than reduced real-world vulnerabilities. This is load-bearing because every headline number — Table 1, the Swift/Go results, and the abstract's '2.5x improvement' — is filtered through this judge. No human audit, no static-analyzer validation, and no vulnerability-count ground truth is provided. The auxiliary false-positive experiment in Section 7 does not establish the judge's validity as an absolute oracle; it only compares a base judge to an SFT judge without describing the ground-truth labels used. The paper's own language — 'a consistent measurement instrument rather than an absolute oracle' — concedes the limitation, but the consistency argument only addresses noise, not systematic bias from training-data overlap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"GoodVibe proposes a neuron-level fine-tuning method to make code LLMs generate secure code by default. The method identifies a small set of 'security-critical' neurons via gradient-based attribution on a binary secure/insecure classification task, clusters these neurons by their gradient-importance profiles, and fine-tunes only the clustered update directions (i.e., only rows of weight matrices corresponding to selected neurons). The paper evaluates GoodVibe on six open-source LLMs (CodeLlama, Llama-3, Qwen3, Gemma-3) across C++, Java, Swift, and Go, using a fine-tuned Qwen3-0.6B judge model to label generated samples as secure or insecure. The main empirical claims are: (1) security-related reasoning is localized to a small neuron subspace; (2) GoodVibe improves safe-response rates from an average baseline of 35.1% to 87.5% (C++) and from 59.3% to 76.0% (Java), approaching or exceeding full fine-tuning; (3) this is achieved with ~1.9M trainable parameters on average and 2.4 PFLOPs on CodeLlama-7B, versus 6.2M and 8.6 PFLOPs for LoRA; and (4) general utility on GSM8K/ARC/MMLU is largely preserved (average 0.84% drop). The paper includes ablations on neuron-identification method, clustering, and hyperparameters, plus a discussion of the judge's reliability.","tokens_in":20538,"tokens_out":3818,"duration_ms":34271,"significance":"If the empirical claims are correct, GoodVibe is a meaningful contribution: it provides rare evidence that security-relevant behavior in code LLMs is concentrated in a small, identifiable subset of neurons, and that targeted adaptation can improve security at a fraction of full fine-tuning's parameter and compute cost. The paper is unusually transparent in documenting its pipeline, hyperparameter choices, ablations, and limitations. The use of multiple model families and languages, plus utility benchmarks, strengthens the generalizability story. However, the central result depends entirely on the validity of the automated judge model. Since the judge's training data is not disclosed and may overlap with the dataset used for neuron identification and fine-tuning, the headline improvements could reflect overfitting to the judge's learned patterns rather than real reductions in vulnerabilities. The paper also selects key hyperparameters (top-k=50, silhouette=0.05, epochs=2) on the same judge-based metric and reports single-run percentages without error bars. These issues must be resolved before the central claims can be accepted.","major_comments":[{"comment":"The validity of the automated judge is not established. All security metrics (Tables 1, 4–9) are produced by a Qwen3-0.6B judge fine-tuned on undisclosed security-labeled data. Section 5.1 states that CyberNative supplies the 424 secure/insecure pairs used for both neuron identification and fine-tuning, and that evaluation prompts are 'held-out' but 'drawn from the dataset.' Section 7 only says the judge is 'trained independently from all generation models,' which does not rule out overlap with CyberNative. If the judge's training distribution overlaps CyberNative, GoodVibe's improvements may reflect alignment with the judge's learned patterns rather than reduced real-world vulnerabilities. The paper explicitly concedes the judge is 'a consistent measurement instrument rather than an absolute oracle,' but consistency addresses noise, not systematic bias. The authors should disclose the j","section":"§4.3, §5.1, §7"},{"comment":"Key hyperparameters (top-k=50, silhouette threshold=0.05, epochs=2) are selected by maximizing the same judge-based safe-response rate that is then reported as the main result. Section 6.2 shows that these values are peaks on the reported evaluation set. This is selection on the evaluation metric; without a separate validation set or nested cross-validation, the reported averages (87.5% C++, 76.0% Java) are optimistically biased. In addition, all tables report single-run percentages with no variance or confidence intervals. Given that the judge itself is stochastic and model generation is sampled, the lack of repeated runs makes it impossible to assess whether differences like GoodVibe's 87.5% vs LoRA's 87.4% on C++ are meaningful. Please provide runs/error bars and a holdout-based hyperparameter selection protocol.","section":"§5.1, §6.2"},{"comment":"The claim 'matching or exceeding full fine-tuning' is not supported by Table 1. On Java, the GoodVibe average is 76.0% vs full fine-tuning's 83.0%; only gemma-3-4b-it does better. On C++, the averages are close (87.5% vs 86.3%) but individual models vary widely (e.g., Meta-Llama-3-8B: 85.9% vs 96.5%). The abstract's 'matching or exceeding full fine-tuning' overstates the Java result. The paper should either present per-language averages and explain the discrepancy, or temper the claim to 'competitive with full fine-tuning on C++ but below it on Java.'","section":"§5.2, Table 1"},{"comment":"The clustering ablation undermines the claim that clustering is necessary for good security. Without clustering, average safe-response rates are higher on both C++ (90.0% vs 87.5%) and Java (82.6% vs 76.0%), with a large drop for gemma-3-12b on Java (84.0% → 61.3%). The paper attributes clustering's value to reduced parameters and training cost, but the title and abstract highlight 'security-by-vibe' improvements. If clustering consistently reduces security performance, the conclusion that 'both components are necessary' is misleading. The paper should either present the security-performance cost transparently and justify it as an efficiency trade-off, or reconsider the framing of clustering as a core security-enabling component.","section":"§6.1, Table 6"},{"comment":"The FLOPs numbers are surprising and lack supporting methodology. Why does LoRA (8.6 PFLOPs) cost almost twice full fine-tuning (4.5 PFLOPs), and why does GoodVibe (2.4 PFLOPs) reduce FLOPs by 46% relative to full fine-tuning when all methods must run forward and backward passes through the entire frozen base model? The paper says LoRA's increase stems from low-rank projections, but that does not explain a ~1.9x overhead; and GoodVibe's savings are not clearly tied to the number of trainable parameters (which do not directly affect FLOPs). Please provide the exact FLOPs-estimation procedure (layer-wise counting, attention included, optimizer states, etc.) and verify that the comparisons are apples-to-apples.","section":"§5.2, Table 3"}],"minor_comments":[{"comment":"The abstract says 'over 4,700× fewer trainable parameters' and 'more than 3.6×' vs LoRA, but Table 3 shows 8.6/2.4 = 3.58, which rounds to 'over 3.5×' rather than 'more than 3.6×.' Also, Table 1 uses non-standard decimal conventions ('51,7%', '85,6') and the release time for Meta-Llama-3-8B-Instruct (2024.12) appears inconsistent with the known Llama-3 release; please verify.","section":"Abstract and Table 1"},{"comment":"Baseline (pre-trained) per-model results are omitted from Table 4 and only averages are given in the text. Since the table is meant to show GoodVibe vs LoRA, providing baselines would help readers assess the absolute improvement in Swift/Go.","section":"§5.3, Table 4"},{"comment":"The expectation in Eq. (1) is not well-defined: it uses E_{(x,y)∼D} over gradient magnitudes, but gradient magnitudes are not necessarily integrable if the model has unbounded gradients. The practical averaging over a finite dataset is fine; please state that this is an empirical average and note any clipping.","section":"§3.3, Eq. (1)"},{"comment":"The silhouette score threshold is described as controlling 'the number of clusters,' but silhouette scoring is a quality measure, not a clustering parameter. Please explain how a threshold on silhouette translates into a specific number of clusters, and clarify whether k is chosen beforehand or inferred from the criterion.","section":"§4.2, §6.2"},{"comment":"There are typos: 'lanugaes' and 'caan.' Also, the Go example uses 'copy(buffer[:], input)' in the baseline; the `copy` function is safe in Go (it copies at most len(dst) bytes), so calling this a buffer overflow is inaccurate. If the intended vulnerability is an unbounded copy into a fixed-size buffer, this is not a valid illustration in Go.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid methodological skeleton and could be a significant contribution, but the empirical core is currently not verifiable because the judge's training data is undisclosed and possibly overlapping with the tuning data. I would encourage the editor to require the authors to disclose the judge's training source, demonstrate disjointness, and add an independent validation experiment. The FLOPs methodology also needs clarification. If those are addressed, the paper may be suitable for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: GoodVibe is a sensible idea with a clean mechanistic story and an honest write-up, but the experimental evidence is not yet trustworthy enough to support the headline claim. The whole evaluation depends on a learned judge model whose training data is undisclosed and plausibly drawn from the same CyberNative dataset used to select and fine-tune security neurons. If that overlap exists, Table 1 measures how well the model matches the judge's learned patterns, not real vulnerability reduction.\n\nWhat is genuinely new: the combination of gradient-based neuron attribution, activation-driven clustering, and selective fine-tuning applied to code security. The ablation showing gradient selection beats activation selection is informative. The efficiency numbers are structurally believable — if you only update a few million parameters, the FLOPs and parameter counts should drop. The paper is clearly written and the authors are appropriately cautious about the judge in Section 7, calling it 'a consistent measurement instrument rather than an absolute oracle.'\n\nThe soft spots are real. First, the judge's training data is never disclosed. Section 4.3 says it is 'trained independently from all generation models,' but that does not rule out overlapping with CyberNative. Since the same dataset supplies neuron identification, fine-tuning, and held-out evaluation prompts, a judge trained on that distribution would create a circularity that inflates the security gains. The auxiliary false-positive experiment in Section 7 does not fix this, because the ground-truth labels behind the judge's accuracy are not described. Second, all results are single-run percentages with no error bars or seeds. Third, the hyperparameters (top-k=50, silhouette=0.05, epochs=2) were selected on the same judge-based metric, so the reported performance is likely optimistic. Fourth, no code is released, making verification harder.\n\nNone of these are fatal in principle. The method is plausible and the efficiency gains are likely robust. But the central security claim needs independent validation: human audit, a vetted static analyzer, or at minimum a judge trained on a disjoint, disclosed dataset. I would send this to peer review — the idea is worth referee time — but the authors should be pushed to release code, report variance, and close the judge-overlap gap.","headline":"A neat, clearly-written neuron-selective fine-tuning pipeline, but every security number relies on a judge that may share data with the fine-tuning set, so the headline results are not yet established.","tokens_in":21021,"tokens_out":3810,"would_cite":false,"duration_ms":30903,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Security reasoning in code LLMs is concentrated in a small neuron subset, and tuning only that subset makes generated code secure by default.","keywords":["code LLM security","neuron-level fine-tuning","gradient-based attribution","security neuron identification","secure code generation by default","parameter-efficient fine-tuning","vulnerability reduction","vibe coding"],"falsifier":"Have independent security experts (or a vetted static analyzer) label the same generated-code samples that the judge scored, blinding them to which model produced each sample. If expert-labeled safe rates for GoodVibe-hardened and baseline models do not reproduce the reported gaps — roughly 35% to 87% on C++ and 59% to 76% on Java — the central claim fails.","tokens_in":20056,"feed_emoji":"🛡️","tokens_out":5262,"duration_ms":52636,"temperature":0.7,"pith_summary":"GoodVibe claims that insecure-by-default code generation is not a capability problem but a targeting problem: inside a pretrained code LLM, the reasoning that decides whether code is secure is concentrated in a small subset of neurons, so updating only those neurons changes default behavior. The method finds these neurons by turning security into a binary classification task, reading the gradient magnitude each neuron contributes to that loss, and fine-tuning only the top neurons per layer while freezing the rest. It then groups correlated neurons so they share an update direction, cutting trainable parameters to an average of 1.9 million. Across six models and four languages, this lifts average safe-response rates from 35.1% to 87.5% on C++ and from 59.3% to 76.0% on Java, matching or beating full fine-tuning while keeping utility loss to an average of 0.84% on reasoning benchmarks. If correct, the result matters because it offers a cheap, inspectable route to secure-by-default code in fast 'vibe coding' workflows where security prompts are absent.","feed_headline":"Tuning 1.9M parameters lifts secure code output to 87.5%","feed_subtitle":"GoodVibe adapts only security-critical neurons, matching full fine-tuning with over 4,700x fewer parameters.","key_machinery":"The load-bearing mechanism is a gradient-based neuron importance score: for each linear layer of the transformer, GoodVibe computes the mean absolute gradient of a binary security-classification loss with respect to each neuron (a hidden dimension of the layer), selects the top-k neurons per layer, and fine-tunes only those weight rows while freezing all others. Clustering is the efficiency device: neurons with similar gradient profiles are grouped with k-means and share one update vector, so the number of trainable parameters scales with the number of clusters rather than the number of selected neurons. The same mechanism doubles as a structural claim — security judgment is distributed but","core_discovery":"The paper argues that security-relevant reasoning in code language models is structurally localized: a small subset of neurons, spread across low-level memory handling and higher-level API and control-flow logic, carries most of the influence over whether generated code is secure. GoodVibe identifies these neurons with gradient-based attribution — it recasts security assessment as a binary token-prediction task and measures the average magnitude of the loss gradient with respect to each neuron — then fine-tunes only that security-critical subspace, freezing everything else. It further clusters neurons with similar gradient profiles so they share a single update direction, making training cos","pith_inferences":["A testable extension: re-score a held-out sample of generated code with independent human auditors or a production static analyzer; this would separate genuine vulnerability reduction from alignment with the paper's learned judge.","Because the same security-labeled dataset is used for neuron identification, fine-tuning, and judge training, distributional overlap could inflate the reported gains; the paper does not address this directly.","The low-dimensional security subspace suggests the recipe could transfer to other default behaviors — privacy, license compliance, or policy adherence — which the paper names as future work but does not demonstrate.","If localization generalizes, task-specific neuron subspaces might be composed in a single model without retraining, enabling modular, inspectable behavior control; the paper leaves composition untested."],"forward_implications":["Code LLMs can be made secure by default without full retraining, so security hardening becomes practical on modest compute for many open-weight models.","Because only a security-critical subspace changes, general coding and reasoning benchmarks stay within about one percent of the original model, mitigating the catastrophic forgetting often seen with full fine-tuning.","The gradient-attribution step identifies which internal units carry security reasoning, opening the door to inspecting and controlling security behavior instead of treating the model as a black box.","The method transfers to programming languages beyond the main experiments, indicating the identified subspace captures language-general security logic rather than dataset memorization.","Secure-by-default outputs reduce reliance on users writing security-specific prompts, which matters for informal 'vibe coding' workflows where prompts are short and security-agnostic."],"fun_headline_variants":["Neuron-selective tuning matches full finetune with 4,700x fewer params","GoodVibe: Security neurons tuned, training cost cut 3.6x vs LoRA","2.5x more secure code via neuron-level finetuning","Target 1.9M security neurons, boost code safety 2.5x","Security-by-vibe: Tune only safety neurons, match full finetune"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim rests on the learned judge model being a faithful measure of real code vulnerabilities; the paper itself calls the judge 'a consistent measurement instrument rather than an absolute oracle,' and no independent ground-truth evaluation is supplied.","fun_headline_variants_meta":{"raw":{"variants":["Neuron-selective tuning matches full finetune with 4,700x fewer params","GoodVibe: Security neurons tuned, training cost cut 3.6x vs LoRA","2.5x more secure code via neuron-level finetuning","Target 1.9M security neurons, boost code safety 2.5x","Security-by-vibe: Tune only safety neurons, match full finetune"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00155,"raw_usage":{"total_tokens":6062,"prompt_tokens":799,"completion_tokens":5263,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":5154}},"tokens_in":543,"tokens_out":5263,"duration_ms":37668,"temperature":1.0,"reasoning_tokens":5154,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T00:58:07.207043+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent security experts (or a vetted static analyzer) label the same generated-code samples that the judge scored, blinding them to which model produced each sample. If expert-labeled safe rates for GoodVibe-hardened and baseline models do not reproduce the reported gaps — roughly 35% to 87% on C++ and 59% to 76% on Java — the central claim fails.","supporting_citations":[],"review_version":1}