{"id":"add1f1cd-5598-4c21-83c6-61627b44aee3","arxiv_id":"2506.18887","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"G-ACT improves per-layer probes for steering LLMs toward C++ code generation, yet the paper's main evidence is probe accuracy rather than actual output-language statistics.","lead":"Large language models have built-in preferences for programming languages, and this paper tries to override them by perturbing internal neuron activations. The authors propose a gradient-refined steering method, G-ACT, and report gains in probe classification accuracy for steering models toward C++, but direct measurement of generated language is largely missing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"G-ACT's headline claim is supported only by probe accuracy on training-derived cluster labels; output-language distribution under G-ACT is never reported, so the steering claim is unverified.","rationale":"The reader's weakest assumption is exactly the load-bearing gap: probe classification accuracy on cluster labels is used as the sole quantitative measure of steering success, with no reported output-language distribution for G-ACT. My reading of the full text confirms this: Sec. 4.3.1 defines the evaluation metric purely as accuracy/F1 against cluster IDs, explicitly separates probe quality from downstream generation effects, and the only G-ACT outputs are two qualitative snippets (Appendix A.5) with no distributional or correctness analysis. The abstract's 'reliably biases generation' is therefore not backed by the paper's numbers. I do not see a different concern that is more load-bearing. The static-method examples in Appendix A.4 are highly suggestive that 'CPP output' figures in the static-method analysis may include syntactically invalid code, which reinforces the need for a direct output-language measurement. Because the claimed contribution is a steering method that changes generated language, lack of any measured effect on generated language is a fundamental evidence gap, not a stylistic issue. The paper also reports a 70B result with mean accuracy 19.1% for G-ACT, which is far below chance for three clusters (33.3%); the conclusion's claim for 70B that 'targeted injections at key layers still improve language selection' is likewise not connected to any generation measurement. The reader's verdict of REJECT with high confidence is appropriate: the central claim is unverified, and the paper even lacks the code/data needed for an external check. My recommendation is unchanged.","tokens_in":21251,"tokens_out":1763,"duration_ms":15244,"concrete_test":"Re-run the G-ACT pipeline on the 84-prompt benchmark with Llama-3.2-3B-Instruct as described in Sec. 4.3, and measure the output-language distribution of the generated code (by first-token/markdown language tag plus a parser-based language detector) for G-ACT vs. standard ACT vs. vanilla. Report the fraction of outputs that are actually C++, the fraction that compile (e.g., with g++), and the per-problem shift relative to baseline. If G-ACT increases the fraction of valid C++ outputs over both baselines, the central claim is supported; if not, probe accuracy is an unreliable proxy.","verdict_should_be":"REJECT","load_bearing_attack":"The abstract and conclusion claim that G-ACT 'reliably biases generation toward CPP,' but the only quantitative evidence is probe classification accuracy (Sec. 4.3.1, Fig. 11): agreement of per-layer probes with K-means cluster IDs derived from CPP-vs-Python activation differences on the training set. The paper itself states this metric 'isolates probe quality from downstream generation effects' (Sec. 4.3.1), and throughout the evaluation, accuracy and macro-F1 are computed against cluster labels, not against the language of generated code. No quantitative distribution of generated languages under G-ACT is reported; the only G-ACT outputs shown (Appendix A.5) are two hand-picked, plausibly non-representative snippets. The conclusion's statement that 'even imperfect probes suffice to steer the model output toward the desired style, yielding substantial qualitative gains in CPP code generation' is an assertion, not a measured result. Meanwhile, the static-method examples in Appendix A.4, which the paper itself discusses as evidence of language shift, contain syntactically invalid C++ (e.g., 'int N phonons = -1', 'double N_Bohr_Q *= 0', truncated expressions like '1 +orus', 'std::sqrt(omega\\ni))'). This raises the possibility that 'CPP output' is counted merely by the presence of '#include' or markdown fences, not by valid C++. If the probe-accuracy gains do not translate into measured increases in C++ output (or in compilable C++), the central claim of reliable language steering is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether activating latent subspaces in LLMs can steer scientific code generation toward the C++/CPP language. The authors first evaluate five instruction-tuned LLMs on a curated set of 84 scientific coding prompts and report per-model language biases. They then present a static neuron-attribution method that locates and amplifies a single MLP neuron associated with the CPP token, and show that this can shift language selection and code output for LLaMA-3.2-3B, while also noting its fragility across models and prompts. The main contribution is G-ACT (Gradient-Refined Adaptive Activation Steering), which clusters per-prompt CPP-versus-Python activation differences into a small set of steering directions and trains lightweight per-layer probes to select among these directions during inference, refining the probes online with a cross-entropy loss. The quantitative evaluation of G-ACT measures probe classification accuracy and macro-F1 against cluster labels derived from K-means on the training set. Reported results include an average probe-accuracy increase from 0.405 to 0.556 for LLaMA-3.2-3B and an increase from 0.063 to 0.191 for LLaMA-3.3-70B. The paper concludes that G-ACT reliably biases generation toward CPP.","tokens_in":21684,"tokens_out":8476,"duration_ms":73366,"significance":"If the central claim were supported, the work would be relevant to mechanistic interpretability and practical agentic code generation: an inference-time, model-agnostic method for controlling output language with modest overhead would be a useful contribution. The idea of refining lightweight probes online during generation is interesting and the authors provide a detailed algorithmic description. However, the paper's end-to-end claim is not supported by its evidence. The primary metric measures probe agreement with K-means cluster labels, not any property of the generated text, so the reported accuracy numbers do not establish that G-ACT changes the language of generated code. No code, dataset, or systematic output analysis is provided; the only generation examples are two hand-picked snippets in Appendix A.5 with an unquantified assertion of reliability. These deficiencies are load-bearing because they leave the paper's stated contribution—reliable language steering—unverified.","major_comments":[{"comment":"The headline claim that G-ACT 'reliably biases generation towards the CPP language' is supported only by probe classification accuracy and macro-F1 computed against K-means cluster labels, not by any measurement of the language of generated outputs. The paper explicitly states that this metric 'isolates probe quality from downstream generation effects,' which concedes that it does not measure generation. No quantitative language distribution of G-ACT-generated code is reported; Appendix A.5 shows two hand-picked outputs and asserts 'only CPP implementations' without supporting numbers. Therefore the abstract's central claim is unsupported by the evidence presented.","section":"Sec. 4.3.1, Fig. 11, Abstract"},{"comment":"The probe evaluation is circular with respect to the construction of the steering vectors. The 'true' cluster labels are the cluster IDs produced by the same K-means procedure applied to the same CPP-versus-Python activation differences used to build the steering centroids, and for test prompts the true label is the nearest-centroid assignment of the averaged difference vector. Accuracy therefore measures how well the linear probe reproduces the K-means assignment on held-out points—a self-consistency check—rather than how well the steering changes model behavior. The reported +15% average accuracy improvement is an internal consistency measure, not an external validation of steering efficacy.","section":"Sec. 4.3.1, evaluation protocol"},{"comment":"For LLaMA-3.3-70B, the paper reports G-ACT probe accuracy of 19.1% with C=3 clusters, which is below the 33.3% random baseline. The conclusion nevertheless states that 'targeted injections at key layers can reliably bias generation toward CPP despite overall weak activations.' A below-chance score on the method's own proxy metric cannot support a claim of reliable language steering; the 70B results actually undermine the generalizability claim made in the abstract and conclusion.","section":"Sec. 4.3.1, Fig. 11B, Conclusion"},{"comment":"The static method is argued to override language choice 'while maintaining functional correctness,' but the C++ example in Appendix A.4 contains syntax errors and undefined identifiers (e.g., 'int N phonons = -1', 'double N_Bohr_Q *= 0', 'std::sqrt(omega\\ni))'), and Sec. 4.2.2 itself states that 'comprehensive validation of its correctness and performance has not been conducted and is beyond the scope of this study.' The functional-correctness claim is therefore contradicted by the paper's own evidence, and the qualitative examples cannot be taken as demonstrations of valid C++ generation.","section":"Sec. 4.2.2 and Appendix A.4"}],"minor_comments":[{"comment":"The phrase 'improving the probe classification accuracy by 61.5%' is ambiguous; the body reports an increase from 0% to 61.5% (Sec. 4.3.1), so the abstract should state absolute percentage-point improvement rather than a relative percentage.","section":"Abstract"},{"comment":"Section 3.3 mentions an 80/20 train/test split, while Section 4.3 uses a 70/30 split; these numbers should be reconciled.","section":"Sec. 3.3 vs Sec. 4.3"},{"comment":"The caption of Figure 16 says 'colored by four K-means clusters,' but the surrounding text and Section 4.3.1 state that the 70B model uses three clusters; the caption and text should agree.","section":"Fig. 16 caption"},{"comment":"The magnitude of the static neuron perturbation is never reported, although the text advises tuning it; this omission prevents replication of the static-method experiments.","section":"Sec. 4.2.2"},{"comment":"The claim of 'reproducible transformation matrices' is not substantiated: no code, matrices, or other artifacts are made available, so the reproducibility guarantee is unverifiable.","section":"Contribution 3, Conclusion"},{"comment":"The G-ACT example for LLaMA-3.2-3B (P1) also appears to contain C++ errors, such as passing a raw pointer to a function that expects std::vector and an orphaned 'std::endl;' statement; describing these outputs as generated 'CPP implementations' should be qualified, since they may not compile.","section":"Appendix A.5"},{"comment":"The text says Llama-3.2-3B 'defaults strongly to Java (76.2%)' in the aggregate results, while the discussion of Figure 2 states that Python predominates with 'roughly 70–80% of outputs'; this apparent contradiction in the baseline characterization should be clarified.","section":"Sec. 4.1 vs Fig. 4"}],"recommendation":"reject","confidential_remarks":"The paper's central claim of reliable CPP steering rests on a circular proxy metric: probe accuracy against cluster labels derived from the same data used to construct the steering vectors. The 70B results are below chance, and the provided generation examples do not demonstrate valid C++. The baseline characterization also contains inconsistencies. In my assessment, the missing output-language evaluation is not a local fix but a fundamental deficiency; the authors would need to add a systematic study of generated code under G-ACT, which goes beyond a normal revision. I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's headline claim — that G-ACT reliably steers LLaMA toward C++ — is supported only by probe accuracy on cluster labels the authors define themselves, not by any measurement of the language of generated code. That is a real gap, but the paper is not empty: the gradient-refined probe idea is a legitimate variation on ACT, and the baseline language-preference study across five models is careful, with sampling checks and honest reporting of the static method's limits.\n\nWhat is new: applying activation steering to programming-language selection in scientific code generation, and the specific gradient-refinement of ACT probes during autoregressive inference. The authors also report inference timings and discuss the trade-off honestly. The static neuron-attribution experiments, while called brittle, do show large output-language shifts (Figs 6, 7, 9), which is the kind of evidence the G-ACT section should have repeated but didn't.\n\nThe soft spot is load-bearing. Section 4.3.1 reports probe classification accuracy and macro-F1 against cluster IDs produced by K-means on the same CPP-minus-Python activation differences used to build the steering vectors. The paper even says this \"isolates probe quality from downstream generation effects\" — which is exactly why it cannot support the abstract's claim of reliable generation bias. No quantitative distribution of output languages under G-ACT is reported. The only G-ACT examples in Appendix A.5 are two hand-picked snippets, and one of them (the Gauss-Seidel template) is not valid C++: std::endl is used in a void function without a stream, x[j] is read before initialization for j=n-1, and A[j*(j+1)] is nonsense. The static-method examples in A.4 are worse: 'int N phonons' and 'double N_Bohr_Q *= 0' do not compile, plus truncated tokens like '1 +orus'. So the paper itself contains evidence that \"CPP output\" is sometimes just a fence or an #include, not compilable C++.\n\nThe citation pattern is fine; the method builds on ACT and related work properly. The free parameters are all explicit. The benchmark is not released, which makes replication harder.\n\nWho this is for: people working on activation steering for style control in code models would get a clear description of a plausible variant and a cautionary example of evaluation mismatch. I would not cite it in my own work as evidence of effective steering. But the idea deserves referee time: the fix is straightforward — measure the language distribution of generated outputs under G-ACT, ideally with a compiler check, and compare against baseline. I'd send it to peer review with that as the required revision.","headline":"A plausible ACT variant for code-language steering, but the central claim rests on probe accuracy against self-defined cluster labels, not on the language of generated code.","tokens_in":22062,"tokens_out":2203,"would_cite":false,"duration_ms":22406,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"G-ACT steers LLM code generation toward C++ by clustering CPP-minus-Python activation differences and using refined per-layer probes to pick the steering vector to inject.","keywords":["Activation Steering","Latent Space Activation","Activation Patching","Neuron Attribution","Privileged Basis","Code Generation","Reproducibility","Scientific Code Generation"],"falsifier":"Run G-ACT on the same 84-prompt benchmark and count the language of each generated code block, for example by markdown fence or compiler check. If the C++ share of outputs does not rise substantially above the unsteered baseline despite the reported probe-accuracy gains, then probe accuracy is not a valid proxy for steering and the central claim fails.","tokens_in":21045,"feed_emoji":"🎯","tokens_out":8187,"duration_ms":77459,"temperature":0.7,"pith_summary":"This paper sets out to show that an LLM's code-generation language choice can be steered at inference time by activating latent subspaces rather than by fine-tuning. It first documents strong, model-specific language biases on 84 scientific coding prompts across five LLMs. It then shows that perturbing a single neuron, found by decoding MLP weights through the LM head, can flip output toward C++, but that this static edit is brittle across prompt styles and model scales. To fix that, the paper introduces G-ACT: per-prompt CPP-minus-Python activation differences are clustered into a few steering directions, and lightweight per-layer probes, refined online by gradient descent on cross-entropy loss, pick which direction to inject at each layer. In LLaMA-3.2 3B the probes' average classification accuracy rises from 0.405 to 0.556, with early layers (0–6) jumping from 0% to 61.5%; the paper takes this as evidence that the model can be consistently biased toward C++.","feed_headline":"Gradient-refined probes lift LLM C++ steering accuracy 15%","feed_subtitle":"Cluster-and-inject steering biases scientific code toward a chosen language without fine-tuning, at modest inference cost.","key_machinery":"The load-bearing mechanism is the per-layer steering-vector probe trained and refined inside G-ACT. The construction starts with per-prompt difference vectors $\\Delta_{i,\\ell}=h^+_{i,\\ell}-h^-_{i,\\ell}$ between CPP- and Python-conditioned activations at each layer; these are clustered with K-means into $C$ centroids $\\{c_{k,\\ell}\\}$. A linear probe $\\pi_\\ell: \\mathbb{R}^D \\to \\mathbb{R}^C$ at each layer predicts the cluster label of the current activation, and the selected centroid is injected into the residual stream as $h_{i,\\ell} \\leftarrow h_{i,\\ell} + \\alpha c_{j,\\ell}$. The refinement stage runs autoregressive inference under gradient tracking and backpropagates cross-entropy loss only into the probe parameters, leaving the pretrained model frozen. The static method it replaces decodes effective neuron weights $W_{\\mathrm{eff}} = W_{\\mathrm{up}} \\odot \\sigma(W_{\\mathrm{gate}})$ through the LM head to find a single most-CPP-correlated neuron, then amplifies that neuron's activation.","core_discovery":"G-ACT is the paper's central proposal. For each prompt, the model is run with a CPP-requesting answer and a Python-requesting answer, and the per-layer activation differences are stored. These difference vectors are clustered with K-means; each centroid becomes a steering vector. At generation time, a per-layer linear probe classifies the current activation into a cluster and the corresponding centroid is added to the residual stream. Unlike standard ACT, the probes are placed inside an autoregressive forward pass under gradient tracking and refined by backpropagating cross-entropy loss only through probe parameters, leaving the base model's weights fixed. On LLaMA-3.2 3B this raises average probe accuracy from 0.405 to 0.556 and macro-F1 from 0.165 to 0.238; on early layers 0–6 accuracy goes from 0 to 61.5%. On LLaMA-3.3 70B, where attention-head signals are diffuse, accuracy improves from 6.3% to 19.1%, and the paper recommends hidden-state difference vectors in that regime. The paper also reports that across repeated trials G-ACT produces only C++ implementations on two example prompts, while explicitly leaving a full validation of generated-code correctness out of scope.","pith_inferences":["If probe accuracy does track output language, the same cluster-and-inject recipe should transfer to other style axes, such as Python versus Julia or code verbosity, since no part of the difference-vector construction is CPP-specific.","The paper does not verify that the generated C++ compiles or runs; a natural extension would be to compile and execute the steered outputs on the same benchmark.","Cluster labels come from K-means, so reported accuracy measures agreement with an unsupervised partition; if clusters do not align with language choice, accuracy gains could occur without language steering, and a direct output-language measurement would settle this.","Activation steering may offer an alternative to prompt-based or retrieval-based interventions in scientific-coding agents, avoiding the repeated-token and attention-sink failure modes the paper mentions for long agentic interactions, though this is not tested here."],"forward_implications":["If G-ACT works as claimed, LLM-based scientific-code agents can bias their output toward C++ or another target language at inference time, without fine-tuning or changing the prompt.","The early-layer result (0–6: 0% to 61.5% probe accuracy) suggests that gradient refinement can make previously uninformative layers usable for steering, widening the set of layers available for targeted control.","Because only probe parameters are updated, the base model's weights remain fixed, which should preserve general capabilities while adding a controllable style direction.","Steering only a subset of layers keeps the runtime overhead near 1.3–1.4x base generation, making the method practical for agentic systems that generate many code snippets.","The persistent steering matrices embedded by G-ACT are presented as a route to reproducible model behavior across deployments."],"supporting_citations":[{"why":"Supplies the base Adaptive Activation Steering framework (per-prompt difference vectors clustered into centroids, per-layer probes) that G-ACT extends with gradient-refined probes.","marker":"[Wang et al., 2025]"},{"why":"Provides the privileged-basis argument that residual-stream coordinates are approximately disentangled, justifying steering along single activation axes.","marker":"[Nelson Elhage and Olah, 2023]"},{"why":"Motivates decoding neuron weights into token-level probability distributions to find concept-correlated neurons, the idea behind the static attribution method.","marker":"[Templeton et al., 2024]"},{"why":"Supports the claim that neurons aligned with features via high cosine similarity can be located and edited, which underlies the static neuron-perturbation experiments.","marker":"[Meng et al., 2022]"},{"why":"Supplies the SciCode corpus from which the 84 scientific coding challenge prompts are inspired.","marker":"[Tian et al., 2024]"},{"why":"Frames the native MLP activation axes and activation-patching methodology used to justify steering along single coordinates.","marker":"[Zhang and Nanda, 2024]"},{"why":"Demonstrates decoding neuron weights through the LM head to identify specialised feature neurons, the core operation of the static method.","marker":"[Davies, 2025]"}],"fun_headline_variants":["Adaptive probes steer LLM code bias to C++ without fine-tuning","G-ACT lifts early-layer C++ steering accuracy to 61.5%","Cluster-and-inject steering biases scientific code generation toward C++","Gradient-refined probes boost LLM language selection by 15%","Steer LLMs to write C++ with per-layer adaptive steering vectors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that probe classification accuracy—how often a probe's predicted cluster label matches the training-set cluster assignment—is a valid proxy for actually steering generated output into the target programming language, and the paper does not report a quantitative language distribution of G-ACT's outputs to test this.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive probes steer LLM code bias to C++ without fine-tuning","G-ACT lifts early-layer C++ steering accuracy to 61.5%","Cluster-and-inject steering biases scientific code generation toward C++","Gradient-refined probes boost LLM language selection by 15%","Steer LLMs to write C++ with per-layer adaptive steering vectors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1422,"prompt_tokens":1050,"completion_tokens":372,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":276}},"tokens_in":666,"tokens_out":372,"duration_ms":3980,"temperature":1.0,"reasoning_tokens":276,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:40:35.244200+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run G-ACT on the same 84-prompt benchmark and count the language of each generated code block, for example by markdown fence or compiler check. If the C++ share of outputs does not rise substantially above the unsteered baseline despite the reported probe-accuracy gains, then probe accuracy is not a valid proxy for steering and the central claim fails.","supporting_citations":[{"cited_title":"A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6):186345, 2024a","cited_arxiv_id":null,"evidence_quote":"Supplies the base Adaptive Activation Steering framework (per-prompt difference vectors clustered into centroids, per-layer probes) that G-ACT extends with gradient-refined probes."}],"review_version":1}