{"id":"4e5709df-c584-4d67-9dfe-3d9c410a2a3a","arxiv_id":"2505.15038","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Scaling up the most discriminative sparse autoencoder latents before reconstructing hidden states improves LLM concept steering vectors built by linear probing and difference-in-mean.","lead":"This paper presents a preprocessing step that cleans the hidden states of a large language model before computing a steering vector: it amplifies the sparse autoencoder features that best separate positive from negative examples and rebuilds the hidden state from them. In tests on two open models, the approach improved steering success rates on most of six concepts, but the gains are measured on a small, heavily tuned evaluation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 4-16% gains are best-over-grid values on the same 50-prompt test set, with no validation or error bars; 'consistently improves' is contradicted by Gemma results in Table 3 (flat or negative on 2 of 6 concepts).","rationale":"The reader's formally stated weakest_assumption is the influence-score mechanism (Eqs. 1-3), but the more decisive vulnerability for the central claim is the evaluation protocol. The reader's own rationale already notes the hyperparameter selection and absence of error bars, and I agree those are the primary weaknesses. The mechanism concern is secondary because even an imperfect latent selection could produce genuine gains, whereas the reported magnitudes are currently unverifiable. The Gemma results provide a direct internal contradiction with the word 'consistently.' The issue is addressable by a proper validation split and significance testing, so the verdict stays CONDITIONAL rather than REJECT. I mark agreement as partial because the formally stated weakest_assumption differs from what I find most load-bearing, though the reader's narrative rationale overlaps substantially with mine.","tokens_in":15000,"tokens_out":5400,"duration_ms":42707,"concrete_test":"Reserve a held-out set from the 50 test prompts (e.g., 25 prompts) and select k, m, and alpha on the remaining 25 prompts, then report success rate on the held-out set for all six concepts and both models, using the same procedure for baselines. If the held-out improvement over baselines is no longer 4-16% or reverses sign, the reported gains are attributable to test-set selection. Additionally, compute bootstrap confidence intervals over the 50 prompts for the reported configurations; if the confidence intervals for the SDCV-minus-baseline difference include zero in more than a minority of concept-model pairs, the 'consistently improves' claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim is that SDCV 'consistently improves steering success rates by 4-16% across six concepts.' The support is Table 1, but the reported success rates are the best result over a very large hyperparameter sweep per concept: k in {50,100,1000,5000,10000,20000,50000}, m in {10,20,40,80,100}, and steering strength alpha in [0.3,0.8] with step 0.015 (34 values), i.e., 7*5*34=1190 configurations per concept for Llama, all evaluated on the same 50-prompt test set used for the final comparison. With 50 prompts, a 4% improvement is just 2 questions and a 16% improvement is 8 questions; the maximum over 1190 configurations can easily produce such gaps by chance. No validation split, error bars, or significance tests are reported, so the 4-16% range is not established as a real effect. Moreover, the 'consistently' qualifier is contradicted by the authors' own Gemma results (Table 3): for political speech, SDCV-enhanced difference-in-mean ties the baseline at 16%, and for Twitter sentiments it is worse (8% vs 10%). The central claim as stated is therefore not supported by the evidence as presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SDCV, a preprocessing step for concept-vector steering in LLMs. Given contrastive positive and negative samples, it maps hidden states through a pretrained sparse autoencoder (SAE), ranks latents by an influence score s = |a_p - a_n| / (sigma_p^2 + sigma_n^2), scales the top-k latent activations by a factor m, reconstructs the hidden states from the scaled activations, and then builds linear-probing or difference-in-mean concept vectors from the reconstructed states. The central empirical claim is that SDCV improves steering success rates by 4-16% relative to plain linear probing and difference-in-mean on six concepts while maintaining topic relevance. Experiments use Llama-3.1-8B and Gemma-2-2B with 50 open-ended test questions per concept and GPT-4.1-mini as an automated judge.","tokens_in":15268,"tokens_out":5470,"duration_ms":46499,"significance":"The idea of using SAE latent selection as a denoising step for concept vectors is timely and plausible, and a working version would be a useful plug-in module for existing steering methods. The paper also contains a good-faith attempt at mechanistic evidence: the counterfactual noise-injection experiment in Section 3.4 speaks directly to the claim that noise in non-selected latents hurts steering. The limitations paragraph is honest about the empirical nature of k, m, and alpha. However, the evidence as presented does not establish the headline claim. The reported gains are best-over-grid values on the same 50-prompt test set, with no held-out validation or statistical uncertainty, and the Gemma results in Table 3 explicitly contradict the word 'consistently' in the abstract. With Ntest = 50, a 4% improvement is two questions, so the claimed range is within the noise of the evaluation as currently reported.","major_comments":[{"comment":"As written, Eq. (4) defines h'_i <- SAE(hi) + epsilon, and the text states 'epsilon is the reconstruction error of original hi.' Appendix A, Eq. (12), defines the SAE reconstruction error by z = SAE(z) + epsilon. Taken literally, substituting epsilon = hi - SAE(hi) into Eq. (4) gives h'_i = hi, so the proposed denoising would have no effect at all. If the intended meaning is that epsilon is the residual from the original, unscaled SAE reconstruction and that this fixed residual is added to the scaled reconstruction, this must be stated explicitly with unambiguous notation; otherwise the central mechanism of the paper is undefined.","section":"Section 2.1, Eq. (4)"},{"comment":"The success rates in Table 1 are selected as the best result over the hyperparameter grid k in {50, 100, 1000, 5000, 10000, 20000, 50000}, m in {10, 20, 40, 80, 100}, and alpha in {0.3, ..., 0.8} with step 0.015, i.e., 1190 configurations per concept for Llama, with all configurations evaluated on the same 50-prompt test set used for the final comparison. With Ntest = 50, the reported 4-16% improvements correspond to 2-8 questions, and the maximum over 1190 configurations can easily produce such gaps by chance. The paper reports no validation split, no error bars, and no significance tests. To support the claim, the authors should either fix hyperparameters on a separate validation set and report test-set results once, or report distributions over configurations with bootstrap intervals and a paired significance test.","section":"Section 3.1 and Table 1"},{"comment":"The claim in the abstract and in Section 3.2 that SDCV 'consistently improves steering success rates by 4-16%' is contradicted by the authors' own Gemma-2-2B results. For political speech, both SDCV-enhanced linear probing and SDCV-enhanced difference-in-mean tie their baselines at 16%. For Twitter sentiments, SDCV-enhanced difference-in-mean is worse than the baseline (8% vs. 10%). Even if the intended primary claim is about Llama-3.1-8B, the wording of the abstract and Section 3.2 is too strong and should be revised to match the evidence, or the Gemma discrepancy should be explicitly explained rather than described as consistently comparable or better.","section":"Table 3 (Appendix G)"}],"minor_comments":[{"comment":"The 'emotion steering task' used in Section 3.3 and Figures 2 and 4 is not defined in the main text or Appendix B, and it does not appear among the six concepts in Table 1 or Table 3; please specify the dataset, its size, and the evaluation rubric so this analysis is reproducible.","section":"Section 3.3 and Figures 2 and 4"},{"comment":"The text says the score uses 'standard deviation' while Eq. (1) uses sigma^2_n + sigma^2_p, i.e., variances; please align the prose and the equation.","section":"Section 2.1, Eq. (1)"},{"comment":"The heading for the Twitter sentiments subsection repeats the corrigibility definition ('The desired steering behavior for corrigibility is that the model is willing to be changed by human user') and should be corrected.","section":"Appendix H.6"},{"comment":"The concept name 'Corrigibiity' is misspelled in the table header.","section":"Table 3"},{"comment":"There are two nearly identical entries for Elhage et al. on toy models of superposition (2022a and 2022b); please consolidate or distinguish them clearly.","section":"References"},{"comment":"The sentence 'the reconstruction error is implicitly captured through the difference from znew' is unclear because znew is defined only after the vector is added; please rewrite for readability.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"I agree with the stress-test concern: the main empirical claim is not supported as presented because the reported gains are selected over a large grid on the same 50-prompt test set, and the Gemma table contradicts 'consistently improves.' The Eq. (4) issue may be a notational slip rather than a fatal flaw, but it needs a prompt correction. The method itself is reasonable and the paper is within the journal's scope; the required fixes are empirical and presentational rather than a reworking of the core idea."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: SDCV is a new and sensible recipe — pick the SAE latents that best separate positive and negative examples, scale them up, reconstruct the hidden state, then build the concept vector from that. That specific pipeline is not in the cited literature, and the counterfactual noise experiment is a good idea; it gives some evidence that the benefit is due to noise removal rather than a scaling artifact. So the method deserves attention.\n\nThe problem is the evaluation. The reported 4-16% improvements are the best numbers over a grid of 7 k values, 5 m values, and 34 alpha values — 1190 configurations per concept — all evaluated on the same 50-prompt test set. With 50 prompts, 4% is two questions. The maximum over 1190 tries can easily produce that by chance. There's no validation split, no error bars, no significance test. And the 'consistently' in the abstract is contradicted by their own Gemma results: on political speech, SDCV difference-in-mean ties the baseline at 16%, and on Twitter sentiments it's worse (8% vs 10%). So the central claim as stated is not supported by what's shown.\n\nThere are a couple of other soft spots. They cite SAIF and CAA but don't compare against them, which leaves the marginal benefit over existing SAE-based steering unclear. And the influence score in Eq (1) is plausible but not independently validated — the only evidence it works is the steering numbers, which brings us back to the evaluation problem.\n\nNone of this is fatal. The recipe is simple and the idea is worth testing properly. If the authors add a held-out validation split, report error bars or significance tests, and show that the gains survive averaging over the hyperparameter grid instead of taking the max, this could be a solid contribution. The missing baselines are easy to add.\n\nWho's it for: people working on activation steering or representation engineering will want to know about this method, even if they shouldn't trust the headline numbers yet. I'd send it to peer review rather than desk-reject — the core idea deserves referee time — but as a reviewer I'd push for a real evaluation before accepting. I would not cite the current version in my own work.","headline":"A plausible SAE-denoising recipe for concept vectors, but the headline 4-16% gains are best-over-grid numbers on 50 prompts and don't hold on Gemma, so the claim is premature.","tokens_in":15844,"tokens_out":2330,"would_cite":false,"duration_ms":20361,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Selectively scaling the sparse-autoencoder latents that best separate positive from negative examples, then reconstructing hidden states, raises concept-steering success by 4–16% across six concepts without hurting topic relevance.","keywords":["concept steering","sparse autoencoders","denoising","representation engineering","linear probing","difference-in-mean","SAE latents","large language model interpretability"],"falsifier":"Run SDCV on the same six concepts but replace the influence-score ranking with a random permutation of latents, keeping k and m fixed. If random selection matches or beats top-k selection on steering success, the influence score is not the mechanism; if random selection also beats the raw baseline, the improvement comes from SAE reconstruction itself rather than discriminative denoising.","tokens_in":14732,"feed_emoji":"🎯","tokens_out":11775,"duration_ms":86110,"temperature":0.7,"pith_summary":"The paper proposes Sparse Autoencoder-Denoised Concept Vectors (SDCV), a way to clean the hidden representations used to build linear concept vectors for steering LLMs. Its central claim is that the noise in these representations—spurious features from the contrastive datasets—can be reduced by passing hidden states through a sparse autoencoder, scaling up only the handful of 'latents' (feature dimensions) that best distinguish positive from negative examples, and reconstructing from that modified code. When applied on top of linear probing and difference-in-mean, SDCV consistently improves steering success rates by 4–16% across six challenging concepts (sycophancy, corrigibility, myopic reward, Amazon reviews, political speech, Twitter sentiment) while keeping topic relevance scores roughly unchanged. If it holds, it offers a cheap, inference-time denoising module that makes existing concept-vector steering more robust.","feed_headline":"Sparse-autoencoder denoising lifts concept steering 4-16%","feed_subtitle":"Denoising concept vectors before steering preserves relevance while raising success across six concepts.","key_machinery":"The machinery is the SDCV pipeline around a pre-trained sparse autoencoder: (1) an influence score $s = |a_p(h) - a_n(h)| / (\\sigma_n^2 + \\sigma_p^2)$ that ranks SAE latents by how much their mean activation differs between the positive and negative sets relative to within-class variance; (2) top-$k$ selection of the highest-scoring latents, with the chosen latents' activations multiplied by a scale factor $m$ while all other latents are left unchanged; (3) SAE decoding of the modified activation vector, plus the reconstruction error, to produce a denoised hidden state from which the concept vector is derived. The design rests on the assumption that concept-relevant features are concentrated in a small set of latents whose class separation is large and whose variance is small, and that everything else is noise.","core_discovery":"SDCV's central claim is that concept-relevant signal and dataset noise live in different sparse-autoencoder latents, and that a simple selection rule can separate them. Using a contrastive dataset, the paper scores each SAE latent by the absolute difference in average activation between positive and negative samples, normalized by the within-class spread (equation 1). The top-k latents are then scaled by a factor m (equation 2) and fed through the SAE decoder (equation 3), with the original reconstruction error added back to form a denoised hidden state. Concept vectors built from these denoised states—by either a logistic-regression probe or difference-in-mean—steer the model better than vectors built from raw states. The paper further shows via counterfactual amplification of non-top-k ('noisy') latents that steering degrades as noise is added, supporting the denoising interpretation.","pith_inferences":["If the influence-score mechanism is right, the same top-k scaling could be applied to other SAE-based interventions—such as activation editing for truthfulness or debiasing—where the denoised hidden state might transfer.","The paper's six concepts are a convenience sample; a natural stress test is to see whether the 4–16% improvement survives on a broader, more adversarial set of concepts, especially ones where positive and negative sets are near-duplicates.","The nonlinear interaction between k and m suggests that replacing the hard top-k mask with a learned or continuous weighting of latents could yield further gains, at the cost of more hyperparameters.","Since SDCV requires a trained SAE for the target model, the results give a concrete motivation for training high-quality SAEs as infrastructure for controllable generation."],"forward_implications":["SDCV acts as a drop-in preprocessing module: it can be inserted before any existing concept-vector construction (linear probing or difference-in-mean) without changing the steering equation or requiring retraining of the language model.","For hard, high-level concepts where raw vectors barely steer (success rates of 6–18% on sycophancy, corrigibility, and myopic reward), SDCV raises success to 20–30% in the reported Llama-3.1-8B experiments.","The counterfactual noise-amplification experiments suggest that the non-top-k latents are a genuine source of steering failure: steering success drops as those latents are amplified, supporting the denoising interpretation.","The consistent gains on both Llama-3.1-8B and Gemma-2-2B indicate the denoising mechanism transfers across model families and SAE architectures, rather than being an artifact of a single SAE.","Because optimal k and scale factor are found empirically per concept, the method's practical adoption hinges on finding cheap selection rules for these two hyperparameters."],"supporting_citations":[{"why":"Supplies the SAE used for Llama-3.1-8B, whose latents and decoder are used to denoise hidden states.","marker":"(He et al., 2024)"},{"why":"Supplies the SAE used for Gemma-2-2B, enabling the cross-model generalization test.","marker":"(Lieberum et al., 2024)"},{"why":"Establishes the dictionary-learning view of SAEs that SDCV builds on for monosemantic feature decomposition.","marker":"(Bricken et al., 2023)"},{"why":"Introduces linear probes, one of the two concept-vector baselines that SDCV enhances.","marker":"Alain and Bengio (2016)"},{"why":"Introduces concept vectors via linear classifiers, the origin of the linear probing approach used here.","marker":"(Kim et al., 2018)"},{"why":"Provides the difference-in-mean baseline and the finding that simple baselines can outperform SAEs, which SDCV aims to beat.","marker":"(Wu et al., 2025)"},{"why":"Contributes the sycophancy, corrigibility, and myopic-reward steering datasets and the contrastive-activation baseline.","marker":"(Rimsky et al., 2023)"}],"fun_headline_variants":["SAE denoising separates signal from noise to steer LLMs better","Top-k SAE latents boost concept steering by up to 16%","Sparse autoencoders strip noise from concept vectors for sharper steering","Denoised concept vectors via SAE latents lift steering success 4-16%","Filtering SAE latents by discriminative power improves LLM steering"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the influence score—the gap in average activation between positive and negative examples, divided by within-class variance—correctly identifies the SAE latents that carry concept-relevant signal, so that scaling those latents up and reconstructing leaves the steering-relevant content intact while discarding noise.","fun_headline_variants_meta":{"raw":{"variants":["SAE denoising separates signal from noise to steer LLMs better","Top-k SAE latents boost concept steering by up to 16%","Sparse autoencoders strip noise from concept vectors for sharper steering","Denoised concept vectors via SAE latents lift steering success 4-16%","Filtering SAE latents by discriminative power improves LLM steering"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1196,"prompt_tokens":819,"completion_tokens":377,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":435,"completion_tokens_details":{"reasoning_tokens":277}},"tokens_in":435,"tokens_out":377,"duration_ms":4256,"temperature":1.0,"reasoning_tokens":277,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:24:21.040602+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SDCV on the same six concepts but replace the influence-score ranking with a random permutation of latents, keeping k and m fixed. If random selection matches or beats top-k selection on steering success, the influence score is not the mechanism; if random selection also beats the raw baseline, the improvement comes from SAE reconstruction itself rather than discriminative denoising.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces concept vectors via linear classifiers, the origin of the linear probing approach used here."}],"review_version":1}