{"id":"60b6fa7b-6d1b-4a71-93ea-e4a736d61a37","arxiv_id":"2605.31183","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Sparse autoencoders with supervised feature selection achieve near-LoRA performance on the AxBench steering benchmark and identify causal features.","lead":"This paper finds that sparse autoencoders can steer large language models nearly as well as LoRA fine-tuning when features are chosen and labeled via a supervised pipeline. A smart generalist might read it to see whether interpretability tools can become practical for controlling AI outputs instead of relying on opaque adaptation methods.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"Supervised pipeline for feature selection may embed task-specific bias, making SAE steering gains on AxBench non-comparable to unsupervised baselines","rationale":"The reader's weakest assumption is precisely the load-bearing point; the abstract supplies no further methodological detail that would falsify or strengthen it, so the concern remains the central one and the UNVERDICTED status is appropriate.","tokens_in":1702,"tokens_out":310,"duration_ms":11691,"concrete_test":"Re-run the full AxBench evaluation using only the interpretability-based subset of the pipeline (no task labels at any stage) on a held-out steering task or a different benchmark; if the gap to LoRA widens by more than the original margin, the supervised components are necessary for the reported performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result—that SAEs reach near-LoRA performance on AxBench once features are chosen and labeled via the authors' supervised pipeline—depends on the pipeline not leaking benchmark-specific information. If selection or labeling uses AxBench task labels (even indirectly) to rank or filter SAE features, then the reported causality and steering scores are no longer evidence that interpretability alone recovers causal directions; they become evidence that supervised feature selection works. The abstract's secondary claim (causality persists with only interpretability components) does not resolve this, because it still requires showing that those components were applied without any task-derived signal and that the resulting features generalize beyond the supervision distribution.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript claims that Sparse Autoencoders (SAEs) achieve steering performance on the AxBench benchmark close to the reference LoRA baseline when features are selected and labeled via the authors' supervised pipeline. It further claims that the interpretability-based components of this pipeline identify features that are causal for their assigned labels, and that high sparsity (low l0) is not required for effective interpretability-based steering, in contrast to prior results.","tokens_in":1834,"tokens_out":495,"duration_ms":21926,"significance":"If the supervised pipeline is shown to avoid task-specific leakage from AxBench, the result would be significant for LLM interpretability: it would indicate that SAEs can recover near-optimal steering directions via feature selection without full fine-tuning, and that interpretability tools alone can surface causal directions. The sparsity finding would also revise assumptions in the SAE steering literature.","major_comments":[{"comment":"The section describing the supervised pipeline for feature selection and labeling does not explicitly demonstrate that AxBench task labels or related signals are withheld during ranking or filtering of SAE features. This is load-bearing for the central performance-parity and causality claims, because any indirect use of benchmark labels would render the results evidence of supervised selection rather than unsupervised interpretability.","section":"supervised pipeline section"},{"comment":"The claim that 'causality persists with only its interpretability-based components' requires an ablation in which feature selection and labeling are performed entirely without any task-derived signal; the current description leaves open whether the labeling step uses AxBench labels, which would make the causality result circular with the supervision signal.","section":"interpretability components subsection"}],"minor_comments":[{"comment":"The abstract states performance is 'close to on par' with LoRA but supplies no numerical deltas, error bars, or dataset-split details; these should be added to the results section for reproducibility.","section":"Abstract and results tables"},{"comment":"The citations to Wu et al. (2025) and Wang et al. (2025) are used to frame the contribution; ensure the bibliography contains complete, consistent entries and that the comparison baselines are reproduced exactly as described in those works.","section":"Related work and experimental setup"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for highlighting the critical need to rule out task-specific leakage from AxBench in our supervised pipeline. This concern directly affects the strength of our performance and causality claims. We address both major comments below and will revise the manuscript to provide the requested explicit demonstrations and ablations.","responses":[{"response":"We agree that the current manuscript does not contain an explicit statement or diagram confirming that AxBench labels are withheld from the ranking and filtering stages. In the revision we will add a dedicated subsection (or expanded methods paragraph) that details the exact inputs to each step of the pipeline, states that no AxBench task labels or derived signals are provided to the ranking or filtering modules, and includes pseudocode showing the data flow. This will make clear that the supervision signal originates from separate, non-AxBench sources or from purely interpretability-derived statistics.","revision_made":"yes","referee_comment":"[supervised pipeline section] The section describing the supervised pipeline for feature selection and labeling does not explicitly demonstrate that AxBench task labels or related signals are withheld during ranking or filtering of SAE features. This is load-bearing for the central performance-parity and causality claims, because any indirect use of benchmark labels would render the results evidence of supervised selection rather than unsupervised interpretability."},{"response":"We acknowledge that the existing description of the interpretability-only ablation does not explicitly rule out any residual task-derived signal in the labeling step. We will revise the subsection to report a new (or more clearly documented) ablation in which both feature ranking and label assignment are performed using only activation statistics, reconstruction error, and semantic similarity measures with no access to any task labels whatsoever. The results of this ablation will be added to the main text or appendix, together with the corresponding causality metrics, so that the claim is supported by an experiment that matches the referee's specification.","revision_made":"yes","referee_comment":"[interpretability components subsection] The claim that 'causality persists with only its interpretability-based components' requires an ablation in which feature selection and labeling are performed entirely without any task-derived signal; the current description leaves open whether the labeling step uses AxBench labels, which would make the causality result circular with the supervision signal."}],"tokens_in":1324,"tokens_out":485,"duration_ms":14865,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The one thing to know is that this paper claims SAEs can match LoRA on the AxBench steering benchmark when features are picked and labeled through a supervised pipeline. The second is that the authors also report some causality from the interpretability parts alone and that sparsity may not matter as much as prior work suggested.\n\nThe work is new in showing these improved numbers with their pipeline and in the additional observations on causality and sparsity. It does a reasonable job of engaging with the prior benchmark result from Wu et al. and trying to explain why SAEs looked weak before.\n\nThe main soft spot is the one flagged in the stress test. The supervised pipeline for selecting and labeling features could easily introduce task-specific information from AxBench itself. If that happens, the performance comparison is no longer between unsupervised interpretability and baselines. It becomes supervised selection versus baselines. The abstract does not give enough detail on the pipeline to rule this out, and the claim that causality holds with only interpretability components still leaves open how those components were isolated from the supervision.\n\nThere are no error bars or dataset details mentioned, which makes it hard to assess how robust the numbers are. That said, the paper does not appear to have internal contradictions or obvious fitting issues.\n\nThis paper is for people in the LLM interpretability community who care about steering methods and SAE evaluations. A reader who wants to see how SAEs can be made competitive will find it useful. It deserves a serious referee because it addresses a concrete benchmark and offers a method that could be tested further.\n\nI would send it to peer review, but with instructions to reviewers to focus on whether the pipeline leaks benchmark information.","headline":"Supervised feature selection seems to drive the SAE performance gains on AxBench, so the interpretability story needs clearer separation from the supervision signal.","tokens_in":2314,"tokens_out":411,"would_cite":false,"duration_ms":22685,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Sparse Autoencoders match LoRA steering performance on AxBench when features are selected with a supervised pipeline.","keywords":["Sparse Autoencoders","LLM steering","AxBench benchmark","feature selection","model interpretability","LoRA","causal features"],"falsifier":"Measure AxBench steering performance after running feature selection with only the interpretability-based components of the pipeline and no supervised labeling step.","tokens_in":2597,"feed_emoji":"","tokens_out":572,"duration_ms":16257,"temperature":0.7,"pith_summary":"Earlier work found Sparse Autoencoders performed poorly for steering LLMs compared to simple baselines on the AxBench benchmark. This paper shows that SAEs reach performance close to the LoRA reference once features are chosen and labeled through the authors' supervised pipeline. The same pipeline identifies features with clear causal effects on their assigned labels even when supervision is removed from the selection step. Evidence also indicates that high sparsity levels are not required for steering success when selection relies on interpretability signals.","feed_headline":"SAEs match LoRA steering with supervised feature selection","feed_subtitle":"A pipeline that labels SAE features lets them reach near-reference performance on AxBench, revising earlier negative assessments.","key_machinery":"The supervised pipeline that selects and labels SAE features by combining task supervision with interpretability signals to extract causal directions from model activations.","core_discovery":"Sparse Autoencoders can perform close to on par with the reference LoRA performance on the AxBench benchmark when features are selected and labelled with the supervised pipeline. The pipeline selects features that are surprisingly causal of their identified labels when using only its interpretability-based components. High sparsity may not be crucial for successful steering based on interpretability.","pith_inferences":["The same selection approach could be tested on other steering benchmarks to check whether gains hold outside AxBench.","Removing the supervision component entirely might still yield usable features if interpretability metrics are strengthened.","Design efforts for future SAEs could de-emphasize extreme sparsity in favor of better downstream feature usability."],"forward_implications":["SAEs become competitive with parameter-efficient fine-tuning methods for steering once feature choice is addressed.","Interpretability signals alone can surface causally effective features for model interventions.","Steering performance does not depend on achieving the lowest possible l0 sparsity in the autoencoder.","Prior negative SAE steering results likely reflect suboptimal feature selection rather than a fundamental limitation."],"fun_headline_variants":["SAEs match LoRA with supervised feature selection","Supervised labels let SAEs reach LoRA on AxBench","SAEs near LoRA levels via supervised pipeline","Feature supervision matches SAEs to LoRA steering"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The supervised pipeline for feature selection and labeling does not introduce task-specific bias or overfitting that inflates steering performance on AxBench.","fun_headline_variants_meta":{"raw":{"variants":["SAEs match LoRA with supervised feature selection","Supervised labels let SAEs reach LoRA on AxBench","SAEs near LoRA levels via supervised pipeline","Feature supervision matches SAEs to LoRA steering"]},"model":"grok-4.3","cost_usd":0.004177,"raw_usage":{"total_tokens":2096,"prompt_tokens":635,"num_sources_used":0,"completion_tokens":60,"cost_in_usd_ticks":41774500,"prompt_tokens_details":{"text_tokens":635,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1401,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":635,"tokens_out":60,"duration_ms":9763,"temperature":1.0,"reasoning_tokens":1401,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T22:32:10.761634+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Measure AxBench steering performance after running feature selection with only the interpretability-based components of the pipeline and no supervised labeling step.","supporting_citations":[],"review_version":1}