{"id":"eb210668-04be-496c-b93c-b635be78dc27","arxiv_id":"2601.03595","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SAE-Steering finds, via keyword-logit recall plus effectiveness ranking, sparse-autoencoder features that steer a reasoning model into a chosen reasoning strategy, beating baseline steering by ~15% on a judge-based metric and correcting 7% more wrong answers.","lead":"This paper introduces SAE-Steering, a two-stage pipeline that finds specific 'reasoning-strategy features' inside large reasoning models: first keep only the features that amplify strategy keywords, then test the survivors for real steering power on a small validation set. Injecting those features during generation measurably shifts the model's strategy, and re-steering a chain that already went wrong fixes about 7% more problems than simply letting the model keep thinking.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline +15% control-effectiveness margin may be an artifact of using the same LLM-judge objective for feature selection and evaluation; baselines are not selected on this metric, so the reported margin is not a fair comparison.","rationale":"The paper makes two central quantitative claims: a >15% improvement in control effectiveness (RQ1) and a 7% absolute accuracy improvement in error correction (RQ3). The reader's conditional verdict correctly identifies RQ1's control-effectiveness metric as the weakest link because the same LLM-judge criterion is used for feature selection in Stage 2 and for the headline evaluation in Table 1, with baselines receiving no analogous selection. This is an internal measurement issue rather than a disagreement with external consensus, and it directly affects the paper's first headline claim. The independent RQ3 final-answer accuracy signal is positive and more objective, which is why I do not recommend rejection. The matched-selection test is a single, concrete check that removes the selection asymmetry; if the margin disappears or shrinks materially, the abstract's first quantitative claim and the RQ1 conclusion would need significant qualification.","tokens_in":15191,"tokens_out":8059,"duration_ms":85014,"concrete_test":"Re-run the RQ1 comparison with a matched-selection protocol: for each baseline (Vector Steering, Think Intervention, Logit Boosting), build a candidate pool of ≥30 variants (e.g., different contrastive pair samples for Vector Steering, different prompt phrasings for Think Intervention, different keyword sets/weights for Logit Boosting), select the best variant on the 50-problem AIME validation set using the exact Stage-2 GPT-4o judge, then evaluate the selected variants on AIME'24/25 and GPQA with the 3-judge majority protocol used in Table 1. If SAE-Steering's average margin over the best-selected baseline drops below 15% (or below a bootstrap confidence bound excluding 15%), the reported margin is substantially selection alignment; if it survives, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central RQ1 claim—SAE-Steering outperforms existing methods by >15% in control effectiveness (Abstract/Table 1)—is evaluated and selected by the same LLM-judge protocol. Section 3.2 ranks candidate features by Effectiveness(f_j) (Eq. 7), which is the binary judge outcome in Figure 7. Section 4.1 then says control effectiveness is evaluated 'following the procedure described in Stage 2.' Stage 1 already filters candidates by whether they amplify strategy-keyword logits (top-10 tokens, keyword list in Appendix B), so the surviving candidates are enriched for features that raise the surface tokens the judge is likely to count. Baselines receive no analogous selection: Logit Boosting is a direct keyword booster, Think Intervention is a fixed prompt, Vector Steering is one contrastive vector. Thus part of the reported margin is selection alignment, not intrinsically better control. The mitigations are partial: human agreement 0.82 rests on only 200 samples (40 per strategy) with Krippendorff alpha 0.78, and the judge prompt explicitly instructs 'count the number of times the strategy appears', which can be satisfied by keyword-level surface behavior rather than structural strategy change (cf. Figure 3's Logit Boosting example, where the model says 'another' without changing strategy). If the judge is rewarding keyword frequency, Stage-1 recall and Stage-2 ranking will both exploit that, and the Table 1 margin will overstate real strategy control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAE-Steering, a two-stage method for identifying sparse-autoencoder features that control specific reasoning strategies in large reasoning models. Stage 1 recalls SAE features whose direct logit contribution amplifies strategy-specific keywords, filtering out over 99% of features; Stage 2 ranks the survivors by an LLM-judge-based control-effectiveness score on a validation set. The selected feature directions are then used as steering vectors during decoding. The authors claim that this method outperforms existing prompt- and activation-based baselines by over 15% in control effectiveness on AIME and GPQA, and that a strategy router built from these features improves error-correction accuracy by 7% absolute over Budget Forcing.","tokens_in":15415,"tokens_out":5023,"duration_ms":50904,"significance":"If the claims hold, the paper would be a useful contribution to interpretability and controllable reasoning: it proposes a practical way to search a large SAE feature space, releases code and data, and demonstrates a downstream application (error correction) with an objective accuracy-based metric. The RQ3 error-correction results are the strongest independent evidence, since they do not rely on the same LLM judge used for feature selection. However, the headline control-effectiveness comparison is currently not fully supported because the same judge-based metric is used both for feature selection and for Table 1 evaluation, and all quantitative claims lack uncertainty quantification. The core mechanism is plausible and the paper is internally consistent, but the main comparative claim needs stronger evidence.","major_comments":[{"comment":"The headline control-effectiveness comparison is not independent of the selection criterion. Stage 2 ranks features by Effectiveness(f_j), computed with the LLM judge protocol in Figure 7, and §4.1 states that evaluation 'follow[s] the procedure described in Stage 2.' Thus SAE-Steering is selecting features on the same judge-based metric that Table 1 reports, while the baselines receive no equivalent selection. The reported >15% margin therefore conflates method quality with selection alignment. I request an evaluation with a genuinely independent metric (e.g., human structural annotation or a judge not involved in selection), and/or a baseline that is also selected by the same Stage-2 procedure. The 0.82 human-agreement check on 200 samples is partial mitigation but does not remove the selection/evaluation overlap.","section":"§3.2, Eq. (7); §4.1; Table 1"},{"comment":"All headline numbers are point estimates without variance or significance testing. The sample sizes make this important: in Table 3, AIME25 R1-Llama-8B improves from 0.0123 (2/163) to 0.0552 (9/163); with 163 samples, a bootstrap CI would likely overlap substantially. Similarly, Table 2 precision values are single draws from a candidate pool. The paper should report confidence intervals or paired significance tests (e.g., bootstrap or McNemar) for the 15% control-effectiveness margin and the 7% error-correction improvement. Without this, the 'consistently outperforms' claim is not yet established.","section":"Tables 1–3; §4.4"},{"comment":"The Figure 7 judge prompt explicitly tells the model to 'count the number of times the target cognitive reasoning strategy appears,' and Stage 1 selects features that amplify the logits of strategy keywords. This creates a route by which the judge rewards keyword-frequency increases without genuine strategy restructuring. The Logit Boosting example in Figure 3 is reassuring but anecdotal. The paper should provide a quantitative analysis that separates surface keyword matching from structural strategy change, e.g., by measuring performance under an instruction that forbids counting keywords, or by ablating keyword-frequency-matched baselines.","section":"Figure 7 prompt; §3.2 Stage 1"}],"minor_comments":[{"comment":"The hyperparameters n=2, τ=0.1, top-10 tokens, T=512, and the α search starting at 15 are described but not analyzed for sensitivity. Since τ and the keyword list directly determine the candidate pool, a small sensitivity study would strengthen the robustness claims.","section":"§4.1; Appendix A"},{"comment":"The layer-wise analysis reports average control effectiveness of the top-3 features without error bars or per-layer variance. Also, the analysis is only on Qwen3-8B; the authors should state whether the same pattern is expected for R1-Llama-8B.","section":"§4.3, Figures 4–5"},{"comment":"The observation that GPQA is easier because reasoning length is shorter ('7k vs. 15k tokens') is not supported by any quantitative correlation. Consider reporting per-sample reasoning lengths or a stratified analysis.","section":"Table 1; §4.2"},{"comment":"Human agreement is assessed on 200 samples (40 per strategy) with Krippendorff's alpha 0.78. This is acceptable but not high; reporting per-strategy agreement with confidence intervals, and ideally a larger sample, would increase confidence in the judge-based evaluation.","section":"Appendix C"},{"comment":"The description of the baselines is brief. In particular, it is unclear whether Vector Steering's steering strength and injection layer were tuned with the same procedure as SAE-Steering, and where Think Intervention inserts its instructions. This is needed for a fair comparison.","section":"§4.1 baselines"}],"recommendation":"major_revision","confidential_remarks":"The paper has a plausible and useful core idea, and the RQ3 error-correction experiments give objective evidence that partially compensates for the judge-metric issue. The main blocker is the selection/evaluation overlap for RQ1/RQ2: the authors should provide an independent evaluation or equally-selected baselines, plus uncertainty quantification. If those additions are made, I would view the paper favorably for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper before you cite the +15% control-effectiveness number: SAE-Steering’s Stage 2 feature ranking and Table 1 evaluation use the same LLM-judge protocol, and the baselines get no analogous selection. So part of the margin is metric alignment, not intrinsically better control. Read the error-correction experiments (RQ3) for the cleaner evidence.\n\nWhat’s new: the logit-lens recall over SAE decoder columns (L = W_dec^T U) is a simple, cheap alternative to activation-based feature selection, and it beats ReasonScore on recall precision. The smart part is filtering first by keyword-logit contribution and then spending the expensive judge evaluations on a few dozen candidates. The error-correction router, trained with contrastive learning to pick a strategy feature at the point of a wrong answer, is a neat application and the 7% average gain over Budget Forcing is the most convincing result in the paper.\n\nThe soft spots are the ones the stress-test names, and they are real. Table 1’s effectiveness metric is the Stage 2 objective; Logit Boosting, Think Intervention, and Vector Steering are not selected on that metric. The judge prompt in Figure 7 asks to “count the number of times the strategy appears,” which invites keyword-frequency scoring, and Stage 1 deliberately retains features that amplify strategy keywords. So the +15% and the 0.61 vs 0.33 precision numbers are upper bounds on the method’s true advantage. The human-agreement check (0.82) is partial mitigation, but it is only 200 samples and doesn’t undo the circularity for the headline comparison. Tables 1 and 3 report point estimates without intervals or significance tests; several AIME deltas rest on tiny counts (e.g., ~2 vs ~9 of 163). The paper also gives a GitHub link but no commit hash, so the artifacts aren’t independently verifiable.\n\nThat said, the core idea is sound. Logit-based recall is a plausible and cheap heuristic, the two-stage design is practical, and the error-correction signal is objective and positive across two models and three datasets. This paper deserves a serious referee. A strong revision would decouple selection from evaluation (e.g., evaluate all baselines after the same Stage-2 selection), report error bars, and make the judge protocol less dependent on surface keywords.\n\nWho gets value: people building test-time interventions for reasoning models and anyone working on SAE-based steering. I’d bring it to reading group and would probably cite it after the metric concerns are addressed. My recommendation: send it to peer review, expect a moderate revision.","headline":"SAE-Steering has a genuinely useful idea and a real error-correction result, but the headline +15% advantage is partly selection-on-the-metric.","tokens_in":16070,"tokens_out":2264,"would_cite":true,"duration_ms":21357,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single sparse-autoencoder feature direction can act as a control vector that shifts a reasoning model into a chosen reasoning strategy, beating prompt- and vector-based baselines and correcting more already-wrong answers.","keywords":["sparse autoencoders","reasoning strategy control","activation steering","feature identification","large reasoning models","logit lens","test-time intervention","error correction"],"falsifier":"Take the released models and the five strategies, generate matched steered and unsteered continuations, and have independent human annotators mark concrete reasoning moves—backtracking to an earlier step, applying a second method, restating the problem—after stripping the strategy keywords from the texts. If the steered continuations do not contain more such moves than the baselines at a rate comparable to the reported control effectiveness, the central claim is false. Alternatively, run the error-correction comparison against unguided extended thinking with the router's strategy labels hidden","tokens_in":14941,"feed_emoji":"🧭","tokens_out":8612,"duration_ms":76634,"temperature":0.7,"pith_summary":"The paper seeks to make the reasoning strategies of large reasoning models—problem understanding, procedural planning, backtracking, multi-perspective verification, and hypothesis reasoning—controllable at fine grain. It claims that sparse autoencoders trained on the residual stream disentangle these strategies into separate feature directions, and that its two-stage search, SAE-Steering, can find one effective direction per strategy: first keep features that amplify strategy-keyword logits, filtering out over 99% of candidates, then rank survivors by how often an LLM judge says a steered continuation shows the target strategy more explicitly than an unsteered one. Steering with the top feature as a control vector is reported to outperform prompt-based and contrastive-vector baselines by more than 15% in control effectiveness. When a trained router picks among strategy features during extended reasoning after a wrong answer, the method corrects about 7% more errors than simply forcing further thinking. If true, this gives a cheap, test-time handle on how a reasoning model thinks.","feed_headline":"Steer reasoning with sparse features, beating baselines by 15%","feed_subtitle":"Hidden-state directions, found by a two-stage search, steer a model's next reasoning steps and correct more wrong answers.","key_machinery":"Sparse-autoencoder feature directions—the decoder columns of an SAE trained on residual-stream activations, each intended to be a monosemantic direction. One such direction serves as the control vector added to a middle-to-late layer. The other load-bearing mechanism is the two-stage identification pipeline: a logit-contribution matrix (decoder weights times the unembedding matrix) recalls features that amplify keyword logits, filtering out over 99% of features; then a validation-set intervention, scored by a judge, ranks the survivors by actual control effectiveness. This pipeline converts a search over tens of thousands of features into a search over a few dozen.","core_discovery":"The central claim is that strategy concepts in a reasoning model's hidden states are not hopelessly entangled: a sparse autoencoder can isolate directions that correspond to individual reasoning strategies, and injecting one such direction into the residual stream for a block of tokens reliably pushes subsequent reasoning toward that strategy. The identification method is deliberately two-stage: a logit-contribution readout filters the tens of thousands of SAE features down to a few dozen candidates that amplify strategy-specific keyword logits, and an intervention-based judge then ranks those candidates by how much more explicitly the steered continuation exhibits the target strategy compar","pith_inferences":["The authors leave implicit that the same recall-and-rank recipe could be reused for any controllable attribute—tone, length, factuality, cautiousness—wherever a keyword proxy and an intervention judge are available.","Because the judge metric is used both to select features and to measure success, an independent behavioral test (e.g., counting actual backtracking moves or verification steps after masking strategy words) would tell whether the control is substantive or partly surface-level.","A natural next step the paper does not test is steering at the very start of reasoning, or switching strategies mid-chain based on router confidence, which could turn error correction into a closed-loop controller."],"forward_implications":["One feature direction per strategy is sufficient to shift a reasoning trajectory into the target strategy, at least for the five strategies tested.","Strategy control can operate as a test-time intervention without retraining the reasoning model.","Features found on math reasoning transfer to out-of-domain science reasoning, suggesting the directions capture general strategy concepts.","Logit-based recall is a cheaper and more precise way to find steering-relevant SAE features than activation-strength-based selection.","Middle-to-late transformer layers are the right place to inject strategy control; shallow-layer features are scarce and weak."],"fun_headline_variants":["SAE directions steer LLM reasoning strategies","Sparse autoencoders give fine-grained control over reasoning","Two-stage SAE steering boosts reasoning control by 15%","Isolate reasoning strategies with sparse autoencoder guidance","Steer LLM reasoning with disentangled SAE features"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that an LLM judge's binary verdict—that a steered continuation 'more explicitly demonstrates' the target strategy than an unsteered one—measures actual strategy use; the same verdict is used both to select the feature and to report the headline effectiveness, so if the judge mainly keys on surface marker words like 'wait' or 'another approach', the reported gains could be largely cosmetic.","fun_headline_variants_meta":{"raw":{"variants":["SAE directions steer LLM reasoning strategies","Sparse autoencoders give fine-grained control over reasoning","Two-stage SAE steering boosts reasoning control by 15%","Isolate reasoning strategies with sparse autoencoder guidance","Steer LLM reasoning with disentangled SAE features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0006,"raw_usage":{"total_tokens":2644,"prompt_tokens":754,"completion_tokens":1890,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":1823}},"tokens_in":498,"tokens_out":1890,"duration_ms":13092,"temperature":1.0,"reasoning_tokens":1823,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T12:15:29.775690+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the released models and the five strategies, generate matched steered and unsteered continuations, and have independent human annotators mark concrete reasoning moves—backtracking to an earlier step, applying a second method, restating the problem—after stripping the strategy keywords from the texts. If the steered continuations do not contain more such moves than the baselines at a rate comparable to the reported control effectiveness, the central claim is false. Alternatively, run the error-correction comparison against unguided extended thinking with the router's strategy labels hidden","supporting_citations":[],"review_version":1}