{"id":"904d839c-f0a4-48c5-99f8-94115165a479","arxiv_id":"2608.04904","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Steering the top SAE features selected by target-language contrast improves multilingual benchmark accuracy in Gemma-3-12B-it without parameter updates.","lead":"The paper proposes a training-free way to strengthen target-language features in a multilingual LLM using sparse autoencoders, reporting accuracy gains on three benchmarks. The method could make multilingual adaptation cheaper, though the reported headline gains are inconsistent across the abstract and the main text.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported effect sizes are internally inconsistent: abstract, Table 1, and Appendix G give different gains, so the central empirical claim is not currently verifiable.","rationale":"I read the paper in good faith. The method is plausible and has useful controls: hidden-state scaling stays near baseline, direct hidden-dimension steering is weak, and the English-reference variants behave similarly to the multilingual-reference variants, suggesting the effect is not simply activation magnitude or English alignment. The code is released, and the limitation section candidly notes the dependence on model-matched SAEs. These are real strengths. However, the central claim is a quantitative empirical claim, and the paper's own numbers contradict one another. The reader's chosen weakest assumption, the language-neutral multilingual centroid of Eq. 5-6, is a legitimate methodological concern: with only 3-8 reference languages and mean-pooling over tokens, the centroid may not fully cancel language-specific or syntactic variation, so the selected features might not be purely 'target-language.' But that concern is secondary to the fact that the reported evidence for the headline gains is internally inconsistent. Even if the centroid assumption is imperfect, the method could still improve accuracy; conversely, if the headline gains cannot be reproduced or traced, the claim fails regardless of the centroid's validity. I therefore recommend a CONDITIONAL verdict: revise the manuscript so that the abstract, Section 5, Appendix G, and all figures report a single, traceable set of results, and provide the reproducibility check above. This is not an accusation of dishonesty; the discrepancies may stem from versioning or an averaging mistake, but they are load-bearing because they directly undermine the empirical support for the paper's contribution.","tokens_in":12813,"tokens_out":4221,"duration_ms":44885,"concrete_test":"Run the released pipeline (https://github.com/HungsingWong/sae-language-steering) with the settings described in Section 4 (Gemma-3-12B-it, Gemma Scope 2 SAEs, layer 47, top-3 features, FLORES-200 with N=500, validation-selected alpha) and reproduce the test-set averages. Then compute MGSM averages over (a) the four languages in Table 1 (de, es, fr, ja) and (b) the three languages named in Appendix G (de, es, fr), and check which configuration yields 80.5/81.1, which yields 81.7/83.6, and which yields the abstract's +1.9 gain. If only one configuration reproduces one set of numbers, identify which tables and which appendix text are stale, and report the corrected headline. This single check settles whether the discrepancy is a reporting error or a substantive difference in experimental setup.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is empirical: adding a target-language SAE steering vector at layer 47 improves held-out accuracy on XCOPA, XNLI, and MGSM. But the paper reports at least three incompatible sets of headline numbers. The abstract claims average gains of +10.9 (XCOPA), +5.3 (XNLI), and +1.9 (MGSM). Section 5 and Table 1 report +11.3, +4.1, and +0.6. Appendix G adds a third MGSM story: the text says 'our target-language signal achieves the highest accuracy of 83.6, compared with the 81.7 baseline' (gain +1.9), while Figure 10's labels show 80.5 and 81.1 (gain +0.6, matching Table 1). The Appendix G caption also says the MGSM average is over German, Spanish, and French, whereas Table 1 averages over four languages including Japanese; neither 83.6 nor 81.7 matches the three-language averages (80.3 baseline and 81.3 steered, calculated from Table 1). These are not rounding artifacts: the discrepancies are 0.4-1.9 percentage points. Because the contribution is precisely a quantitative improvement claim, and because layer and alpha are selected on validation sets, every reported number must be traceable to one configuration. As written, the reader cannot tell which configuration produced the advertised gains, so the central claim is unsupported by the manuscript's own evidence. This is a correctness/reporting risk, not an accusation of fraud; it could be a versioning error, but it must be resolved before the claim can be evaluated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an inference-time method for improving multilingual task performance in LLMs. Using frozen layer-specific sparse autoencoders (SAEs) from Gemma Scope 2, the authors identify the top-k SAE features whose activations differ most between a target language and a multilingual centroid (or an English reference) computed from parallel FLORES-200 sentences. These features are decoded into a sparse steering vector and added to the hidden state at the final prompt position of a chosen layer. Experiments on Gemma-3-12B-it report average held-out accuracy gains on XCOPA, XNLI, and MGSM, with ablations comparing multilingual versus English references, language-only versus full-representation signals, and non-SAE controls. The paper claims the method requires no parameter updates and improves multilingual performance across all three benchmarks.","tokens_in":13171,"tokens_out":6323,"duration_ms":64793,"significance":"If the reported results are accurate, the method is a lightweight, no-training alternative to multilingual adaptation that leverages SAE feature interpretability for downstream task performance rather than only language generation. The paper has concrete strengths: steering coefficients are selected on validation splits before held-out evaluation, a shared-coefficient robustness check is included, several control interventions (hidden-state scaling, direct hidden-dimension steering, least-squares English projection) are compared, and the code is publicly released. The central empirical claim is not obviously circular, since the steering vectors are computed from FLORES-200 parallel sentences and the test sets are held out. However, the manuscript currently contains mutually inconsistent headline numbers across the abstract, Section 5, Table 1, and Appendix G, so the exact effect sizes claimed by the paper cannot be verified as written.","major_comments":[{"comment":"The paper's central quantitative claims are internally inconsistent. The abstract and the introduction report average gains of +10.9 on XCOPA, +5.3 on XNLI, and +1.9 on MGSM; Section 5 and Table 1 report +11.3, +4.1, and +0.6 for the same settings. Appendix G, in turn, states for MGSM that the target-language signal achieves 83.6 against an 81.7 baseline, a gain of 1.9 that matches neither the Table 1 values (80.5 to 81.1) nor the three-language averages computed from Table 1 (80.3 to 81.3). Because the contribution is precisely a quantitative improvement claim, every reported number must be traceable to one experimental configuration. The authors must reconcile these discrepancies or clearly identify which configuration underlies each table and figure; as written, the reader cannot determine which results are being claimed.","section":"Abstract and Section 5 versus Table 1 and Appendix G"},{"comment":"The Appendix G MGSM text says that 'all intervention settings improve over the baseline,' but Figure 10 shows the English-relative language-only strategy at 78.2 and the English full-representation strategy at 79.1, both below the printed baseline of 80.5 (and also below the three-language average of 80.3). The caption states that the averages are over German, Spanish, and French, yet the displayed values match a four-language average including Japanese. This is not a typo in one digit: the text, table, and figure each tell a different story about the MGSM ablation. These numbers must be aligned before the paper's empirical claims can be assessed.","section":"Appendix G, MGSM paragraph and Figure 10"},{"comment":"The mechanistic interpretation of the method rests on the assumption that the multilingual centroid approximates language-neutral semantic content, so that the target-minus-centroid difference isolates target-language features. This assumption is not directly validated. The parallel-sentence control reduces but does not eliminate the possibility that the top features capture translation residuals or third-language signal, especially because the centroid averages over only the eight or three non-target languages used in each benchmark and because the selected features are identified by activation contrast alone. I recommend adding a control that disrupts the language pairing (for example, computing the centroid from randomly mismatched parallel sentences and comparing the selected features and downstream gains), or otherwise providing direct evidence that the selected features are language-discriminative rather than semantic. This would materially strengthen the paper's central interpretation that the steering signal is a target-language feature, rather than merely a beneficial hidden-state perturbation.","section":"Section 3.2, Equations (5)-(6)"}],"minor_comments":[{"comment":"The text in Section 4.3 says the feature-identification procedure is described in Section 3.3, but the procedure is in Section 3.2; the cross-reference should be corrected.","section":"Section 4.3 and Section 3.2"},{"comment":"Appendix B presents a layer-20 analysis in Figure 5 but concludes that the effect is 'not limited to layer 30'; the layer number cited in the sentence should match the figure (either layer 20 or an additional layer-30 figure should be provided).","section":"Appendix B"},{"comment":"The Figure 10 caption says the MGSM averages are over German, Spanish, and French, but the numbers shown match the four-language averages in Table 1, which include Japanese; the caption and the underlying data should be made consistent.","section":"Appendix G, MGSM caption"},{"comment":"Several references contain spacing artifacts (e.g., 'V os' in the Shi et al. entry and 'Cideron, Jean bastien' in the Gemma Team entry); these should be cleaned up before final submission.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The numerical inconsistencies between the abstract, Section 5, Table 1, and Appendix G are the main obstacle to publication; they appear to be a versioning or reporting error rather than evidence of bad faith, but they must be fully resolved before the paper can be evaluated. I would also ask the authors to make the Appendix G MGSM figure and text refer to the same language set and the same baseline values, and to provide per-language numbers for the appendix so a reader can reproduce the averages. The core idea is promising and the evaluation design is mostly sound; the revision is about making the evidence traceable and internally consistent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this paper has a clean idea and some real evidence, but the reporting is inconsistent enough that the advertised effect sizes are not currently verifiable. The abstract and Section 5 give different gains for the same experiments, and Appendix G contradicts its own figure. That has to be fixed before the numbers can be taken seriously.\n\nWhat's actually new: prior work used SAE features to flip the language of generated text; this paper instead uses them to improve downstream accuracy on XCOPA, XNLI, and MGSM. The method is straightforward – identify features with the largest target-vs-multilingual-centroid activation differences on FLORES-200 parallel sentences, decode the top-k (k=3) into a steering vector, add it at the final prompt position of layer 47. No training. The ablation is well designed: it includes hidden-state scaling, direct top-k hidden dimensions, and a least-squares English projection as controls, and it compares two reference choices (centroid vs English). The fact that English-reference and centroid-reference give similar results is a good sign that the result doesn't hinge on the centroid being perfectly language-neutral. The shared-coefficient (alpha=0.6) robustness run is also a nice touch.\n\nThe soft spots are mostly about the numbers. The abstract says +10.9/+5.3/+1.9; Table 1 says +11.3/+4.1/+0.6; Appendix G's text says MGSM is 83.6 vs 81.7, but Figure 10's own labels show 80.5 vs 81.1, and its caption says three languages while the values correspond to the four-language average from Table 1. These aren't rounding errors – they're off by 0.4 to 1.9 points. Since the paper's entire thesis is a quantitative improvement, each advertised gain has to trace to a single configuration. Right now the reader can't tell which configuration produced the headline numbers.\n\nThere are also no error bars or significance tests, and the intervention layer (47) was chosen on the XCOPA validation set and then reused for everything, with no sensitivity check on the other datasets. The per-language alpha tuning could inflate gains, though the fixed-alpha run shows the effect survives without per-language tuning.\n\nBottom line: the method is plausible and the controls are good, but the manuscript needs a major revision to clean up the numbers. Worth sending to peer review – a serious referee could sort this out. I'd bring it to the reading group once the number problem is fixed.","headline":"Clean idea with decent controls, but the headline numbers don't match across abstract, Table 1, and Appendix G – needs revision before the claim is evaluable.","tokens_in":13686,"tokens_out":4732,"would_cite":false,"duration_ms":38808,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that boosting a small set of target-language features, identified by comparing sparse autoencoder activations across parallel sentences, and adding them to the final prompt hidden state improves multilingual task accuracy…","keywords":["sparse autoencoders","multilingual inference","activation steering","target-language features","inference-time adaptation","parallel sentences","Gemma-3","language representation"],"falsifier":"Shuffle the alignments of the parallel sentences so that supposedly parallel sentences do not match semantically, then repeat feature selection and steering; if held-out gains largely persist, the difference vector is not cleanly 'target language minus shared meaning'. Independently, run the same pipeline on a second model family with its own matched pretrained SAEs; if the average gains disappear, the effect is tied to this model's SAE decomposition rather than a general property of multilingual representations.","tokens_in":12619,"feed_emoji":"🌐","tokens_out":9803,"duration_ms":86084,"temperature":0.7,"pith_summary":"The paper argues that a language's identity inside a multilingual model is concentrated in a few sparse-autoencoder features, and that amplifying them is enough to improve how the model answers in that language. The proposed method computes, for each transformer layer, the average SAE activation of the target language on parallel sentences and subtracts the average over the other languages, then keeps the three features with the largest difference, decodes them back into hidden space, and adds the result to the final prompt token's hidden state. No parameters are updated. On Gemma-3-12B-it the authors report average held-out accuracy improvements on XCOPA, XNLI, and MGSM, with the largest stated gain around 10.9 points, and ablations indicating that the sparse SAE representation, rather than activation scaling or raw hidden dimensions, is what carries the effect.","feed_headline":"Sparse steering lifts multilingual accuracy by 10.9 points","feed_subtitle":"Top-three language-tagged features, added at one hidden layer, beat baselines on XCOPA, XNLI, and MGSM.","key_machinery":"The load-bearing object is the target--multilingual-centroid activation difference, $\\Delta_{\\tau,\\mathrm{multi}}^l=\\mu_\\tau^l-\\mu_{\\mathrm{multi}}^l$, where $\\mu_\\lambda^l$ is the sentence- and token-average SAE activation for language $\\lambda$ at layer $l$ and $\\mu_{\\mathrm{multi}}^l$ averages over all non-target languages. The method keeps the three entries of $\\Delta$ with the largest absolute value, sets everything else to zero, and maps the sparse code back with the SAE decoder $g^l$ to form the steering vector $v^l$. The intervention is $\\tilde h_q^l = h_q^l + \\alpha v^l$ applied at the final prompt position of layer 47, after which the forward pass continues normally.","core_discovery":"The central claim is that steering by target-language SAE features is a causal intervention: the top-three features with the largest target-minus-centroid activation difference at layer 47, when decoded and added with strength $\\alpha$ to the final prompt hidden state, shift the model's next-token prediction toward the correct answer in that language. The authors define the multilingual centroid as the average SAE activation over all non-target languages and treat it as the language-neutral semantic content of the parallel sentences. They report that this intervention improves average accuracy on all three datasets, that a single shared coefficient $\\alpha=0.6$ still gives positive average gains, and that the language-only sparse signal is a competitive or better steering vector compared with full-representation and non-SAE alternatives.","pith_inferences":["A natural extension is to compare the selected features with those from contrasting two non-target languages; if the same features appear, the centroid difference may capture general multilingual variation rather than target-language identity alone.","The similar performance of the multilingual and English reference variants suggests the gain does not depend on an English-centered internal space; using a non-English high-resource language as reference would test this directly.","If the effect transfers across model families, it would make gradient-free language adaptation feasible for low-resource languages using only a few hundred translated sentences, a testable claim for future work."],"forward_implications":["Held-out accuracy rises on average on XCOPA, XNLI, and MGSM, with the largest gains on XCOPA and on specific languages such as Italian, Indonesian, and Turkish.","Using the same steering coefficient for every language and dataset still yields positive average gains, so the effect does not depend on per-language tuning.","The SAE-based sparse signal outperforms steering in raw hidden dimensions and hidden-state scaling, indicating that the sparse decomposition is what isolates a useful language signal.","Because the intervention changes no parameters and needs only a few hundred parallel sentences, it can be applied to any model that has compatible pretrained SAEs."],"supporting_citations":[{"why":"Introduces sparse autoencoders that decompose language-model activations into sparse, interpretable features, grounding the paper's feature-level view.","marker":"Cunningham et al. 2023"},{"why":"Develops dictionary learning for language models and motivates the use of sparse monosemantic features for steering.","marker":"Bricken et al. 2023"},{"why":"Shows that SAE features can exhibit language-specific activation patterns, the basis for ranking features by target-reference difference.","marker":"Deng et al. 2025"},{"why":"Demonstrates causal language control by intervening on sparse SAE features, the direct predecessor of the steering mechanism.","marker":"Chou et al. 2025"},{"why":"Supplies the activation-steering formulation of adding a steering vector to a hidden state, which the intervention builds on.","marker":"Turner et al. 2023"},{"why":"Provides an inference-time cross-lingual intervention baseline and the least-squares English projection strategy used in ablations.","marker":"Wang et al. 2025"},{"why":"Provides pretrained sparse autoencoders trained on residual-stream activations of Gemma models, enabling the experiments.","marker":"Lieberum et al. 2024"},{"why":"Supplies the model-matched sparse autoencoder suite for Gemma-3-12B-it used in layer-wise feature selection.","marker":"McDougall et al. 2025"},{"why":"Provides FLORES-200, the parallel sentence corpus used to estimate per-language SAE activation statistics.","marker":"Costa-Jussà et al. 2022"},{"why":"Cites averaging across languages to attenuate language-specific variation, supporting the multilingual centroid approximation.","marker":"Kirtane and Huang 2026"}],"fun_headline_variants":["SAE steering lifts multilingual accuracy by 10.9 points","No-training multilingual boost: SAE steering adds 10.9 points","Inference-time SAE steering improves multilingual reasoning 10.9 pts","Target-language features steer multilingual inference up 10.9 points","Sparse autoencoder steering boosts multilingual scores 10.9 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the average of non-target-language SAE activations is a faithful language-neutral semantic centroid, so the target-minus-centroid difference isolates language-specific content; if the centroid leaks semantic or third-language signal, the steering vector is not doing what the paper claims.","fun_headline_variants_meta":{"raw":{"variants":["SAE steering lifts multilingual accuracy by 10.9 points","No-training multilingual boost: SAE steering adds 10.9 points","Inference-time SAE steering improves multilingual reasoning 10.9 pts","Target-language features steer multilingual inference up 10.9 points","Sparse autoencoder steering boosts multilingual scores 10.9 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1606,"prompt_tokens":821,"completion_tokens":785,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":437,"completion_tokens_details":{"reasoning_tokens":694}},"tokens_in":437,"tokens_out":785,"duration_ms":7519,"temperature":1.0,"reasoning_tokens":694,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T17:20:36.415616+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shuffle the alignments of the parallel sentences so that supposedly parallel sentences do not match semantically, then repeat feature selection and steering; if held-out gains largely persist, the difference vector is not cleanly 'target language minus shared meaning'. Independently, run the same pipeline on a second model family with its own matched pretrained SAEs; if the average gains disappear, the effect is tied to this model's SAE decomposition rather than a general property of multilingual representations.","supporting_citations":[],"review_version":2}