{"id":"cad5281d-b31f-42a6-92fc-8ae228436df2","arxiv_id":"2607.19386","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Pipeline choices, not SAE architecture, dominate the variance of autointerpretability scores across four metrics and two models, making cross-paper score comparisons unreliable without standardization.","lead":"This paper asks whether the standard numbers used to compare how interpretable sparse autoencoder features are can be trusted across papers. It finds that choices in the evaluation pipeline, such as which text corpus or which example sentences are used, cause more variation in the scores than the architecture of the autoencoder being tested.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"R1's variance decomposition is not identifiable from the non-crossed design: sigma2_arch is estimated from only 3 architectures with unmatched features, so the claim that pipeline choices dominate architecture variance is not yet established.","rationale":"The paper carefully documents instability via ICC, Jaccard lift, and mean-score trajectories, and the qualitative message that autointerpretability scores are sensitive to pipeline choices is well supported. R2 and R3 do not depend on the variance decomposition. However, R1, the headline quantitative claim, is exactly the variance-component comparison in Eq. 3. The reader's weakest assumption identifies the structural mismatch: sigma2_arch is estimated from a separate experiment with unmatched features and only three architecture levels, while sigma2_method is estimated from within-architecture experiments on BatchTopK features. I agree with the reader and would sharpen the point: the narrow confidence intervals in Table 14 for the architecture percentage are not credible with only three groups, and the crossed random-effects model in Eq. 7 does not match the realized experimental design. This does not mean the paper's conclusion is false; it means the central quantitative claim requires a design-appropriate reanalysis or a fully crossed follow-up. The paper's own caveat about the partially crossed design in Section 5 acknowledges the limitation but does not quantify its impact on the variance-component estimates. Thus the CONDITIONAL verdict remains appropriate, with the condition that R1 be re-established under a valid variance-component analysis. I am not moving to REJECT because the direct evidence of pipeline instability is strong, and the proposed Stability Check and Minimum Reporting Checklist stand independently of the variance decomposition.","tokens_in":38511,"tokens_out":11106,"duration_ms":106424,"concrete_test":"Re-estimate sigma2_arch using only Exp. 3 data with architecture as a fixed effect, computing a parametric bootstrap or Kenward-Roger confidence interval over the three architecture groups. If the 95% CI for the architecture percentage overlaps or exceeds the sum of the method-component percentages for any metric, Eq. 3 is not supported. As a stronger check, run a small fully crossed follow-up (e.g., 20 features from each of the three architectures scored under 2 corpora x 2 draws x 2 explainers) and refit Eq. 7 on the balanced design; the dominance claim should reproduce.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the variance decomposition underlying R1. Equations 7-9 fit a crossed random-effects model with feature, corpus, draw, explainer, and architecture effects, but the data are not crossed: corpus is varied only in Exp. 1, draw only in Exp. 2a, explainer only in Exp. 2b, paraphrase only in Exp. 2c, and architecture only in Exp. 3. In Exp. 3, features are not matched across architectures and each feature is scored under a single fixed condition, so sigma2_arch is identified from differences among only three architecture group means. With three groups, the sampling variance of the between-group variance component is very large; the narrow confidence intervals reported in Table 14 (e.g., arch = 2% [1,5] for Pythia simulation) are not plausible under standard REML. Moreover, method components are estimated on BatchTopK features only, while architecture is varied on a different feature population, so the sum in Eq. 3 compares variance components from different designs and feature pools. The paper acknowledges partial crossing in Section C.4 but does not show that the resulting estimates are unbiased or that the dominance in Eq. 3 is robust to the design. The central claim therefore rests on a statistical comparison that has not been validated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the reliability of autointerpretability scores for sparse autoencoder (SAE) features, where an LM explains a feature and a second LM scores the explanation. The authors systematically vary four methodological axes (evaluation corpus, random draw of activating examples, explainer LLM, and explanation paraphrasing) plus SAE architecture, across two base models (Pythia-160M, Apertus-8B) and four metrics (simulation, detection, fuzzing, purity). They report three main findings: R1) methodological variance collectively exceeds architectural variance; R2) each metric has a distinct instability profile, with detection most stable and fuzzing unreliable; R3) top-k feature rankings are unstable across corpus and draw conditions even when mean scores are stable. They also show that explanation similarity is not a diagnostic for score instability and propose a Stability Check and a Minimum Reporting Checklist for practitioners.","tokens_in":38872,"tokens_out":5792,"duration_ms":55583,"significance":"If the central claims hold, this is a timely and practically important result for the SAE interpretability community, because cross-paper and cross-architecture comparisons of autointerpretability scores are common and often used as evidence in the SAE usefulness debate. The paper's strengths include a broad experimental grid (two models, four metrics, four method axes plus architecture), explicit LLM prompts and protocol details, honest limitation statements, and a concrete, low-cost Stability Check that many groups could adopt. The paper also reports a clean falsifiable observation: explanation semantic similarity stays high even when scores are unstable, ruling out a cheap diagnostic. However, the headline R1 claim rests on a variance decomposition fitted to a non-crossed, partially unmatched design, and the manuscript does not provide enough statistical detail to establish that the comparison in Eq. (3) is identifiable. That issue is load-bearing and needs to be addressed before the main claim can be accepted.","major_comments":[{"comment":"The central hypothesis in Eq. (3) is tested using variance components estimated from a design that is neither fully crossed nor matched at the architecture level. Method components (corpus, draw, explainer, phrasing) are estimated on BatchTopK features within Experiments 1 and 2, whereas sigma2_arch is estimated in Experiment 3 from three architectures whose features are not matched across architectures and where all method factors are fixed. Section C.4 states only that 'partially overlapping subsets' are used and gives no fitting procedure, identifiability argument, or sensitivity analysis. Since the architecture variance component is identified from differences among only three architecture group means, the narrow confidence intervals in Table 14 (e.g., arch = 2% [1,5]) are not self-evidently plausible without additional detail. As written, R1's claim that methodological variance exceeds architectural variance is not established by the reported analysis.","section":"C.4 / Table 14 / Eq. (7)-(9)"},{"comment":"The Fuzzing metric is described as token-level, with an AUC over token-level predictions, but the scoring prompt in B.8.2 asks the LLM to return one binary label per example ('return 1 if the sentence is correctly labeled or 0 if the tokens are mislabeled'). The paper never states how token-level predictions are obtained from these sequence-level labels. Without this detail, the fuzzing scores underlying Tables 2 and 14 and the R2 claim that fuzzing is unreliable across all conditions cannot be reproduced or interpreted.","section":"B.8.1 / B.8.2 (Fuzzing)"},{"comment":"The top-k Jaccard lift analysis compares rankings across conditions with different numbers of evaluable features. Table 7 shows that the corpus condition substantially changes the feature pool, reducing the number of features with ten activating examples by about half on Apertus-8B and producing different overlap sets across corpora. The chance baseline in Eq. (4) assumes a fixed pool of size N, but when the eligible feature set is condition-dependent, the expected Jaccard under random selection is not k/(2N-k). The paper does not specify whether the lift is computed only on the common feature intersection or on the union of condition-specific pools; if the latter, R3's near-chance results may be an artifact of a misspecified chance model rather than genuine ranking instability.","section":"Section 3 (R3) / Eq. (4) / Table 7"}],"minor_comments":[{"comment":"The main text says the architecture contribution on Apertus-8B has a peak of 11% for DETECTION, but Table 14 reports sigma2_arch = 15% [9,23] for Apertus-8B DETECTION; the text and table should be reconciled.","section":"Section 3, R1"},{"comment":"The text states 'The last metric, PURITY, does not depend on any LLM', but the immediately following description and the scoring prompt in B.8.2 describe PURITY as rated by a scorer LLM; one of these statements is incorrect.","section":"B.8.1 (Purity)"},{"comment":"Negative ICC values are interpreted as 'less consistent than chance'; this phrasing is statistically inaccurate, since negative estimates can arise from sampling variability in a one-way random-effects model. The results should be described as poor reliability rather than worse-than-chance consistency.","section":"Table 2 / Section 3, R2"},{"comment":"The manuscript contains incomplete sentences: Section D.3 ends with 'The time for one experiment on PYTHIA-160M is roughly .' and Section D.2 ends with 'limitatios'; these should be completed or removed.","section":"D.3 / D.2"},{"comment":"The Stability Check thresholds (ICC at least 0.5, Jaccard lift at least 2x) are presented as prescriptive enough to 'pass' or 'fail' a metric. The authors acknowledge in Section 5 that these are pragmatic cutoffs, but the main-text presentation should more prominently frame them as heuristics rather than validated standards.","section":"Section 4.1 / Table 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an early draft in places: several sentences are incomplete, and the Purity metric description is internally contradictory. The main statistical concern is the variance decomposition under the non-crossed design; this is fixable by either restricting R1 to a properly identified comparison, adding a sensitivity analysis, or substantially qualifying the claim, but it must be addressed. I do not see grounds for rejection if the authors can show that R1 is robust to the design issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper is worth reading and worth sending to referees, but its central claim is not as solid as its presentation suggests. The systematic variation of corpus, draw, explainer, and paraphrase across four metrics and two models is genuinely new. Prior work found individual sources of instability; this paper tries to weigh them against architecture, and that framing is the right question. The per-feature ICC and Jaccard results are well done, and the dissociation between stable explanation similarity and unstable scores is a real and useful finding. The proposed Stability Check and Minimum Reporting Checklist are practical and I could see them being adopted.\n\nWhere the paper goes soft is R1. The variance decomposition in Eqs. 7--9 assumes a crossed random-effects model, but the data are not crossed. Corpus is varied in one experiment, draw in another, explainer in a third, paraphrase in a fourth, and architecture in a fifth. The architecture component is estimated from only three architectures, with unmatched features, under a single fixed pipeline. With three groups, the sampling variance of a between-group variance component is large, so the tight confidence intervals in Table 14 look implausible. Method components come from BatchTopK features only; architecture variation comes from a different feature population. The paper acknowledges partial crossing and calls the estimates approximate, but then R1 is stated flatly as a finding. That is the gap.\n\nNone of this kills the paper. R2 and R3 do not depend on the variance decomposition, and the qualitative claim that pipeline choices are large relative to architecture differences may well be true. But as it stands, R1 is a design-dependent estimate, not an established comparison. A referee should ask for a properly identified analysis or a softer claim.\n\nTwo smaller issues: the arXiv version does not include the code or Apertus checkpoints, and there are leftover placeholders. Both are fixable. I would not desk-reject this; I would send it to review with the expectation of major revision. The audience is anyone building or comparing SAE evaluations, and they should read the R2/R3 results and the checklist even if they treat R1 with caution. I would cite the Jaccard/ICC findings, not the variance decomposition as-is.","headline":"A genuinely useful empirical study of SAE evaluation instability whose headline variance-decomposition claim outruns its statistics; R2 and R3 stand, R1 needs rework before the paper can support field-wide conclusions.","tokens_in":730,"tokens_out":807,"would_cite":true,"duration_ms":27813,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Evaluation pipeline choices, not SAE architecture, dominate autointerpretability score variance.","keywords":["sparse autoencoders","autointerpretability","evaluation reliability","variance decomposition","intraclass correlation coefficient","feature ranking stability","LLM-as-judge evaluation","SAE interpretability"],"falsifier":"A fully crossed experiment that scores the same set of features across three SAE architectures and all pipeline axes (corpora, draws, explainers, phrasings) in one design would settle the claim: if the architecture variance component reaches or exceeds the summed pipeline components, the central claim collapses. A lighter check would be to recompute the decomposition with a different scorer model, since the paper fixes one scorer.","tokens_in":38274,"feed_emoji":"📊","tokens_out":7218,"duration_ms":54115,"temperature":0.7,"pith_summary":"Autointerpretability scores—where one language model explains what a sparse-autoencoder feature detects and a second language model scores the explanation—are the field's main instrument for comparing sparse autoencoders (SAEs). This paper tries to establish that those scores mostly reflect pipeline choices rather than the features or architectures they are meant to compare. Across four metrics, two large language models, and three SAE architectures, the authors test whether the combined variance from corpus, random example draw, explainer, and phrasing exceeds the variance attributable to SAE architecture; they report it does for every metric. A sympathetic reader would care because if this holds, published cross-paper comparisons of SAEs based on such scores may be comparing evaluation setups instead of architectures.","feed_headline":"Evaluation pipeline, not SAE choice, dominates interpretability scores","feed_subtitle":"Cross-paper SAE comparisons may measure corpus, samples, and explainer choice, not architecture quality.","key_machinery":"The carrying object is a variance-components model fit by restricted maximum likelihood. For score $s_{ijklm}$ of feature $i$ under corpus $j$, draw $k$, explainer $l$, and architecture $m$, the model writes $s_{ijklm} = \\mu + \\alpha_i + \\beta_j + \\gamma_k + \\delta_l + \\epsilon_m$ with independent mean-zero random effects, giving total variance $\\sigma^2_{\\text{total}} = \\sigma^2_{\\alpha} + \\sigma^2_{\\beta} + \\sigma^2_{\\gamma} + \\sigma^2_{\\delta} + \\sigma^2_{\\epsilon}$. The test is whether $\\sigma^2_{\\text{method}} = \\sigma^2_{\\beta} + \\sigma^2_{\\gamma} + \\sigma^2_{\\delta} + \\sigma^2_{\\text{phrasing}}$ exceeds $\\sigma^2_{\\epsilon} = \\sigma^2_{\\text{arch}}$. Two supporting measures track within-feature instability: the intraclass correlation coefficient (the proportion of total variance that is stable between features) and top-k Jaccard lift (overlap of top-k feature sets normalized by chance). The variance decomposition gives the global verdict; the ICC and Jaccard lift localize which pipeline axis breaks which metric.","core_discovery":"The central claim is that the reliability precondition for cross-paper autointerpretability comparison fails: for simulation, detection, fuzzing, and purity scores on both Pythia-160M and Apertus-8B, methodological variance ($\\sigma^2_{\\text{method}} = \\sigma^2_{\\text{corpus}} + \\sigma^2_{\\text{draw}} + \\sigma^2_{\\text{explainer}} + \\sigma^2_{\\text{phrasing}}$) exceeds architectural variance ($\\sigma^2_{\\text{arch}}$). The architecture term contributes less than 7% of total variance in most cases, while corpus and draw dominate for simulation and purity. Each metric has its own failure profile: detection is the most stable, fuzzing is unreliable under every variation, and simulation and purity are highly sensitive to which random examples are drawn. Mean scores look stable across conditions, but top-k feature rankings have near-chance overlap across corpus and draw conditions, and explanation similarity stays high even when scores move substantially, so similarity cannot serve as a cheap diagnostic. The paper concludes that autointerpretability scores should not be treated as a stable basis for cross-SAE comparison without running a Stability Check and reporting the Minimum Reporting Checklist.","pith_inferences":["If the finding generalizes, many existing single-corpus, single-draw SAE comparisons in the literature would need to be treated as pipeline-dependent; the paper itself does not re-evaluate prior results.","A natural extension is to vary the scorer model and the SAE layer, which the paper fixed; including those axes might make the pipeline-variance dominance even larger.","The proposed Stability Check could be run prospectively as a reporting standard: authors would report ICC and Jaccard lift alongside mean scores, and readers could discount comparisons that fail the thresholds.","The dissociation between stable explanation similarity and unstable scores suggests the noise enters through the held-out evaluation sequences, pointing toward the scoring stage rather than the explanation stage as the main site of measurement error."],"forward_implications":["Cross-architecture comparisons made with one corpus and one random draw cannot reliably attribute score differences to architecture; the observed effect sizes are small and inconsistent in direction.","Feature selection applications such as steering, circuit analysis, and manual inspection inherit the ranking instability: the top-k set on one draw or corpus can be near-chance relative to another.","Multi-metric evaluation is necessary because no single metric is reliable everywhere: detection is most stable, fuzzing unreliable across conditions, and simulation and purity are sensitive to draw.","Explanation similarity (cosine or BERTScore) cannot be used as a diagnostic for score instability, since it stays high even when scores vary.","A Stability Check (ICC at least 0.5 and top-k Jaccard lift at least 2x) is the minimum bar before interpreting scores as architectural evidence."],"supporting_citations":[{"why":"Supplies the simulation metric and the core LM-explains-LM-scores pipeline that the paper evaluates.","marker":"[2]"},{"why":"Supplies detection and fuzzing metrics and earlier evidence that example selection and explainer capacity shift scores.","marker":"[27]"},{"why":"Supplies the purity metric and prior work on explanation failures and scoring artifacts.","marker":"[28]"},{"why":"Provides the Pythia-160M SAE checkpoints and the architecture comparisons that motivate the study.","marker":"[16]"},{"why":"Supplies the ICC formulation and the reliability thresholds used to judge metric stability.","marker":"[17]"},{"why":"Supplies the top-k Jaccard stability index and the chance-normalized lift used for ranking stability.","marker":"[19]"},{"why":"Supplies BERTScore, one of the explanation-similarity measures shown to be insensitive to score instability.","marker":"[36]"},{"why":"Supplies the Pile evaluation corpus used as the in-distribution baseline for the corpus experiments.","marker":"[11]"}],"fun_headline_variants":["Pipe choices, not SAE models, drive interpretability score variance","Method beats model: SAE interpretability scores are pipeline artifacts","Autointerpretability scores unstable: corpus and draw decide outcome","Fuzzing unreliable, detection stable? SAE scores need stability checks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central result assumes that the variance in scores can be cleanly divided into independent contributions from pipeline choices and from SAE architecture, even though the architecture effect is measured in a separate experiment while the pipeline effects are measured without varying architecture.","fun_headline_variants_meta":{"raw":{"variants":["Pipe choices, not SAE models, drive interpretability score variance","Method beats model: SAE interpretability scores are pipeline artifacts","Autointerpretability scores unstable: corpus and draw decide outcome","Fuzzing unreliable, detection stable? SAE scores need stability checks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1718,"prompt_tokens":1052,"completion_tokens":666,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":593}},"tokens_in":668,"tokens_out":666,"duration_ms":6774,"temperature":1.0,"reasoning_tokens":593,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:35:48.343853+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A fully crossed experiment that scores the same set of features across three SAE architectures and all pipeline axes (corpora, draws, explainers, phrasings) in one design would settle the claim: if the architecture variance component reaches or exceeds the summed pipeline components, the central claim collapses. A lighter check would be to recompute the decomposition with a different scorer model, since the paper fixes one scorer.","supporting_citations":[{"cited_title":"Automatically interpreting millions of features in large language models","cited_arxiv_id":null,"evidence_quote":"Supplies detection and fuzzing metrics and earlier evidence that example selection and explainer capacity shift scores."},{"cited_title":"FADE: Why bad descriptions happen to good features","cited_arxiv_id":null,"evidence_quote":"Supplies the purity metric and prior work on explanation failures and scoring artifacts."},{"cited_title":"SAEBench: A comprehensive benchmark for sparse autoencoders in language model interpretability","cited_arxiv_id":null,"evidence_quote":"Provides the Pythia-160M SAE checkpoints and the architecture comparisons that motivate the study."},{"cited_title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research.Journal of chiropractic medicine, 15(2):155–163, 2016","cited_arxiv_id":null,"evidence_quote":"Supplies the ICC formulation and the reliability thresholds used to judge metric stability."},{"cited_title":"A stability index for feature selection","cited_arxiv_id":null,"evidence_quote":"Supplies the top-k Jaccard stability index and the chance-normalized lift used for ranking stability."},{"cited_title":"Mathematical equations and formulas","cited_arxiv_id":null,"evidence_quote":"Supplies BERTScore, one of the explanation-similarity measures shown to be insensitive to score instability."},{"cited_title":"The pile: An 800gb dataset of diverse text for language modeling, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the Pile evaluation corpus used as the in-distribution baseline for the corpus experiments."}],"review_version":1}