{"id":"3ea6532e-bdaf-4d37-8558-e2a12aeb22f2","arxiv_id":"2506.23845","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Sparse autoencoders are best used to discover unknown concepts, not to act on known concepts.","lead":"This position paper argues that sparse autoencoders are better at surfacing unknown concepts than at detecting or steering known ones. It draws a line between negative results (concept detection, steering) and positive results (hypothesis generation, explaining LLM internals) to guide future SAE applications.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lossy SAE representations undermine discovery too: the paper's downstream-validation defense covers false positives, not false negatives, so 'powerful for discovering unknown concepts' needs recall/coverage evidence.","rationale":"The paper is a well-structured position statement, and the reader's conditional verdict is appropriate. The most load-bearing concern is not simply that the known/unknown taxonomy is post hoc; many useful scientific distinctions are retrospective. The sharper issue is internal to the argument. Section 3 attributes the negative results to information loss in the SAE reconstruction. Section 4 then claims that discovery tasks are robust to SAE feature unreliability because hypotheses can be validated downstream. But that robustness applies to false positives, not false negatives. If the reconstruction is lossy, some concepts may not correspond to any SAE feature, and those concepts are silently unavailable for discovery. The paper does not measure recall or coverage of SAE feature sets over any concept space, so the central claim that SAEs are 'powerful tools for discovering unknown concepts' is not yet supported. The proposed test would directly assess whether the lossy representation materially reduces the set of discoverable concepts. This does not move the verdict: the reader already assigned CONDITIONAL, and the condition should include evidence of recall/coverage. My concern overlaps with the reader's weakest assumption in that both question the clean separation of task types, but mine targets the logical asymmetry in the paper's own tolerance argument rather than external confounds such as model size or evaluation protocol.","tokens_in":11668,"tokens_out":4886,"duration_ms":58497,"concrete_test":"Evaluate discovery-mode recall on the concept detection benchmark of Kantamneni et al. (2025): train an SAE on the same LM representations, then for each of the 113 binary concepts, check whether any SAE feature's activation is significantly associated with the concept (e.g., AUROC above a fixed threshold using only SAE features). Measure the fraction of concepts discoverable this way, and compare that fraction to the recall achievable from raw LM representations (e.g., via a linear probe). If the SAE's discoverable set is substantially smaller (e.g., under 50% of the concepts), the paper's claim that lossy reconstruction is harmless for discovery fails, because many unknown concepts could be missed entirely.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 explains SAE failures on known-concept tasks by noting that an SAE reconstruction encodes strictly less information than the original representation. The paper then argues that this lossiness is harmless for discovery because a discovered hypothesis can be validated downstream. That defense addresses only false positives: a feature that fires on unrelated texts can be caught. It does not address false negatives: if a concept is not captured by any SAE feature, it will never be proposed as a hypothesis, and no downstream validation can recover it. Since the same lossy reconstruction that hurts concept detection also constrains the set of concepts available to discovery, the asymmetry carrying the paper's reconciliation is incomplete. The two positive results, Movva et al. (2025) and Lindsey et al. (2025), show that SAEs found some useful concepts on specific tasks, but neither measures how many relevant concepts were missed, nor compares the recall of SAE-based discovery against a raw-representation baseline. Without such coverage/recall analysis, 'powerful tools for discovering unknown concepts' is not established; the observed successes could be confined to concepts that happen to be well-represented in SAE features.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a conceptual distinction between two uses of sparse autoencoders: acting on known concepts, where recent large-scale evaluations find SAEs underperform simple baselines, and discovering unknown concepts, where positive results have been reported in hypothesis generation and in the biology of LLMs. It argues that this distinction reconciles the apparently conflicting SAE literature and then outlines applications in ML interpretability, fairness, auditing, safety, and the social and health sciences. The paper is a position paper rather than a new empirical study; its central evidence is a reinterpretation of cited results, chiefly Kantamneni et al. (2025), Wu et al. (2025), Movva et al. (2025), and Lindsey et al. (2025).","tokens_in":11849,"tokens_out":5062,"duration_ms":57749,"significance":"If the proposed distinction holds, it gives the field a clear way to separate domains where SAEs are currently underperforming from domains where they are genuinely promising, and it would redirect research effort toward discovery-oriented applications. The paper is clearly written, engages honestly with negative results, and offers a useful agenda for applying SAEs beyond mechanistic interpretability. However, the central claim is stated more strongly than the evidence supports: the known-versus-unknown taxonomy is constructed post hoc from the very results it is meant to explain, and the two positive-result papers provide evidence of precision but not of recall or coverage. The paper would be substantially stronger if it specified what empirical evidence could falsify the taxonomy, and if it either softened the 'powerful tools' claim or supported it with coverage/recall analysis. As written, it is a valuable organizing framework but not yet an established reconciliation.","major_comments":[{"comment":"The central taxonomy is constructed post hoc: tasks are assigned to 'acting on known concepts' or 'discovering unknown concepts' based on whether the concept was prespecified, and the paper then claims this distinction 'cleanly separates' the cited positive and negative results. The paper provides no controlled comparison or analysis showing that task category, rather than evaluation protocol, model size, feature selection, or baseline choice, is the causal variable behind SAE success or failure. For instance, the negative results use logistic-regression probes and prompting baselines, while the positive results compare against topic models, n-grams, and LLM-prompting pipelines; these differences in baselines and evaluation metrics are confounded with the known-versus-unknown distinction. The paper should acknowledge this limitation explicitly and state what evidence would falsify or support the taxonomy, otherwise the reconciliation remains an assertion rather than a demonstrated explanation.","section":"§3–§4 and Table 2"},{"comment":"The downstream-validation defense addresses only false positives: if an SAE feature fires on unrelated texts, a downstream check can catch that. It does not address false negatives: if a concept is not captured by any SAE feature, it will never be proposed as a hypothesis, and no downstream validation can recover it. This matters because the same lossy reconstruction argument used in §3 to explain SAE failures on concept detection also constrains the set of concepts available to discovery. The two positive results (Movva et al. 2025 and Lindsey et al. 2025) show that SAEs found some useful concepts on particular tasks, but neither measures how many relevant concepts were missed, nor compares the recall of SAE-based discovery against a raw-representation baseline. Without such coverage/recall evidence, the claim that SAEs are 'powerful tools for discovering unknown concepts' is not established; the observed successes could be confined to concepts that happen to be well-represented in SAE features.","section":"§4, 'Also note that after identifying concepts...'"},{"comment":"The explanatory mechanism offered for the positive results—that SAEs produce a set of concepts that is 'both tractable and expressive'—is speculative and is not tested against alternative explanations. For example, the success of Movva et al. (2025) could be due to the particular density of the embedding space, the target variable selection procedure, or the autointerpretation prompt, rather than to the discovery of unknown concepts per se. Moreover, one of the two positive-result papers (Movva et al. 2025) is authored by four of the current paper's six authors, which increases the need for independent evidence. The paper should either present additional empirical support for the mechanism or frame the mechanism as a hypothesis to be tested in future work.","section":"§4, 'What explains these positive results?'"}],"minor_comments":[{"comment":"The sentence 'In both works, the goal is use a dataset as input and surface unknown, task-relevant concepts as output' appears to contain a typo; it should read 'the goal is to use a dataset as input.'","section":"§4, first paragraph of 'Hypothesis generation'"},{"comment":"The transition from 'Though, methodological innovations may make SAEs more competitive at these tasks as well' to 'However, these baselines are less equipped to perform another simple task' is confusing: it is not clear whether 'these baselines' refers to non-SAE baselines or to the alternative SAE-based methods from Arad et al. (2025). Please clarify.","section":"§3, final paragraph"},{"comment":"The statement that 'a reconstruction encodes strictly less information about a token than the original LM representation' is too general: an overcomplete sparse code can in principle preserve information on the data manifold, and whether information is lost is an empirical property of the trained SAE. The argument should be qualified to the SAE configurations evaluated in the cited negative results.","section":"§3, 'What explains these negative results?'"},{"comment":"The claim that SAEs enable the same applications as keyword or topic-model methods 'but potentially with significantly higher quality' is an unsupported assertion; providing citations or a brief argument for why SAE features would yield higher-quality concepts would strengthen this section.","section":"§5, 'Health and social sciences'"},{"comment":"The abstract says the paper 'establish[es] a conceptual distinction,' while the body repeatedly uses hedged language such as 'we claim' and 'we speculate' (e.g., §3 and §4). Aligning the abstract with the evidential status of the argument would better represent the paper's contribution.","section":"Abstract and §1"}],"recommendation":"major_revision","confidential_remarks":"One of the two positive-result anchors (Movva et al. 2025) has four of this paper's six authors as co-authors. This is not by itself a validity concern, but it means the reconciliation rests in part on the authors' own work; independent positive results and recall/coverage evidence would substantially strengthen the case. The editor may also wish to consider whether the paper's scope, as a position paper, is a good fit for the journal's expectations regarding evidence for central claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what this paper gets right. The known-vs-unknown concept distinction is a genuinely useful way to organize the scattered SAE literature. It cleanly separates the negative results on concept detection and steering (where concepts are prespecified) from the positive results on hypothesis generation and \"biology of LLMs\" (where concepts are the output). That framing is new, it explains the apparent contradiction, and it points to concrete use cases in interpretability, fairness, and social/health science. The writing is clear and the survey of prior work is fair, including the negative results.\n\nThe soft spots are real but not fatal. The taxonomy is post hoc: tasks are classified by whether concepts were known ahead of time, and the paper asserts rather than tests that task type is the causal factor. There's also a circularity burden: one of the two positive-result pillars (Movva et al.) comes from the same research group, though Lindsey et al. is independent and strengthens the case.\n\nThe more substantive concern is the one about false negatives. The paper's defense of discovery—that downstream validation catches bad features—only handles false positives. If an SAE never encodes a concept, no validation step recovers it. The positive results show SAEs found some useful concepts on specific tasks, but neither measures recall nor compares against raw-representation baselines. So \"powerful tools for discovering unknown concepts\" is not established; the evidence shows SAEs can be useful in discovery, and the conceptual asymmetry is plausible, but coverage remains an open empirical question.\n\nThat said, the paper is a position paper, not an empirical claim. As a conceptual contribution it largely holds up: it correctly identifies that the negative results don't transfer to discovery tasks, and it gives the field a productive way to think about where SAEs belong. The missing recall evidence is a limitation to flag, not a reason to reject.\n\nFor whom: anyone working on interpretability, SAEs, or text-as-data in social science. It deserves a serious peer review, ideally with demands for a more careful discussion of coverage and a clearer separation of the authors' own prior work from independent evidence.","headline":"A useful conceptual reframing of SAE results, but the discovery claim is stronger than the evidence: no recall/coverage analysis.","tokens_in":12350,"tokens_out":2553,"would_cite":true,"duration_ms":27063,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that sparse autoencoders fail when acting on predefined concepts but excel when discovering unknown ones, a distinction that reconciles negative and positive results.","keywords":["sparse autoencoders","interpretability","concept discovery","hypothesis generation","mechanistic interpretability","model steering","concept detection","text-as-data"],"falsifier":"Run a large-scale evaluation of an explicitly open-ended discovery task, for example enumerating concepts that predict engagement or partisanship from a held-out corpus with the target concepts unknown to all methods at selection time, and compare SAE features against prompt-generated candidate concepts and n-gram rankings; if SAEs do not yield more validated hypotheses than the baselines, the discovery claim fails. Alternatively, reformulate a steering benchmark so the concept must be discovered rather than given; if SAEs still underperform prompting, then the taxonomy is not the operative variable.","tokens_in":11454,"feed_emoji":"🔎","tokens_out":6983,"duration_ms":67849,"temperature":0.7,"pith_summary":"This position paper argues that the conflicting results on sparse autoencoders (SAEs) resolve once tasks are separated by whether the concept is known in advance. On detection, steering, and unlearning tasks, where the concept is an input, SAEs underperform simple baselines, and this is where the negative evaluations concentrate. On discovery tasks such as hypothesis generation and explaining a language model's internal computations, the concept is an output to be found, and SAEs produce a tractable, precise list of interpretable concepts that outperforms non-SAE pipelines. The authors claim that SAEs are therefore not a general tool for acting on concepts but are especially well-suited to discovering unknown ones, and they trace applications in ML fairness, auditing, interpretability, safety, and the social and health sciences. A sympathetic reader would come away with a clearer allocation of where SAE research is most likely to pay off.","feed_headline":"Sparse autoencoders fail at known concepts, excel at discovery","feed_subtitle":"A known-versus-unknown split reconciles negative and positive SAE results and maps new uses.","key_machinery":"The load-bearing object is the known-versus-unknown concept taxonomy, a two-way split of tasks by whether the concept is an input (prespecified) or an output (to be discovered). The mechanism that carries the positive results is the SAE trained on model activations or dense text embeddings, whose sparse, roughly monosemantic neurons each fire on a single natural-language concept, together with autointerpretation, in which a language model names each neuron from its most-activating texts. By enumerating a tractable and expressive set of precise concepts and then checking which ones satisfy a target property, the SAE turns uninterpretable embeddings into interpretable ones, and this enumeration-plus-validation pipeline is what the paper says the acting-on-known-concepts baselines cannot do.","core_discovery":"On the paper's own terms, the central claim is that a single distinction, whether the concept is an input or an output of the task, cleanly separates the failures of sparse autoencoders from their successes. In concept detection, model steering, and concept unlearning, the concept is prespecified; here SAEs lose to prompting, finetuning, and logistic regression on raw representations, and the paper does not contest those results. In hypothesis generation and the 'biology of LLMs', the concept is unknown in advance; here SAEs win because they enumerate a set of precise monosemantic features that express variation in the data, letting the researcher select and statistically validate a shortlist. The paper's explanation is that reconstruction encodes strictly less information than the original representation, so raw representations are better for acting, while the ability to convert uninterpretable text embeddings into interpretable ones makes SAEs a discovery tool, with downstream validation making feature unreliability tolerable.","pith_inferences":["The taxonomy implies a testable workflow not run by the paper: a benchmark where target concepts are hidden from all methods at selection time should favor SAE enumeration, and failure there would narrow the thesis.","The paper's own speculation that SAE reconstructions lose information predicts a boundary: discovery tasks whose target property depends on fine-grained details lost in reconstruction would fail despite being discovery tasks.","In medical or social-science settings, SAE-generated concept lists could serve as a bounded, preregistered enumeration step, turning the prediction-explanation gap into a checklist of candidate mechanisms for confirmatory testing.","If the distinction holds, the right division of labor is to use raw representations for prediction, SAEs for hypothesis generation, and probes or steering only to validate a small set of discovered concepts."],"forward_implications":["The negative evaluations from concept detection and steering should not be read as a general indictment of SAEs; they diagnose a task class, and deprioritizing all SAE research on that basis would skip the discovery use cases.","For hypothesis generation, SAE-based pipelines should be expected to outperform n-grams, topic models, and direct prompting in producing statistically significant and human-validated hypotheses, as they did on news engagement and partisan speech.","In ML fairness, auditing, and safety, SAEs offer a way to surface unknown concepts that bias model outputs, including input features beyond demographic variables and undesirable output behaviors.","In the social and health sciences, SAEs convert uninterpretable text embeddings into interpretable ones, allowing established text-as-data applications to be revisited with predictive power that keyword and topic-model features lack.","Because discovered concepts can be validated downstream, unreliability of individual SAE features is survivable in discovery tasks, whereas it is directly harmful in concept detection and steering."],"supporting_citations":[{"why":"The 113-task concept-detection study showing SAE representations add no predictive power over raw LM representations.","marker":"[Kantamneni et al., 2025]"},{"why":"The large-scale concept detection and model steering evaluations where simple baselines outperform SAEs.","marker":"[Wu et al., 2025]"},{"why":"The hypothesis-generation results on news headlines and Congressional speeches that SAE pipelines beat.","marker":"[Movva et al., 2025]"},{"why":"The biology-of-LLMs analyses (rhyme planning, addition) demonstrating SAE-discovered mechanisms.","marker":"[Lindsey et al., 2025]"},{"why":"The concept-unlearning negative result the paper classifies as acting on known concepts.","marker":"[Farrell et al., 2024]"},{"why":"Documents the deprioritization of SAE research that motivates the reconciliation.","marker":"[Smith et al., 2025]"},{"why":"Foundation for SAEs finding highly interpretable monosemantic features in language models.","marker":"[Cunningham et al., 2023]"},{"why":"Scales monosemanticity and supplies the autointerpretation approach used to name SAE neurons.","marker":"[Templeton et al., 2024]"}],"fun_headline_variants":["SAEs: Weak on known concepts, powerful on unknowns","Unknown concepts are SAEs' strength, knowns their weakness","Sparse autoencoders: Discover unknowns, but knowns are a miss","For unknowns, SAEs excel; for knowns, they fall short","SAEs: Known concepts fail, unknown concepts discover"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the known-versus-unknown concept distinction, rather than differences in evaluation protocol, feature selection, or model size, is what makes SAEs succeed or fail on a given task.","fun_headline_variants_meta":{"raw":{"variants":["SAEs: Weak on known concepts, powerful on unknowns","Unknown concepts are SAEs' strength, knowns their weakness","Sparse autoencoders: Discover unknowns, but knowns are a miss","For unknowns, SAEs excel; for knowns, they fall short","SAEs: Known concepts fail, unknown concepts discover"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1498,"prompt_tokens":836,"completion_tokens":662,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":452,"completion_tokens_details":{"reasoning_tokens":574}},"tokens_in":452,"tokens_out":662,"duration_ms":6745,"temperature":1.0,"reasoning_tokens":574,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:29:10.523723+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a large-scale evaluation of an explicitly open-ended discovery task, for example enumerating concepts that predict engagement or partisanship from a held-out corpus with the target concepts unknown to all methods at selection time, and compare SAE features against prompt-generated candidate concepts and n-gram rankings; if SAEs do not yield more validated hypotheses than the baselines, the discovery claim fails. Alternatively, reformulate a steering benchmark so the concept must be discovered rather than given; if SAEs still underperform prompting, then the taxonomy is not the operative variable.","supporting_citations":[],"review_version":1}