{"id":"25913aad-dbcb-4677-9447-97a3a51978d4","arxiv_id":"2508.17320","paper_version":3,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"AdaptiveK sparse autoencoders use a linear-probe estimate of input complexity to set per-input sparsity, and the authors report gains over fixed-sparsity SAEs on ten language models.","lead":"This paper introduces AdaptiveK, a sparse autoencoder variant that chooses a different sparsity level for every input based on a predicted measure of that input's complexity. Generalist readers should look at it because it turns a practical tuning knob in interpretability, the sparsity count K, into a quantity predicted from the input itself.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported AdaptiveK gains may be an artifact of unequal sparsity budget: no evidence that variable K, not average K, drives improvements.","rationale":"The abstract alone is enough to raise a specific, testable concern: variable K interacts with capacity. Any method that can choose larger K on hard inputs can trivially improve reconstruction metrics by spending more latents. Since the abstract claims superiority over 'fixed-sparsity approaches' without specifying whether the comparison holds total active-feature budget fixed, the headline is vulnerable. This is not an internal inconsistency, but it is an unresolved confound. The reader's concern about probe leakage is adjacent; the clean matched-budget experiment would also serve as a control for whether the complexity signal is causally involved. A random-label probe control would further separate budget effects from signal effects. I therefore do not move the verdict: without full experimental details the paper remains unverified.","tokens_in":809,"tokens_out":3406,"duration_ms":38034,"concrete_test":"Recompute the headline comparison with a fixed-K baseline whose K is set to the mean number of active features that AdaptiveK selects on the same held-out set, and separately with a baseline matched for total active-feature budget across the corpus. If AdaptiveK's reconstruction/explained-variance advantage over these matched-budget baselines is not significantly positive, the complexity-driven sparsity is not doing the causal work. As a secondary control, train the complexity probe on shuffled labels and rerun; if gains persist, the probe signal is not responsible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing unverified premise is not the linearity of complexity but the causal attribution of the reported gains. AdaptiveK varies the number of active features per token, while a fixed-sparsity SAE holds K constant. If AdaptiveK's average K is higher than the baseline K, its better reconstruction fidelity, explained variance, and cosine similarity are predicted by the standard capacity-fidelity tradeoff, independent of any 'complexity-driven' allocation. The abstract reports no matched-budget control, and no equations or experimental details are accessible in the provided text, so the comparison may be between a variable-budget model and a fixed-budget model. The paper must show that AdaptiveK beats a fixed-K baseline matched to AdaptiveK's mean K (and, ideally, matched total FLOPs or latent allocations) on the same data; otherwise the complexity probe is not identified as the cause of improvement. A secondary circularity risk is that the probe may be fit using the SAE's own reconstructions or reconstruction error; the same matched-budget protocol with a probe trained on shuffled complexity labels would detect leakage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes AdaptiveK SAE, a sparse autoencoder variant that dynamically varies the number of active features per token according to a per-input complexity estimate derived from a linear probe on LLM activations, instead of using a fixed sparsity budget. The abstract claims that this complexity-driven allocation outperforms fixed-sparsity SAEs on reconstruction fidelity, explained variance, cosine similarity, and interpretability metrics across ten language models, while also removing the need for extensive hyperparameter tuning. The manuscript text provided to me is largely corrupted or unreadable beyond the abstract and some fragmentary lines, so this report is based primarily on the abstract and the accompanying review context.","tokens_in":977,"tokens_out":2293,"duration_ms":25889,"significance":"If the claims hold, AdaptiveK would be a practical contribution to SAE-based interpretability, automating sparsity selection and potentially improving reconstruction and interpretability trade-offs across models. The availability of code is a positive step toward reproducibility. However, the significance of the result depends critically on the experimental design, especially whether improvements are due to the complexity signal rather than merely to a different overall capacity budget. The current submission does not provide enough evidence to assess this, and the corrupted full text prevents verification of any details that might already address these concerns.","major_comments":[{"comment":"The central superiority claim over fixed-sparsity SAEs is not accompanied by any indication that the comparison is matched in average number of active features. If AdaptiveK simply uses a higher mean K than the baselines, the reconstruction and fidelity gains would be a trivial capacity effect. The manuscript must report the distribution of K selected by AdaptiveK and include a fixed-K baseline with K equal to AdaptiveK's mean K (and ideally matched total computation or FLOPs) on the same data, to isolate the contribution of the variable allocation itself.","section":"Abstract"},{"comment":"The role of the linear probe creates a circularity risk that is not addressed. If the probe is trained on labels derived from SAE reconstructions or from the same activation statistics that are later optimized, the complexity signal may simply encode reconstruction error. The manuscript must specify the exact training labels for the probe, demonstrate that they are independent of the SAE objective, and include a control in which complexity labels are shuffled and the same training pipeline is run; the shuffled-probe version should not match the reported gains if the complexity signal is causal.","section":"Abstract"},{"comment":"No error bars, confidence intervals, or significance tests are reported, despite claims of superiority across ten models and four metric families. Since SAE training is stochastic and metrics such as interpretability scores are often noisy, the manuscript should provide at least standard deviations over seeds or bootstrap confidence intervals, and ideally a paired test across models or tokens, to establish that the observed advantages are not within run-to-run variance.","section":"Abstract (experimental claims)"}],"minor_comments":[{"comment":"The full text of the manuscript is not readable in the version I received; it appears corrupted or garbled. The authors should ensure that the PDF or source is properly compiled and accessible, because the current submission cannot be meaningfully reviewed beyond the abstract.","section":"Full text"},{"comment":"The phrase 'eliminating the burden of extensive hyperparameter tuning' is overstated unless the paper reports how many hyperparameters remain in the complexity-to-K mapping (e.g., the scaling factor, thresholds, or probe architecture) and shows that the method is robust to reasonable choices of these values.","section":"Abstract"},{"comment":"The abstract does not mention the datasets used for the ten models, the SAE training setup, or the interpretability metric employed; these details are necessary for assessing the generality of the claims and should appear in the paper's experimental section.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising idea, and the code release is a useful step, but the version I reviewed does not permit a proper technical check: the body is corrupted, and the abstract alone cannot support the empirical claims. The authors need to resubmit a readable manuscript and, more importantly, add the matched-budget baseline and the probe-control experiment described in my major comments. The circularity concern is serious but likely addressable with a shuffled-label control. I would be willing to review a revised version that includes those details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The abstract reads well, and the idea is genuinely new: instead of fixing the number of active features per token, AdaptiveK uses a linear probe to estimate per-token complexity and allocates the sparsity budget accordingly. That is a real contribution, and releasing code is a plus. But the central empirical claim—that this beats fixed-K SAEs—is missing the one control that would make it credible. If AdaptiveK's average K is higher than the fixed-K baseline's, better reconstruction and explained variance are just the standard capacity effect. The abstract reports no matched-budget baseline, no error bars, and no significance tests. The circularity risk is real too: if the probe is trained using SAE reconstructions or reconstruction error, the improvement could be leakage; a shuffle test on probe labels would settle that. The claim of eliminating hyperparameter tuning is also overstated, since the complexity-to-K mapping and probe parameters still require choices. These are concerns rather than demonstrated flaws—the full text might address them—but as it stands the empirical results are unverified. I'd send this to peer review because the idea deserves scrutiny and the code lets referees check quickly. But acceptance should hinge on the matched-budget experiment. For SAE researchers, this is worth a look once the full version is readable; for now, treat the numbers as unvalidated.","headline":"Plausible adaptive-SAE idea; the abstract's empirical claims need a matched-budget control before they can be credited.","tokens_in":1492,"tokens_out":1807,"would_cite":false,"duration_ms":17668,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AdaptiveK SAE dynamically adjusts each input's sparsity budget by reading semantic complexity with a linear probe, outperforming fixed-sparsity autoencoders across ten language models.","keywords":["sparse autoencoders","large language model interpretability","adaptive sparsity","linear probing","semantic complexity","feature allocation","reconstruction fidelity","top-k selection"],"falsifier":"Run AdaptiveK against a fixed-K SAE while replacing the probe score with random per-input budgets drawn from the same distribution; if reconstruction fidelity, explained variance, and interpretability do not drop, the complexity signal is not doing the work.","tokens_in":603,"feed_emoji":"🧠","tokens_out":4122,"duration_ms":40303,"temperature":0.7,"pith_summary":"The paper argues that fixed sparsity levels are the wrong default for sparse autoencoders, because inputs differ in how many features they need to be represented faithfully. It proposes AdaptiveK, a sparse autoencoder that measures each input's semantic complexity with a linear probe and uses that score to set the number of active features for that input during training. On ten language models, the authors report that AdaptiveK beats fixed-top-K sparse autoencoders on reconstruction fidelity, explained variance, cosine similarity, and interpretability, while also removing the need to hand-tune K. If the claim holds, SAE users can replace a global hyperparameter with an automatic per-input budget that tracks the complexity of each token or context.","feed_headline":"AdaptiveK tunes sparsity per input and beats fixed budgets","feed_subtitle":"A linear probe reads semantic complexity from LLM activations, allocating features where they are needed.","key_machinery":"The machinery has two parts. The first is a linear probe that maps a model's internal representation to a scalar complexity estimate, exploiting the paper's observation that context complexity is linearly encoded in LLM activations. The second is an adaptive top-K selection mechanism inside the sparse autoencoder that takes that scalar and sets the number of active features per input during training. The probe is the load-bearing component: the entire method depends on the complexity score being accurate enough that allocating more features to higher-scoring inputs is what drives the reported gains.","core_discovery":"The central claim is that semantic complexity is linearly encoded in LLM representations, and that this signal can be used to allocate sparsity dynamically. AdaptiveK SAE trains a linear probe on hidden states to produce a per-input complexity score; a top-K selection layer then uses that score to decide how many features the autoencoder may use for that input. The authors state that this complexity-driven adaptation outperforms fixed-sparsity approaches across ten language models on reconstruction fidelity, explained variance, cosine similarity, and interpretability metrics, and that it eliminates the burden of tuning K. The advance, as the authors frame it, is that sparsity stops being a hyperparameter and becomes a property of the input.","pith_inferences":["A natural extension is to make the complexity signal self-supervised, replacing the probe's labels with reconstruction error or activation statistics, so AdaptiveK works where labeled complexity judgments are unavailable.","The same probe could double as a diagnostic: inputs whose predicted complexity is high but whose reconstruction error stays high are places where the SAE's feature set is insufficient.","If complexity is truly linearly encoded, the idea could transfer to other sparse decompositions, such as transcoders or dictionary learning on MLP outputs, not just autoencoders.","The method invites a stress test: compare AdaptiveK against an oracle that selects the best fixed K per difficulty bucket; if the oracle wins, the continuous probe is not the optimal use of the complexity signal."],"forward_implications":["Users of sparse autoencoders can drop K as a manually tuned hyperparameter and let each input set its own budget.","At the same or lower average sparsity, reconstruction fidelity and explained variance improve relative to fixed-K baselines.","The learned features become more interpretable, since easy inputs are not forced to spend features and hard inputs are not starved.","Because the complexity signal is reported to transfer across ten language models, a probe trained on one set of activations may serve others.","Adaptive budgets make SAE outputs more suitable for downstream interpretability work such as feature steering and circuit analysis."],"supporting_citations":[],"fun_headline_variants":["AdaptiveK autoencoders adjust sparsity to input complexity","LLM features allocated by measured complexity, not fixed K","AdaptiveK: sparsity set per input, outperforms fixed budgets","Complexity probe guides sparse autoencoder feature allocation","AdaptiveK: input complexity tunes sparsity, beats fixed top-K"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method stands or falls on the assumption that a linear probe trained on LLM activations measures semantic complexity accurately enough that giving more features to higher-scoring inputs is the actual cause of the reported gains.","fun_headline_variants_meta":{"raw":{"variants":["AdaptiveK autoencoders adjust sparsity to input complexity","LLM features allocated by measured complexity, not fixed K","AdaptiveK: sparsity set per input, outperforms fixed budgets","Complexity probe guides sparse autoencoder feature allocation","AdaptiveK: input complexity tunes sparsity, beats fixed top-K"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000592,"raw_usage":{"total_tokens":2721,"prompt_tokens":835,"completion_tokens":1886,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":1800}},"tokens_in":451,"tokens_out":1886,"duration_ms":12737,"temperature":1.0,"reasoning_tokens":1800,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:05:14.706434+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AdaptiveK against a fixed-K SAE while replacing the probe score with random per-input budgets drawn from the same distribution; if reconstruction fidelity, explained variance, and interpretability do not drop, the complexity signal is not doing the work.","supporting_citations":[],"review_version":2}