{"id":"a9abf77c-9e57-4646-a66c-66d803f25016","arxiv_id":"2501.18838","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Simulated neuron activations derived from natural language explanations yield loss increases no better than zero-ablating the transcoder or SAE component.","lead":"EleutherAI researchers tried to rewrite part of a small language model in plain English: they explained its internal features, then had another language model simulate those features from the explanations. The rewritten model performed about as well as simply deleting that component, showing today's automated explanations are not yet precise enough for a human-readable rewrite.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The zero-ablation-equivalent result may reflect the single simulator's limits rather than explanation quality; Section 4.1's specificity diagnosis is computed with the same simulator, so the central conclusion is underdetermined.","rationale":"I agree with the reader's weakest assumption: the single-simulator design is the load-bearing dependency. The paper's abstract states the loss increase is statistically similar to zeroing the MLP, and Section 5 concludes explanations are severely wanting. But the chain of inference has an unmeasured link: the simulator's ability to convert an explanation plus context into an activation. Quantile normalization fixes the marginal distribution but not the identity of active latents; if the simulator ranks latents poorly, the final loss will be near the zero floor regardless of explanation content. Section 4.1's specificity diagnosis uses the same simulator outputs, so it cannot break the tie. The proposed test—stronger simulators on the same explanations—settles this directly. The reader's CONDITIONAL verdict already requires alternative-simulator tests and artifact release, so my reading does not change the verdict; it sharpens why the simulator test is necessary. If the stronger-simulator test passed, the paper would need substantial revision of its central claim.","tokens_in":7615,"tokens_out":8675,"duration_ms":99114,"concrete_test":"Re-run the 100%-substitution condition on the same 10K Pile prompts and same explanations with at least two stronger or differently-trained simulators (e.g., GPT-4o and Claude 3.5 Sonnet) using the identical Appendix A prompt, quantile normalization, and TopK. If any simulator's cross-entropy increase falls significantly below the zero baseline (non-overlapping bootstrap CIs), the conclusion that explanations are not detailed enough fails; if all remain statistically indistinguishable from zero, the result is robust to simulator choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that explanations are not detailed enough to beat zero ablation—requires that the measured failure be attributed to explanation content, not to the reader. The pipeline uses one simulator, Llama 3 Instruct 8B, with a coarse integer 0–9 rating (Appendix A), followed by quantile normalization and TopK (Sections 2.1, 3). The paper itself shows this simulator is badly miscalibrated (Figure 1), and the specificity estimate of about 80% (Section 4.1) is measured from the same simulator's predictions, so it cannot independently establish that explanations are the limiting factor. The zero baseline is a floor; a simulator that ignores explanations and predicts all latents inactive is described as achieving 'only slightly more loss' than the real pipeline, so the experiment has little power to detect partial explanation signal. Without varying the simulator or adding a shuffled-explanation control, the abstract's conclusion overreaches.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an end-to-end faithfulness test for natural language explanations of sparse model components. The authors train a transcoder and an SAE on Pythia 160M, generate explanations for their latents with the Paulo et al. (2024) pipeline, and then replace the encoder with an LLM-based simulator that predicts latent activations from the explanation and context. After quantile normalization and TopK post-processing, the simulated activations are patched back into the model, and cross-entropy loss is measured on held-out Pile prompts. The main finding is that replacing all latents with simulated activations increases loss similarly to zeroing the component, and that a trivial predictor always outputting zero activations is almost as good. The authors attribute this to explanations being not specific enough. The same protocol is applied to a residual-stream SAE with similar results.","tokens_in":7756,"tokens_out":7789,"duration_ms":72480,"significance":"If the result stands, the paper provides a valuable negative result for the interpretability community: the current explanation-generation and simulation pipeline does not capture enough information to outperform a zero-ablation baseline. The evaluation is external and uses bootstrap confidence intervals, which is a strength. The code is available, and the comparison to the zero baseline is a clear, falsifiable benchmark. However, the causal attribution to explanation quality is not fully supported, because the simulator is a fixed component and the specificity analysis is computed from the same simulator's predictions. This limits the generality of the conclusion as stated in the abstract and Section 4.1.","major_comments":[{"comment":"The abstract concludes that \"more detailed explanations are needed\" and Section 4.1 is titled \"Explanations are not detailed enough.\" This attribution is underdetermined by the experiments: the pipeline uses a single simulator (Llama 3 Instruct 8B with the 0-9 rating prompt), and the specificity and sensitivity measurements in Section 4.1 are computed from that same simulator's predictions. A simulator that is poorly calibrated or too weak could produce the same result even if the explanations were highly informative. The authors should either add a control that shuffles or randomizes explanations to demonstrate that explanation content matters relative to a no-information baseline, or reframe the conclusion to state that the proposed rewriting pipeline fails to beat the zero-ablation baseline without claiming that explanations themselves are insufficient.","section":"Abstract and Section 4.1"},{"comment":"The sentence in Section 4 that \"Using the empirical distribution as the target distribution for quantile normalization significantly improves this performance and rewritten models perform better than zeroing them\" refers to a condition in which the quantile normalizer is fit to the same evaluation prompts (Figure A3). This is an in-sample calibration and is not a valid measure of generalization; it also sits in tension with the abstract's headline claim that the pipeline is statistically similar to zero ablation. The manuscript should explicitly flag this result as an in-sample diagnostic, specify which normalizer was used for the main results in Figure 3, and avoid presenting the in-sample improvement as evidence for the pipeline's effectiveness.","section":"Section 4 and Appendix B"}],"minor_comments":[{"comment":"The phrase \"statistically similar\" is not accompanied by the actual loss increases or confidence intervals. Since the central claim is a null result, reporting the numeric differences and their bootstrap intervals is essential for readers to assess whether the differences are small or merely noisy.","section":"Section 3 / Figure 3"},{"comment":"The text \"a predictor that ignores the explanations and always predicts that every latent is inactive would achieve only slightly more loss than this setup\" is an important control but is not shown or quantified in any figure; adding this baseline to Figure 3 or reporting its loss increase numerically would strengthen the negative result.","section":"Section 4"},{"comment":"The sentence \"Using the empirical distribution as the target distribution for quantile normalization significantly improves this performance and rewritten models perform better than zeroing them, see Figure\" does not specify which figure is meant; it should refer to Figure A3 explicitly.","section":"Section 4"},{"comment":"There are minor typos: \"We observe than on average\" should read \"We observe that on average,\" and \"If the LLM predictor only can only achieve\" has a duplicated \"only.\"","section":"Section 4.1"},{"comment":"The prompt text contains a small grammatical error: \"a integer\" should be \"an integer.\"","section":"Appendix A"},{"comment":"The caption contains a typo: \"wether\" should be \"whether.\"","section":"Figure 2"},{"comment":"It is unclear whether the zero-ablation baseline zeroes the entire transcoder output (including the skip connection) or only the sparse component; clarifying this would help readers interpret the comparison.","section":"Section 3 / Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The experimental design is solid for a negative result about a specific pipeline, but the causal claim about explanation quality needs to be softened or tested with additional controls. The in-sample quantile-normalization result in Appendix B could be misread as a positive result, so it should be clearly labeled as a non-generalizing diagnostic. The paper is likely to be of interest to the mechanistic interpretability community after these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should read this paper if you care about how we measure whether natural-language explanations of SAE/transcoder latents actually capture the feature's behavior. The authors build a clean pipeline: explain a transcoder's latents, then replace the encoder with an LLM that predicts activations from the explanations, then measure cross-entropy loss against zeroing the whole MLP. The headline result is a clear negative: their simulator-guided rewrite does no better than zero ablation, for both transcoder and SAE.\n\nWhat's genuinely new is the evaluation protocol itself—partially rewriting the encoder and comparing to a zero-vector floor gives a concrete, loss-based benchmark for explanation faithfulness. The quantile normalization of simulated activations is a sensible calibration step and the paper shows it matters a lot. The specificity/sensitivity decomposition (Section 4.1) is instructive, and they are honest that detection scores predict simulation quality. The bootstrapped confidence intervals on 10K prompts are adequate.\n\nWhere I land on the central claim is more cautious than the abstract. The result is about their specific pipeline: one simulator (Llama 3 8B Instruct), one explanation generator, one model (Pythia 160M layer 6). The stress-test worry is fair: specificity is measured from the same simulator's predictions, so it cannot separate 'explanations are not detailed enough' from 'this simulator is not good at using them.' A shuffled-explanation control or a second, stronger simulator would have gone a long way. As it stands, the zero-ablation-equivalent finding is a solid benchmark result, but the causal reading—that better explanations are what's needed—is underdetermined. Also, the zero baseline is weak: predicting everything inactive gets only slightly more loss, so the experiment has limited power to detect partial explanation signal. That's not a flaw in execution; it's a limitation in interpretability.\n\nReproducibility is the other real soft spot. The code is on a branch without a commit hash, and the trained models, explanations, and simulated activations are not released. Recreating 327 million Llama calls is not realistic for most labs. A sample of artifacts would help.\n\nWho is this for? Anyone working on SAE/transcoder interpretability or explanation evaluation. It's a good reading group paper because it raises the right methodological questions. Not a general-audience result.\n\nMy recommendation: send it to peer review. It's a serious, honest empirical study with a novel protocol, and the concerns are addressable in revision rather than fatal. I'd press for artifact release and either a simulator variation or a more circumscribed conclusion.","headline":"A useful new faithfulness benchmark with an honest negative result, but the conclusion that explanations are the bottleneck outruns the single-simulator evidence.","tokens_in":8311,"tokens_out":3298,"would_cite":true,"duration_ms":32347,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rewriting a transformer layer from natural-language explanations loses as much as zeroing the layer out.","keywords":["mechanistic interpretability","transcoder","natural language explanations","activation simulation","quantile normalization","zero ablation","sparse autoencoders","interpretability evaluation"],"falsifier":"A concrete test: replace the LLM simulator with an oracle that, for each latent and context, reads the explanation and returns the true activation (or a much stronger predictor). If patching oracle predictions into the model cuts the loss increase substantially below the zero-ablation baseline, the original bottleneck was the simulator; if loss stays at zero-ablation level even with true activations, the explanations or latent decomposition themselves lack the information needed.","tokens_in":7375,"feed_emoji":"🧠","tokens_out":6731,"duration_ms":61796,"temperature":0.7,"pith_summary":"This paper asks whether a transformer's internal computation can be partially rewritten in natural language: replace the learned encoder of a sparse transcoder with a language model that reads a plain-English explanation of each neuron and predicts how strongly that neuron would activate on a given token. The authors build that pipeline for a single feedforward layer of Pythia 160M, applying quantile normalization to calibrate the simulator's predictions, and measure the resulting cross-entropy loss. Their central finding is that the loss increase is statistically indistinguishable from simply replacing the entire sparse MLP output with the zero vector; the same holds for a sparse autoencoder on the residual stream. They attribute the failure to explanation specificity: the simulator's predictions are only about 80% specific, so false-positive activations swamp the true $k=32$ active latents, and quantile normalization only trades specificity for sensitivity. The takeaway, if right, is that current natural-language explanations are nowhere near detailed enough to support a faithful human-readable rewrite of a model.","feed_headline":"Plain-English rewrite of an LLM loses as much as zeroing its MLP","feed_subtitle":"Sparse-feature explanations are too unspecific for simulated activations to beat the zero-ablation baseline.","key_machinery":"The load-bearing object is the transcoder: a wider MLP with a TopK activation function ($k=32$), a skip connection, and 32,768 sparsely activating latents trained to approximate the target feedforward network, together with a sparse autoencoder on the residual stream as a comparison. The pipeline replaces the transcoder's first layer with an LLM-based simulator: an automated interpretability pipeline first produces a one-sentence explanation per latent, then Llama 3 Instruct 8B is prompted to rate, on a 0-9 scale, how much the last token of a context should activate that latent given the explanation. Because raw ratings are badly miscalibrated, quantile normalization is applied per latent, using empirical CDFs from 10M tokens of true activations and 10K prompts of predicted activations; the normalized predictions are then passed through TopK and decoded through the transcoder's decoder to produce a patched residual-stream contribution. The measured quantity is the increase in cross-entropy loss over 10K (transcoder) or 1K (SAE) Pile prompts, compared against zeroing the MLP output and against substituting randomly or top-scoring latents.","core_discovery":"The paper's central claim is that its pipeline for simulating transcoder latents from natural language explanations does not beat the zero ablation baseline: when all 32,768 latents are replaced with LLM-predicted, quantile-normalized activations, the model's cross-entropy loss increase is statistically similar to replacing the sparse MLP output with the zero vector, and the same result holds for a residual-stream SAE. The authors show that uncalibrated predictions are far worse, because Llama's raw predictions systematically overpredict high activations by orders of magnitude; quantile normalization fixes the marginal distribution by mapping predicted activations onto the empirical activation distribution, but this mostly enforces a sparse prior, increasing specificity at the cost of sensitivity. Since the average explanation has only about 80% specificity, a predictor produces roughly ten times too many active latents, and even perfect specificity would not guarantee the top 32 predictions are correct. The paper concludes that explanations must identify when a feature is inactive as well as when it is active, and that more detailed, contrastive explanations are needed before a natural-language rewrite can outperform the trivial zero-output ablation.","pith_inferences":["Editorial inference: the zero-ablation ceiling means the simulator is currently contributing almost no task-relevant signal; a natural next experiment is to replace Llama with a much stronger or fine-tuned predictor (or a human) to see whether the gap closes, which would separate explanation quality from simulator capability.","Editorial inference: the same loss-based protocol could be used as a benchmark for explanation generation methods, with the zero-ablation line as the null hypothesis and any significant improvement as evidence that explanations carry real information.","Editorial inference: because quantile normalization enforces the empirical sparsity prior, the near-zero-ablation result may partly reflect the feature decomposition rather than the wording of explanations; transcoders with more monosemantic, better-separated latents could plausibly raise the ceiling without any change to the simulator.","A concrete testable extension: run the pipeline on a model with fewer, cleaner latents or with contrastive explanations, and report specificity and sensitivity separately; if specificity can be pushed above roughly 99% while sensitivity stays high, the zero-ablation parity should break."],"forward_implications":["If the result holds, then any future claim that a natural-language explanation 'explains' a feature must be tested by whether simulated activations from that explanation beat the zero-ablation baseline when patched into the model.","More detailed explanations, especially ones that specify when a feature is inactive, are a prerequisite for human-readable rewriting; the paper suggests contrastive pairs of similar features as one route to that.","Detection and fuzzing scores predict which explanations are more specific and sensitive, so interpretability scores can be used to prioritize which latents are worth rewriting.","Quantile normalization recovers most of the performance lost by uncalibrated simulation, but the remaining gap to zero-ablation is itself a measure of explanation quality."],"supporting_citations":[{"why":"Introduces sparse transcoders, the architecture the paper uses to approximate the feedforward network with interpretable latents.","marker":"Dunefsky et al. (2024)"},{"why":"Supplies the automated interpretability pipeline, detection/fuzzing scores, and the explanation evaluation protocol that the simulation task builds on.","marker":"Paulo et al. (2024)"},{"why":"Llama 3 Instruct 8B is the LLM-based simulator that predicts latent activations from explanations and context.","marker":"Dubey et al. (2024)"},{"why":"Provides Pythia 160M and its checkpoints, the model being partially rewritten and the reference points for loss degradation.","marker":"Biderman et al. (2023)"},{"why":"The Pile is the corpus used for training the transcoder/SAE and for evaluation prompts.","marker":"Gao et al. (2020)"},{"why":"Supplies the TopK activation function and SAE scaling methodology used to enforce sparsity.","marker":"Gao et al. (2024)"},{"why":"Gives the optimal-transport justification for quantile normalization, which calibrates simulator predictions.","marker":"Santambrogio (2015)"},{"why":"Introduces sparse autoencoders, the residual-stream comparison object in the paper.","marker":"Cunningham et al. (2023)"}],"fun_headline_variants":["English rewrite of LLM fails to beat zero ablation","Natural language rewrite of LLM matches zero ablation","Explaining features doesn't beat zeroing the layer","Sparse explanations no better than zero ablation","LLM rewrite in English equals zeroing MLP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that Llama 3 Instruct 8B, prompted with the 0-to-9 rating scheme, is a fair and near-optimal reader of the explanations; if a stronger simulator or richer interface could extract more signal from the same words, the zero-ablation-equivalent result would not measure explanation quality.","fun_headline_variants_meta":{"raw":{"variants":["English rewrite of LLM fails to beat zero ablation","Natural language rewrite of LLM matches zero ablation","Explaining features doesn't beat zeroing the layer","Sparse explanations no better than zero ablation","LLM rewrite in English equals zeroing MLP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00077,"raw_usage":{"total_tokens":3418,"prompt_tokens":958,"completion_tokens":2460,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":2387}},"tokens_in":574,"tokens_out":2460,"duration_ms":18125,"temperature":1.0,"reasoning_tokens":2387,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T22:16:12.377934+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: replace the LLM simulator with an oracle that, for each latent and context, reads the explanation and returns the true activation (or a much stronger predictor). If patching oracle predictions into the model cuts the loss increase substantially below the zero-ablation baseline, the original bottleneck was the simulator; if loss stays at zero-ablation level even with true activations, the explanations or latent decomposition themselves lack the information needed.","supporting_citations":[],"review_version":1}