Partially Rewriting a Transformer in Natural Language

Gon\c{c}alo Paulo; Nora Belrose

REVIEW 2 major objections 7 minor 1 cited by

Partially Rewriting a Transformer in Natural Language

T0 review · 2 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Rewriting a transformer layer from natural-language explanations loses as much as zeroing the layer out.

desk verdict A useful new faithfulness benchmark with an honest negative result, but the conclusion that explanations are the bottleneck outruns the single-simulator evidence. read the letter →

arxiv 2501.18838 v1 pith:GGFVPWQ3 submitted 2025-01-31 cs.LG cs.CL

Gonçalo Paulo , Nora Belrose This is my paper · ORCID

classification cs.LGcs.CL

keywords mechanisticinterpretabilitytranscodernaturallanguageexplanationsactivationsimulationquantilenormalizationzeroablationsparseautoencodersevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a transformer's internal computation can be partially rewritten in natural language: replace the learned encoder of a sparse transcoder with a language model that reads a plain-English explanation of each neuron and predicts how strongly that neuron would activate on a given token. The authors build that pipeline for a single feedforward layer of Pythia 160M, applying quantile normalization to calibrate the simulator's predictions, and measure the resulting cross-entropy loss. Their central finding is that the loss increase is statistically indistinguishable from simply replacing the entire sparse MLP output with the zero vector; the same holds for a sparse autoencoder on the residual stream. They attribute the failure to explanation specificity: the simulator's predictions are only about 80% specific, so false-positive activations swamp the true $k=32$ active latents, and quantile normalization only trades specificity for sensitivity. The takeaway, if right, is that current natural-language explanations are nowhere near detailed enough to support a faithful human-readable rewrite of a model.

What carries the argument

The load-bearing object is the transcoder: a wider MLP with a TopK activation function ($k=32$), a skip connection, and 32,768 sparsely activating latents trained to approximate the target feedforward network, together with a sparse autoencoder on the residual stream as a comparison. The pipeline replaces the transcoder's first layer with an LLM-based simulator: an automated interpretability pipeline first produces a one-sentence explanation per latent, then Llama 3 Instruct 8B is prompted to rate, on a 0-9 scale, how much the last token of a context should activate that latent given the explanation. Because raw ratings are badly miscalibrated, quantile normalization is applied per latent, using empirical CDFs from 10M tokens of true activations and 10K prompts of predicted activations; the normalized predictions are then passed through TopK and decoded through the transcoder's decoder to produce a patched residual-stream contribution. The measured quantity is the increase in cross-entropy loss over 10K (transcoder) or 1K (SAE) Pile prompts, compared against zeroing the MLP output and against substituting randomly or top-scoring latents.

What would settle it

A concrete test: replace the LLM simulator with an oracle that, for each latent and context, reads the explanation and returns the true activation (or a much stronger predictor). If patching oracle predictions into the model cuts the loss increase substantially below the zero-ablation baseline, the original bottleneck was the simulator; if loss stays at zero-ablation level even with true activations, the explanations or latent decomposition themselves lack the information needed.

Watch

Extended reading notes

Core claim

The paper's central claim is that its pipeline for simulating transcoder latents from natural language explanations does not beat the zero ablation baseline: when all 32,768 latents are replaced with LLM-predicted, quantile-normalized activations, the model's cross-entropy loss increase is statistically similar to replacing the sparse MLP output with the zero vector, and the same result holds for a residual-stream SAE. The authors show that uncalibrated predictions are far worse, because Llama's raw predictions systematically overpredict high activations by orders of magnitude; quantile normalization fixes the marginal distribution by mapping predicted activations onto the empirical activation distribution, but this mostly enforces a sparse prior, increasing specificity at the cost of sensitivity. Since the average explanation has only about 80% specificity, a predictor produces roughly ten times too many active latents, and even perfect specificity would not guarantee the top 32 predictions are correct. The paper concludes that explanations must identify when a feature is inactive as well as when it is active, and that more detailed, contrastive explanations are needed before a natural-language rewrite can outperform the trivial zero-output ablation.

Load-bearing premise

The load-bearing assumption is that Llama 3 Instruct 8B, prompted with the 0-to-9 rating scheme, is a fair and near-optimal reader of the explanations; if a stronger simulator or richer interface could extract more signal from the same words, the zero-ablation-equivalent result would not measure explanation quality.

Editorial extensions

If this is right

If the result holds, then any future claim that a natural-language explanation 'explains' a feature must be tested by whether simulated activations from that explanation beat the zero-ablation baseline when patched into the model.
More detailed explanations, especially ones that specify when a feature is inactive, are a prerequisite for human-readable rewriting; the paper suggests contrastive pairs of similar features as one route to that.
Detection and fuzzing scores predict which explanations are more specific and sensitive, so interpretability scores can be used to prioritize which latents are worth rewriting.
Quantile normalization recovers most of the performance lost by uncalibrated simulation, but the remaining gap to zero-ablation is itself a measure of explanation quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the zero-ablation ceiling means the simulator is currently contributing almost no task-relevant signal; a natural next experiment is to replace Llama with a much stronger or fine-tuned predictor (or a human) to see whether the gap closes, which would separate explanation quality from simulator capability.
Editorial inference: the same loss-based protocol could be used as a benchmark for explanation generation methods, with the zero-ablation line as the null hypothesis and any significant improvement as evidence that explanations carry real information.
Editorial inference: because quantile normalization enforces the empirical sparsity prior, the near-zero-ablation result may partly reflect the feature decomposition rather than the wording of explanations; transcoders with more monosemantic, better-separated latents could plausibly raise the ceiling without any change to the simulator.
A concrete testable extension: run the pipeline on a model with fewer, cleaner latents or with contrastive explanations, and report specificity and sensitivity separately; if specificity can be pushed above roughly 99% while sensitivity stays high, the zero-ablation parity should break.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

You should read this paper if you care about how we measure whether natural-language explanations of SAE/transcoder latents actually capture the feature's behavior. The authors build a clean pipeline: explain a transcoder's latents, then replace the encoder with an LLM that predicts activations from the explanations, then measure cross-entropy loss against zeroing the whole MLP. The headline result is a clear negative: their simulator-guided rewrite does no better than zero ablation, for both transcoder and SAE.

What's genuinely new is the evaluation protocol itself—partially rewriting the encoder and comparing to a zero-vector floor gives a concrete, loss-based benchmark for explanation faithfulness. The quantile normalization of simulated activations is a sensible calibration step and the paper shows it matters a lot. The specificity/sensitivity decomposition (Section 4.1) is instructive, and they are honest that detection scores predict simulation quality. The bootstrapped confidence intervals on 10K prompts are adequate.

Where I land on the central claim is more cautious than the abstract. The result is about their specific pipeline: one simulator (Llama 3 8B Instruct), one explanation generator, one model (Pythia 160M layer 6). The stress-test worry is fair: specificity is measured from the same simulator's predictions, so it cannot separate 'explanations are not detailed enough' from 'this simulator is not good at using them.' A shuffled-explanation control or a second, stronger simulator would have gone a long way. As it stands, the zero-ablation-equivalent finding is a solid benchmark result, but the causal reading—that better explanations are what's needed—is underdetermined. Also, the zero baseline is weak: predicting everything inactive gets only slightly more loss, so the experiment has limited power to detect partial explanation signal. That's not a flaw in execution; it's a limitation in interpretability.

Reproducibility is the other real soft spot. The code is on a branch without a commit hash, and the trained models, explanations, and simulated activations are not released. Recreating 327 million Llama calls is not realistic for most labs. A sample of artifacts would help.

Who is this for? Anyone working on SAE/transcoder interpretability or explanation evaluation. It's a good reading group paper because it raises the right methodological questions. Not a general-audience result.

My recommendation: send it to peer review. It's a serious, honest empirical study with a novel protocol, and the concerns are addressable in revision rather than fatal. I'd press for artifact release and either a simulator variation or a more circumscribed conclusion.

Referee Report

2 major / 7 minor

Summary. The paper proposes an end-to-end faithfulness test for natural language explanations of sparse model components. The authors train a transcoder and an SAE on Pythia 160M, generate explanations for their latents with the Paulo et al. (2024) pipeline, and then replace the encoder with an LLM-based simulator that predicts latent activations from the explanation and context. After quantile normalization and TopK post-processing, the simulated activations are patched back into the model, and cross-entropy loss is measured on held-out Pile prompts. The main finding is that replacing all latents with simulated activations increases loss similarly to zeroing the component, and that a trivial predictor always outputting zero activations is almost as good. The authors attribute this to explanations being not specific enough. The same protocol is applied to a residual-stream SAE with similar results.

Significance. If the result stands, the paper provides a valuable negative result for the interpretability community: the current explanation-generation and simulation pipeline does not capture enough information to outperform a zero-ablation baseline. The evaluation is external and uses bootstrap confidence intervals, which is a strength. The code is available, and the comparison to the zero baseline is a clear, falsifiable benchmark. However, the causal attribution to explanation quality is not fully supported, because the simulator is a fixed component and the specificity analysis is computed from the same simulator's predictions. This limits the generality of the conclusion as stated in the abstract and Section 4.1.

major comments (2)

[Abstract and Section 4.1] The abstract concludes that "more detailed explanations are needed" and Section 4.1 is titled "Explanations are not detailed enough." This attribution is underdetermined by the experiments: the pipeline uses a single simulator (Llama 3 Instruct 8B with the 0-9 rating prompt), and the specificity and sensitivity measurements in Section 4.1 are computed from that same simulator's predictions. A simulator that is poorly calibrated or too weak could produce the same result even if the explanations were highly informative. The authors should either add a control that shuffles or randomizes explanations to demonstrate that explanation content matters relative to a no-information baseline, or reframe the conclusion to state that the proposed rewriting pipeline fails to beat the zero-ablation baseline without claiming that explanations themselves are insufficient.
[Section 4 and Appendix B] The sentence in Section 4 that "Using the empirical distribution as the target distribution for quantile normalization significantly improves this performance and rewritten models perform better than zeroing them" refers to a condition in which the quantile normalizer is fit to the same evaluation prompts (Figure A3). This is an in-sample calibration and is not a valid measure of generalization; it also sits in tension with the abstract's headline claim that the pipeline is statistically similar to zero ablation. The manuscript should explicitly flag this result as an in-sample diagnostic, specify which normalizer was used for the main results in Figure 3, and avoid presenting the in-sample improvement as evidence for the pipeline's effectiveness.

minor comments (7)

[Section 3 / Figure 3] The phrase "statistically similar" is not accompanied by the actual loss increases or confidence intervals. Since the central claim is a null result, reporting the numeric differences and their bootstrap intervals is essential for readers to assess whether the differences are small or merely noisy.
[Section 4] The text "a predictor that ignores the explanations and always predicts that every latent is inactive would achieve only slightly more loss than this setup" is an important control but is not shown or quantified in any figure; adding this baseline to Figure 3 or reporting its loss increase numerically would strengthen the negative result.
[Section 4] The sentence "Using the empirical distribution as the target distribution for quantile normalization significantly improves this performance and rewritten models perform better than zeroing them, see Figure" does not specify which figure is meant; it should refer to Figure A3 explicitly.
[Section 4.1] There are minor typos: "We observe than on average" should read "We observe that on average," and "If the LLM predictor only can only achieve" has a duplicated "only."
[Appendix A] The prompt text contains a small grammatical error: "a integer" should be "an integer."
[Figure 2] The caption contains a typo: "wether" should be "whether."
[Section 3 / Figure 3] It is unclear whether the zero-ablation baseline zeroes the entire transcoder output (including the skip connection) or only the sparse component; clarifying this would help readers interpret the comparison.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the zero-vector ablation and held-out Pile prompts are external benchmarks, and the self-cited pipeline is the object under test, not a premise that forces the negative result.

full rationale

The paper's derivation chain is self-contained with respect to its central claim: a transcoder/SAE is trained with an external MSE objective, explanations are produced by the authors' prior pipeline (the object under evaluation), activations are simulated by Llama 3 Instruct 8B, and the resulting loss increase is compared against a zero-vector ablation on held-out Pile prompts. The quantile normalizer is a calibration step that matches marginal CDFs and is explicitly analyzed in Appendix B; it is not fitted to minimize cross-entropy loss, so it does not make the zero-ablation-equivalent result true by construction. The detection-score-based selection is validated empirically against simulation outcomes and is not needed for the all-latents result. The Section 4.1 specificity diagnosis is computed with the same simulator, which limits what can be attributed to explanation content versus simulator capability, but that is an underdetermination caveat rather than a definitional or statistical circularity. The paper's stated limitations (finite sample size for the predicted-activation CDF, bias in empirical quantiles, and the expectation rather than proof of improvement with 100K samples) are openly disclosed. Self-citations to Paulo et al. (2024) supply the explanation pipeline under test; because the paper's headline finding is a negative result about that pipeline, the self-citation does not load-bear in favor of the conclusion. Score 1 reflects the presence of a minor self-citation that is not load-bearing.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The central claim rests on two data-derived calibration choices (TopK sparsity k and per-feature quantile normalization CDFs) and on several domain assumptions about the faithfulness of the transcoder, the adequacy of the zero-vector baseline, the representativeness of the Llama 3 simulator, and the sufficiency of cross-entropy loss as a behavioral metric. No new theoretical entities are introduced.

free parameters (2)

TopK sparsity k = 32
Hand-chosen sparsity level for transcoder and SAE latents; the false-positive analysis in Section 4.1 (320 predicted vs. 32 actual active latents) is computed relative to this k.
Per-feature quantile normalization CDFs = Empirical CDFs from 10M true activations and 10K predicted activations
The end-to-end loss of the rewritten model depends on this calibration transform; Appendix B shows the result is sensitive to the sample size (1K vs. 10K predictions).

assumptions (6)

standard math Quantile normalization is the optimal transport map for matching marginal distributions.
Used in Section 2.1 to justify the calibration transform; relies on Santambrogio (2015).
domain assumption The transcoder with skip connection is a sufficiently faithful approximation of the original MLP for the rewriting evaluation.
Replacing the MLP with the transcoder already raises loss to an early Pythia checkpoint (Section 3); the evaluation compares simulated activations against the transcoder, not the original MLP.
domain assumption Zeroing the transcoder output is a meaningful baseline for 'removing the MLP block'.
Used as the reference baseline in Figures 3 and A3; the skip connection means zeroing the transcoder removes both nonlinear and linear paths, which is not explicitly discussed.
domain assumption Llama 3 Instruct 8B, prompted with the 0-9 rating scheme, is a faithful and strong-enough reader of the natural language explanations.
The entire simulation rests on using this model's expected output as the predicted activation; no alternative simulator is tested.
domain assumption Cross-entropy loss on a sample of Pile prompts is a sufficient metric of model behavior for the faithfulness claim.
The paper evaluates next-token prediction CE loss only; other behaviors such as factual consistency or generation quality are not measured.
domain assumption Per-feature quantile normalization is a valid calibration: matching marginal distributions suffices.
Used in Section 2.1; the paper acknowledges inverse-CDF bias and finite-sample issues but assumes monotone marginal matching is the right correction.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Partially Rewriting a Transformer in Natural Language." pith.science (2026). https://pith.science/paper/GGFVPWQ3

@misc{pith2026250118838,
  author       = {Pith},
  title        = {Pith review of: Partially Rewriting a Transformer in Natural Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GGFVPWQ3}},
  note         = {Machine review of arXiv:2501.18838}
}

read the original abstract

The greatest ambition of mechanistic interpretability is to completely rewrite deep neural networks in a format that is more amenable to human understanding, while preserving their behavior and performance. In this paper, we attempt to partially rewrite a large language model using simple natural language explanations. We first approximate one of the feedforward networks in the LLM with a wider MLP with sparsely activating neurons - a transcoder - and use an automated interpretability pipeline to generate explanations for these neurons. We then replace the first layer of this sparse MLP with an LLM-based simulator, which predicts the activation of each neuron given its explanation and the surrounding context. Finally, we measure the degree to which these modifications distort the model's final output. With our pipeline, the model's increase in loss is statistically similar to entirely replacing the sparse MLP output with the zero vector. We employ the same protocol, this time using a sparse autoencoder, on the residual stream of the same layer and obtain similar results. These results suggest that more detailed explanations are needed to improve performance substantially above the zero ablation baseline.

Figures

Figures reproduced from arXiv: 2501.18838 by the authors.

**Figure 1.** Distribution of predicted activations for all latents. On the left we compare the distribution of predicted activations before normalization, and on the right we show what the distribution looks like after quantile normalization. Before normalization, the predictor model systematically over-predicts high activation values by multiple orders of magnitude. Quantile normalization primarily has the effect of enforcing a… view at source ↗

**Figure 2.** Partially rewriting an LLM. After training a Transcoder, or any type of SAE, we generate explanations for all the latents using the contexts where that latent is active. An LLM is tasked to summarize or otherwise find patterns in the activations and output a simple, single sentence explanation for that latent. These explanations are used by another instance of an LLM to predict wether the latent should be active in … view at source ↗

**Figure 3.** Cross entropy loss increase for different fractions of transcoder and SAE substitution. We compute the CE loss over 10K prompts, for the transcoder (left) and SAE (right) respectively, by substituting parts of the encoder with natural language explanations. Bars in green show the average loss increase when choosing the top scoring latents for replacement. Bars in orange show the average loss increase when randomly s… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Detection score predicts sensitivity and specificity. Binning explanations by their scores makes it evident that high-scoring explanations are more specific and sensitive. for a total of 327 million predictions, an expensive endeavor. We expect that, if performed on 10…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Evaluating SAE interpretability without explanations
cs.LG 2025-07 conditional novelty 5.0 of 10

SAE latent interpretability can be scored directly from activation examples via intruder detection and embedding clustering, with LLM scores correlating strongly with human scores.

Reference graph

Works this paper leans on

16 extracted references · 5 canonical work pages · cited by 1 Pith paper

[1]

These features of text are normally identified by looking for specific words or patterns in the text

The higher the score the better the explanation is, and you should be more certain of your response (positive or negative). These features of text are normally identified by looking for specific words or patterns in the text. There are many features associated with a single token, and sometimes the feature is related with the previous token or context. Yo...

work page 2024
[3]

Sparse autoencoders find highly inter- pretable features in language models

Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly inter- pretable features in language models. arXiv preprint arXiv:2309.08600,

arXiv
[4]

The llama 3 herd of models

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

arXiv
[6]

Toy models of superposition

Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., et al. Toy models of superposition. arXiv preprint arXiv:2209.10652,

arXiv
[9]

Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning

Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

arXiv
[11]

C., Kheirkhah, T

Gurnee, W., Horsley, T., Guo, Z. C., Kheirkhah, T. R., Sun, Q., Hathaway, W., Nanda, N., and Bertsimas, D. Uni- versal neurons in gpt2 language models. arXiv preprint arXiv:2401.12181,

arXiv
[14]

doi: 10.23915/distill.00024.001

ISSN 2476-0757. doi: 10.23915/distill.00024.001. URL http://dx.doi. org/10.23915/distill.00024.001. Paulo, G., Mallen, A., Juang, C., and Belrose, N. Automati- cally interpreting millions of features in large language models. arXiv preprint arXiv:2410.13928,

arXiv
[15]

male pronouns

URL https: //transformer-circuits.pub/2024/ scaling-monosemanticity/index.html. 7 Partially Rewriting a Transformer in Natural Language A. Simulation prompt You are an intelligent and meticulous linguistics researcher. You will be given a certain explanation of a feature of text, such as "male pronouns" or "text with negative sentiment" and examples of te...

work page 2024

Show all 16 references

[1996]

Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

arXiv
[2019]

com/posts/X2i9dQQK3gETCyqh2/ chris-olah-s-views-on-agi-safety

URL https://www.lesswrong. com/posts/X2i9dQQK3gETCyqh2/ chris-olah-s-views-on-agi-safety . Accessed: 2025-01-15. Hyndman, R. J. and Fan, Y . Sample quantiles in statistical packages. The American Statistician , 50(4):361–365,

2025
[2020]

D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J

Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scal- ing and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093,

arXiv
[2021]

com/posts/JKj5Krff5oKMb8TjT/ imitative-generalisation-aka-learning-the-prior-1

URL https://www.lesswrong. com/posts/JKj5Krff5oKMb8TjT/ imitative-generalisation-aka-learning-the-prior-1 . Accessed: 2025-01-15. Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. ...

2025
[2022]

The pile: An 800gb dataset of diverse text for language modeling

Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 ,

arXiv
[2023]

Language models can explain neurons in language models

Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J., Wu, J., and Saunders, W. Language models can explain neurons in language models. URL https://openaipublic. blob. core. windows. net/neuron-explainer/paper/index. html.(Date accessed...

2023
[2024]

Transcoders find interpretable llm feature circuits

Dunefsky, J., Chlenski, P., and Nanda, N. Transcoders find interpretable llm feature circuits. arXiv preprint arXiv:2406.11944,

arXiv
[2025]

Finding neurons in a haystack: Case studies with sparse probing

Gurnee, W., Nanda, N., Pauly, M., Harvey, K., Troit- skii, D., and Bertsimas, D. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610,

arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

These features of text are normally identified by looking for specific words or patterns in the text

The higher the score the better the explanation is, and you should be more certain of your response (positive or negative). These features of text are normally identified by looking for specific words or patterns in the text. There are many features associated with a single token, and sometimes the feature is related with the previous token or context. Yo...

work page 2024

[2] [3]

Sparse autoencoders find highly inter- pretable features in language models

Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly inter- pretable features in language models. arXiv preprint arXiv:2309.08600,

arXiv

[3] [4]

The llama 3 herd of models

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

arXiv

[4] [6]

Toy models of superposition

Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., et al. Toy models of superposition. arXiv preprint arXiv:2209.10652,

arXiv

[5] [9]

Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning

Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

arXiv

[6] [11]

C., Kheirkhah, T

Gurnee, W., Horsley, T., Guo, Z. C., Kheirkhah, T. R., Sun, Q., Hathaway, W., Nanda, N., and Bertsimas, D. Uni- versal neurons in gpt2 language models. arXiv preprint arXiv:2401.12181,

arXiv

[7] [14]

doi: 10.23915/distill.00024.001

ISSN 2476-0757. doi: 10.23915/distill.00024.001. URL http://dx.doi. org/10.23915/distill.00024.001. Paulo, G., Mallen, A., Juang, C., and Belrose, N. Automati- cally interpreting millions of features in large language models. arXiv preprint arXiv:2410.13928,

arXiv

[8] [15]

male pronouns

URL https: //transformer-circuits.pub/2024/ scaling-monosemanticity/index.html. 7 Partially Rewriting a Transformer in Natural Language A. Simulation prompt You are an intelligent and meticulous linguistics researcher. You will be given a certain explanation of a feature of text, such as "male pronouns" or "text with negative sentiment" and examples of te...

work page 2024