Pith. sign in

REVIEW 5 major objections 4 minor 41 references

Cross-Layer Discrete Concept Discovery for Interpreting Language Models

T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CLVQ-VAE claims to extract discrete concepts that faithfully track an LLM's decisions, with removing one concept vector dropping probe accuracy from 0.7604 to 0.0782.

desk verdict New discrete cross-layer concept bottleneck, but the headline faithfulness claim is about a probe, not the model—and the abstract promises an LLM-judge experiment the manuscript never reports. read the letter →

arxiv 2506.20040 v3 pith:7KEXZSYS submitted 2025-06-24 cs.LG cs.AIcs.CL

Ankur Garg , Xuemin Yu , Hassan Sajjad , Samira Ebrahimi Kahou This is my paper
classification cs.LGcs.AIcs.CL
keywords interpretabilityconceptdiscoveryvectorquantizationVQ-VAEcross-layeranalysistransformerresidualstreamfaithfulnessevaluationlanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that language-model features duplicated across transformer layers can be folded into a small set of discrete concept vectors, and that these vectors explain why the model makes its predictions. It proposes CLVQ-VAE, a cross-layer vector-quantized autoencoder that reconstructs a higher layer's activations from a lower layer through a discrete bottleneck. The paper argues that the discrete codebook captures persistent cross-layer transformations, and supports this with perturbation, LLM-judge, and human studies. A sympathetic reader would care because discrete, human-checkable concepts are a step toward knowing what a model is actually doing rather than only that it works.

What carries the argument

The machinery is a cross-layer vector-quantized autoencoder. A lower layer's hidden states pass through an adaptive residual encoder $z_e=(1-\alpha)x+\alpha\,\mathrm{LN}(Wx+b)$ with $\alpha\le 0.5$, are discretized by sampling from the top-$k$ nearest codebook vectors at temperature $\tau$, and are decoded by a six-layer transformer decoder that reconstructs the higher layer's hidden states. The codebook is initialized with scaled-spherical k-means++ and updated with exponential moving averages; the reconstruction loss forces the discrete vectors to preserve exactly the features that persist across layers, which is the paper's operational definition of a concept.

What would settle it

Re-run the original fine-tuned model on [CLS] embeddings with the identified most-salient concept vector removed by orthogonal projection, with no probe in between. If the model's own accuracy drops far less than the 0.7604-to-0.0782 probe drop, or not at all, the central faithfulness claim is refuted.

Watch

Extended reading notes

Core claim

The central claim is that discretizing the transformation between two transformer layers yields human-legible concepts that faithfully track the model's reasoning, because the vector-quantization bottleneck collapses duplicated residual-stream features into single codebook vectors. The paper reports that removing the single most salient concept vector from the [CLS] embedding drops probe accuracy from 0.7604 to 0.0782 on RoBERTa-ERASER (clustering baseline: 0.6271), that LLM judges rank its concepts first in 66.7% of comparisons, and that human annotators predict the model's sentiment from CLVQ-VAE word clouds 78.20% of the time versus 54.14% for clustering.

Load-bearing premise

The faithfulness result assumes that a separately trained two-layer probe on the lower-layer [CLS] embedding reacts to the same directions the original language model uses to classify; if that probe does not mirror the model, the reported accuracy drops do not prove the concepts caused the model's predictions.

Editorial extensions

If this is right

  • Concept removal results imply that a single discrete codebook vector can carry much of the task-relevant signal for a given token; removing it can reduce probe accuracy by more than an order of magnitude.
  • The method transfers across encoder-only and decoder-only architectures, so the same discrete bottleneck can be applied to autoregressive models that lack a [CLS] token.
  • Random perturbation leaves accuracy nearly unchanged, so the measured drops are specific to identified concept directions rather than generic embedding noise.
  • Word clouds for false positives and false negatives visibly group the terms that led the model astray, making discrete concepts a usable tool for inspecting failure modes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same codebook could double as a steering mechanism; replacing a selected concept vector instead of deleting it would let a user reshape model behavior in a direction visible in the word cloud.
  • Editorial inference: because the faithfulness evaluation runs through a probe rather than the original model, the reported drops should be treated as upper bounds on true causal influence until direct model-level perturbation is tested.
  • Editorial inference: the cross-layer collapse suggests a testable transfer property—if the same concept vector appears for the same word across seeds or checkpoints, the codebook captures a stable feature rather than a training artifact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes CLVQ-VAE, a vector-quantized transcoder that maps lower-layer transformer representations (e.g., layer 8) to higher-layer representations (e.g., layer 12) through a discrete codebook. The method combines an adaptive residual encoder, temperature-controlled top-k codebook sampling, EMA updates, and scaled-spherical k-means++ initialization. The authors evaluate concept faithfulness by removing the codebook vector associated with the most salient token from the [CLS] embedding and measuring the resulting accuracy drop of a separately trained 2-layer probe, comparing against clustering, single-layer VQ-VAE, and SAE baselines. They also report a small human study comparing word-cloud visualizations, and the abstract additionally claims an LLM-judge ranking experiment. The central claim is that the discovered discrete concepts faithfully capture the transformations driving the model's predictions.

Significance. If the faithfulness claim were properly supported, the paper would make a useful contribution: cross-layer discrete concept discovery is timely, the methodological components are clearly described, and the human evaluation provides an independent behavioral signal that partially mitigates concerns about circularity. The paper also includes detailed hyperparameters and several ablation tables (Tables 2, 3, 7-10), which is a strength. However, the central quantitative evidence for faithfulness is currently invalid as stated, because the original language model is never run on the perturbed representations. The abstract also advertises an LLM-judge experiment that does not appear in the manuscript, and the headline 'outperforms' claim is not uniformly supported by Table 1. These issues must be resolved before the paper's main thesis can be evaluated.

major comments (5)
  1. [4.1.1] The faithfulness evaluation never runs the original language model on the perturbed representations. The procedure trains a 2-layer probe on unmodified lower-layer [CLS] embeddings and measures probe accuracy on original, perturbed, and random-perturbed embeddings; the original transformer is not executed on any perturbed input. Consequently, the abstract's claim that 'removing identified concepts drops model accuracy by up to 93%' is not supported by the reported experiment. The result shows only that the probe relies on the removed direction, not that the original model does. To support the stated claim, the authors should re-run the fine-tuned model (for example, by replacing the layer-8 [CLS] embedding and forwarding through layers 9-12 to the task head) or, if the claim is restricted, explicitly reframe the metric as probe faithfulness rather than model accuracy.
  2. [Abstract] The abstract claims that 'LLM judges rank our concepts first in 66.7% of comparisons,' but no LLM-judge experiment appears anywhere in the manuscript. This is one of the three advertised evaluation axes and a headline quantitative result. The authors must either add the full experiment and its results or remove the claim from the abstract and introduction.
  3. [4.1.1] The procedure for identifying the 'most salient token' is not specified. The perturbation methodology depends on this selection for every method and every model-dataset combination, but the text only says that the most important concept vector associated with the most salient token is removed. Without a precise definition of the saliency attribution, Tables 1 and 3 are not reproducible and the comparison across methods cannot be validated. This missing detail is load-bearing because the saliency choice determines which concept vector is ablated.
  4. [Table 1] The claim that CLVQ-VAE outperforms baselines is not uniformly supported by the reported numbers. In Table 1, on RoBERTa-Jigsaw clustering achieves 0.5628 while CLVQ-VAE achieves 0.6456, and the text in Section 4.1.2 acknowledges that clustering outperforms CLVQ-VAE in that condition. The abstract and introduction should be qualified to state that CLVQ-VAE is best on three of the four encoder-model-dataset combinations, rather than claiming uniform superiority.
  5. [7 (Limitations)] The Limitations section honestly notes that the perturbation metric lacks sensitivity to hyperparameter changes (Table 10) and that the human study is small, but it does not acknowledge the more fundamental issue that the faithfulness metric evaluates a probe rather than the original model. Since the probe substitution is the paper's primary quantitative support, this limitation should be stated and addressed either by a direct model intervention or by a clearly scoped claim.
minor comments (4)
  1. [4.3.1] The human evaluation uses only 19 sentences and 14 annotators, and the paper reports no confidence intervals or significance test for the 78.20% versus 54.14% difference in model alignment rate. A statistical comparison would help assess whether the observed gap is reliable.
  2. [References] Several references are duplicated or inconsistent: Dunefsky et al. 2024a and 2024b appear to be the same arXiv preprint, as do Lindsey et al. 2024 and 2025. Please consolidate the bibliography.
  3. [Equation (4)] Equation (4) defines the sampling distribution over the top-k set, but the notation is slightly ambiguous because the numerator and denominator both use the top-k set while the equation is written as if over all j. Clarify that the softmax is renormalized over the top-k candidates.
  4. [Abstract] The abstract states a drop of 'up to 93%' in model accuracy, but the largest drop calculable from Tables 1 and 5 is about 89.7% (RoBERTa-ERASER: 0.7604 to 0.0782). The number should be reconciled with the reported tables.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the probe-based faithfulness evaluation is an external-validity gap, not a by-construction reduction.

full rationale

The paper's faithfulness evaluation trains a separate 2-layer probe on original [CLS] embeddings and then measures probe accuracy after removing a concept vector; the original transformer is never re-run on perturbed representations. This is a proxy-validity weakness: the abstract's phrase 'model accuracy' overstates what was measured, since the experiment only measures probe accuracy. However, this is not circular by the standards required here. The probe is fitted independently of the perturbation outcomes, the CLVQ-VAE concept vectors are learned from a cross-layer reconstruction objective (L_rec = ||y - ŷ||^2_2) rather than from probe accuracy, and the random-perturbation control provides a comparison baseline. The self-citations present (Yu et al. 2024 as the clustering baseline, NeuroX for activation extraction, and Sajjad et al. surveys) are not load-bearing: no central premise depends on an unverified self-citation, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in through citation. Section 7 itself concedes the faithfulness metric's limited sensitivity, which further confirms that the evaluation choice is a stated limitation rather than a hidden identity. The human evaluation additionally supplies an independent behavioral check on the concept visualizations, so the central claim does not reduce to its own inputs by construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The framework relies on several fitted hyperparameters, a selected layer pair, and two strong evaluation assumptions: probe-as-model and projection-as-ablation. These are not derived from theory, and the core empirical claim depends on them.

free parameters (5)
  • Codebook size K = 400
    Selected from the perplexity and utilization analysis in A.3.1; the faithfulness metric is not very sensitive to this choice.
  • Top-k = 5
    Chosen from Table 2 based on validation perplexity; Table 10 shows faithfulness is insensitive to this.
  • Temperature tau = 1.0
    Chosen from Table 2 as a balance between exploration and stability.
  • Commitment cost beta = 0.1
    Standard VQ-VAE value, tuned with reference to Table 9.
  • Alpha cap = 0.5
    Empirically set in Table 8 because larger alpha reduced codebook utilization.
assumptions (4)
  • domain assumption The residual stream linearly mixes and duplicates features across adjacent layers, so cross-layer analysis is necessary.
    Stated in the Introduction and used to justify the cross-layer design.
  • domain assumption Layer pair 8-12 captures the most meaningful transformation from semantic features to task-specific representations.
    Selected empirically in A.4 using the same faithfulness metric as the main evaluation.
  • ad hoc to paper A probe trained on lower-layer CLS embeddings is a faithful proxy for the original language model's predictions in the ablation test.
    This is the load-bearing assumption in Section 4.1.1; the original model is never re-run.
  • ad hoc to paper Orthogonally projecting a codebook vector out of the CLS embedding constitutes a valid removal of that concept.
    Used in Section 4.1.1 with no theoretical justification connecting token-level concepts to CLS directions.
invented entities (1)
  • Concept vectors (discrete codebook entries) independent evidence
    purpose: Represent cross-layer features as discrete, human-interpretable units for model explanation.
    The human evaluation provides an external behavioral check: annotators infer model predictions from concept word clouds more accurately than from clustering. No other external measurement is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-Layer Discrete Concept Discovery for Interpreting Language Models." pith.science (2026). https://pith.science/paper/7KEXZSYS

@misc{pith2026250620040,
  author       = {Pith},
  title        = {Pith review of: Cross-Layer Discrete Concept Discovery for Interpreting Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7KEXZSYS}},
  note         = {Machine review of arXiv:2506.20040}
}
read the original abstract

Interpreting language models remains challenging due to the existence of residual stream, which linearly mixes and duplicates features across adjacent layers, causing single-layer analyses to miss this cross-layer structure. Cross-layer sparse autoencoders (SAEs) address layer mixing but operate in continuous space, where concepts split across many neurons without clear boundaries. We introduce Cross-Layer Vector Quantized-Variational Autoencoder (CLVQ-VAE), a novel framework which maps representations from a lower layer to a higher layer through a discrete vector-quantization bottleneck, collapsing duplicated residual-stream features into compact, interpretable concept vectors. Our approach combines top-k temperature-based sampling with exponential moving average (EMA) codebook updates, providing controlled exploration of the discrete latent space while maintaining codebook diversity. Across both encoder- and decoder-based models on ERASER-Movie, Jigsaw, and AGNews, CLVQ-VAE outperforms clustering, single-layer vector quantized-variational autoencoder (VQ-VAE), and sparse autoencoder (SAE) baselines across three evaluation axes: removing identified concepts drops model accuracy by up to 93%, LLM judges rank our concepts first in 66.7% of comparisons, and human annotators recover model predictions from our visualizations with 78% accuracy versus 54% for clustering.

Figures

Figures reproduced from arXiv: 2506.20040 by the authors.

Figure 1
Figure 1. Overview of the CLVQ-VAE framework for cross-layer concept discovery. Lower [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. False negative example (Model: 0, Ground Truth: 1) showing concept clusters [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗
Figure 3
Figure 3. False positive example (Model: 1, Ground Truth: 0) showing terms of moderate [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: True negative example (Model: 0, Ground Truth: 0) showing terms expressing [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: True positive example (Model: 1, Ground Truth: 1) showing positive descriptors [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 11 canonical work pages

  1. [2]

    doi: 10.18653/v1/2022.findings-emnlp.502

    Associa- tion for Computational Linguistics. doi: 10.18653/v1/2022.findings-emnlp.502. URL https://aclanthology.org/2022.findings-emnlp.502/. Nikita Balagansky, Ian Maksimov, and Daniil Gavrilov. Mechanistic permutability: Match features across layers,

    work page doi:10.18653/v1/2022.findings-emnlp.502 2022

  2. [5]

    doi: 10.18653/v1/2022

    Association for Computational Linguistics. doi: 10.18653/v1/2022. emnlp-main.455. URL https://aclanthology.org/2022.emnlp-main.455/. M. Celebi, H. Kingravi, and P . Vela. A comparative study of efficient initialization methods for the k-means clustering algorithm. Expert Systems With Applications, 40:200–210,

    doi:10.18653/v1/2022 2022

  3. [6]

    decent imitation of Brando’s godfather

    Unless otherwise noted, all experiments use this configuration. Adaptive α is capped at 0.5, and all weights are initialized using standard PyTorch defaults, with the encoder residual path initialized as an identity transformation. 15 Preprint. Under review. Component Hyperparameter Value Model Architecture Embedding dim (d) 768 Codebook size (K) 400 Comm...

    work page

  4. [11]

    Amirata Ghorbani, James Wexler, James Zou, and Been Kim

    URL https://arxiv.org/abs/2505.16077. Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept- based explanations,

    arXiv

  5. [12]

    Antonio Gulli

    URL https://arxiv.org/abs/1902.03129. Antonio Gulli. Ag’s corpus of news articles,

    arXiv 1902

  6. [14]

    Muhammad Umair Haider, Hammad Rizwan, Hassan Sajjad, Peizhong Ju, and A

    URL https://arxiv.org/abs/2305.01610. Muhammad Umair Haider, Hammad Rizwan, Hassan Sajjad, Peizhong Ju, and A. B. Sid- dique. Neurons speak in ranges: Breaking free from discrete neuronal attribution,

    arXiv

  7. [15]

    Neurons Speak in Ranges: Breaking Free from Discrete Neuronal Attribution

    URL https://arxiv.org/abs/2502.06809. Lifu Huang and Heng Ji. Semi-supervised new event type induction and event detection. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.),Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 718–724, Online, November

    work page Pith review arXiv 2020

  8. [16]

    doi: 10.18653/v1/ 2020.emnlp-main.53

    Association for Computational Linguistics. doi: 10.18653/v1/ 2020.emnlp-main.53. URL https://aclanthology.org/2020.emnlp-main.53/. Mengqi Huang, Zhendong Mao, Zhuowei Chen, and Yongdong Zhang. Towards accu- rate image coding: Improved autoregressive image generation with dynamic vector quantization,

    doi:10.18653/v1/ 2020

Show all 41 references
  1. [17]

    Ruben H¨arle, Felix Friedrich, Manuel Brack, Bj ¨orn Deiseroth, Patrick Schramowski, and Kristian Kersting

    URL https://arxiv.org/abs/2305.11718. Ruben H¨arle, Felix Friedrich, Manuel Brack, Bj ¨orn Deiseroth, Patrick Schramowski, and Kristian Kersting. Scar: Sparse conditioned autoencoders for concept detection and steering in llms,

    arXiv

  2. [19]

    Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, and Neel Nanda

    URL https://arxiv.org/ abs/2305.06754. Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, and Neel Nanda. Are sparse autoencoders useful? a case study in sparse probing. arXiv preprint arXiv:2502.16681,

    arXiv

  3. [20]

    Andrei Kapishnikov, Subhashini Venugopalan, Besim Avci, Ben Wedin, Michael Terry, and Tolga Bolukbasi

    URL https://arxiv.org/abs/2502.16681. Andrei Kapishnikov, Subhashini Venugopalan, Besim Avci, Ben Wedin, Michael Terry, and Tolga Bolukbasi. Guided integrated gradients: An adaptive path method for removing noise. CoRR, abs/2106.09788,

    arXiv

  4. [21]

    Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al

    URL https://arxiv.org/abs/2106.09788. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine ...

    arXiv

  5. [22]

    Michael Lan, Philip Torr, Austin Meek, Ashkan Khakzar, David Krueger, and Fazl Barez

    URL https://arxiv.org/abs/2207.04153. Michael Lan, Philip Torr, Austin Meek, Ashkan Khakzar, David Krueger, and Fazl Barez. Sparse autoencoders reveal universal feature spaces across large language models,

    arXiv

  6. [23]

    Daniil Laptev, Nikita Balagansky, Yaroslav Aksenov, and Daniil Gavrilov

    URL https://arxiv.org/abs/2410.06981. Daniil Laptev, Nikita Balagansky, Yaroslav Aksenov, and Daniil Gavrilov. Analyze feature flow to enhance interpretation and steering in language models,

    arXiv

  7. [24]

    Jack Lindsey, Adly Templeton, Jonathan Marcus, Tom Conerly, Joshua Batson, and Chris Olah

    URL https: //arxiv.org/abs/2502.03032. Jack Lindsey, Adly Templeton, Jonathan Marcus, Tom Conerly, Joshua Batson, and Chris Olah. Sparse crosscoders for cross-layer features and model diffing. https: //transformer-circuits.pub/2024/crosscoders/index.html,

    2024 arXiv

  8. [25]

    12 Preprint

    URL https://transformer-circuits.pub/2024/crosscoders/index.html. 12 Preprint. Under review. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. In Jill Burstein, Christy D...

    2024 arXiv

  9. [27]

    Bo Pang and Lillian Lee

    URL https://arxiv.org/abs/2504.02922. Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, ACL ’04, pp. 271–es, USA,

  10. [29]

    Dheeraj Rajagopal, Vidhisha Balachandran, Eduard H Hovy, and Yulia Tsvetkov

    URL https://arxiv.org/abs/ 2501.16615. Dheeraj Rajagopal, Vidhisha Balachandran, Eduard H Hovy, and Yulia Tsvetkov. SELF- EXPLAIN: A self-explaining architecture for neural text classifiers. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), P...

    2021 arXiv

  11. [30]

    doi: 10.18653/v1/2021.emnlp-main.64

    Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.64. URL https://aclanthology.org/ 2021.emnlp-main.64/. Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2,

    2021 doi

  12. [31]

    Hassan Sajjad, Nadir Durrani, and Fahim Dalvi

    URL https://arxiv.org/abs/1906.00446. Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. Neuron-level interpretation of deep nlp models: A survey. Transactions of the Association for Computational Linguistics, 2022a. Hassan Sajjad, Nadir Durrani, Fahim Dalvi, Firoj Alam, Abdul Kha...

    1906 arXiv

  13. [32]

    doi: 10.18653/v1/2021.acl-long.330

    Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.330. URL https://aclanthology.org/2021. acl-long.330/. Wei Shi, Sihang Li, Tao Liang, Mingyang Wan, Gojun Ma, Xiang Wang, and Xiangnan He. Route sparse autoencoder to interpret large language models,

    2021 doi

  14. [33]

    13 Preprint

    URL https: //arxiv.org/abs/2503.08200. 13 Preprint. Under review. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. CoRR, abs/1703.01365,

    arXiv

  15. [34]

    Yuhta Takida, Takashi Shibuya, WeiHsiang Liao, Chieh-Hsin Lai, Junki Ohmura, Toshimitsu Uesaka, Naoki Murata, Shusuke Takahashi, Toshiyuki Kumakura, and Yuki Mitsufuji

    URL http://arxiv.org/abs/1703.01365. Yuhta Takida, Takashi Shibuya, WeiHsiang Liao, Chieh-Hsin Lai, Junki Ohmura, Toshimitsu Uesaka, Naoki Murata, Shusuke Takahashi, Toshiyuki Kumakura, and Yuki Mitsufuji. Sq-vae: Variational bayes on discrete representation with self-annealed...

    arXiv

  16. [35]

    Transformer Circuits Team

    URL https://arxiv.org/abs/2205.07547. Transformer Circuits Team. Sparse crosscoders for cross-layer features and model diffing. Distill,

    arXiv

  17. [37]

    Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu

    URL https://arxiv.org/abs/2307.09288. Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning,

    arXiv

  18. [39]

    Yi-Fu Wu, Minseung Lee, and Sungjin Ahn

    URL https://arxiv.org/abs/1807.04629. Yi-Fu Wu, Minseung Lee, and Sungjin Ahn. Neural language of thought models,

    arXiv

  19. [40]

    Xuemin Yu, Fahim Dalvi, Nadir Durrani, Marzia Nouri, and Hassan Sajjad

    URL https://arxiv.org/abs/2402.01203. Xuemin Yu, Fahim Dalvi, Nadir Durrani, Marzia Nouri, and Hassan Sajjad. Latent concept- based explanation of nlp models,

    arXiv

  20. [41]

    Yu Zhang, Peter Ti ˇno, Aleˇs Leonardis, and Ke Tang

    URL https://arxiv.org/abs/2404.12545. Yu Zhang, Peter Ti ˇno, Aleˇs Leonardis, and Ke Tang. A survey on neural network inter- pretability. arXiv preprint arXiv:–,

    arXiv

  21. [42]

    doi: 10.18653/v1/2023.acl-long.261

    Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.261. URL https://aclanthology.org/2023. acl-long.261/. Łukasz Kaiser, Aurko Roy, Ashish Vaswani, Niki Parmar, Samy Bengio, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discre...

    2023 doi

  22. [43]

    14 Preprint

    URL https://arxiv.org/abs/1803.03382. 14 Preprint. Under review. A Appendix A.1 EMA update details During training, we perform temperature-based top-k sampling to select codebook vectors, then apply EMA updates using hard assignments: For each codebook vector ej: N(t) j = γN(t...

    arXiv

  23. [2004]

    doi: 10.3115/1218955.1218990

    Association for Computational Linguistics. doi: 10.3115/1218955.1218990. URL https://doi.org/10. 3115/1218955.1218990. Gonc ¸alo Paulo and Nora Belrose. Sparse autoencoders trained on the same data learn different features. arXiv preprint arXiv:2501.16615,

  24. [2013]

    cjadams, Jeffrey Sorensen, Julia Elliott, Lucas Dixon, Mark McDonald, nithum, and Will Cukierski

    doi: 10.1016/j.eswa.2012.07.021. cjadams, Jeffrey Sorensen, Julia Elliott, Lucas Dixon, Mark McDonald, nithum, and Will Cukierski. Toxic comment classification challenge,

    2012 doi

  25. [2017]

    doi: 10.18653/v1/P17-1080

    Association for Computational Linguistics. doi: 10.18653/v1/P17-1080. URL https://aclanthology. org/P17-1080/. Rishabh Bhardwaj, Amrita Saha, Steven C.H. Hoi, and Soujanya Poria. Vector-quantized input-contextualized soft prompts for natural language understanding. In Yoav Gol...

    doi

  26. [2018]

    Hanwei Wu and Markus Flierl

    URL https://arxiv.org/abs/1711.00937. Hanwei Wu and Markus Flierl. Learning product codebooks using vector quantized autoencoders for image retrieval,

    arXiv

  27. [2019]

    doi: 10.18653/v1/N19-1423

    Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423. Jesse Dodge, Maarten Sap, Ana Marasovi´c, William Agnew, Gabriel Ilharco, Dirk Groen- eveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A c...

    doi

  28. [2020]

    Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas

    URL https://arxiv.org/abs/2006.08101. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing,

    2006 arXiv

  29. [2021]

    doi: 10.18653/v1/ 2021.emnlp-main.98

    Association for Computational Linguistics. doi: 10.18653/v1/ 2021.emnlp-main.98. URL https://aclanthology.org/2021.emnlp-main.98/. Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable llm feature circuits, 2024a. URL https://arxiv.org/abs/2406.1194...

    2021 arXiv

  30. [2022]

    David Arps, Younes Samih, Laura Kallmeyer, and Hassan Sajjad

    URL https://openreview.net/forum?id=8uz0EWPQIMu. David Arps, Younes Samih, Laura Kallmeyer, and Hassan Sajjad. Probing for con- stituency structure in neural language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computation...

    2022

  31. [2023]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

    Association for Computational Linguistics. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association...

    2019

  32. [2024]

    Fanny Jourdan, Agustin Picard, Thomas Fel, Laurent Risser, Jean Michel Loubes, and Nicholas Asher

    URL https://arxiv.org/abs/2411.07122. Fanny Jourdan, Agustin Picard, Thomas Fel, Laurent Risser, Jean Michel Loubes, and Nicholas Asher. Cockatiel: Continuous concept ranked attribution with interpretable elements for explaining neural net classifiers on nlp tasks,

    arXiv

  33. [2025]

    Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass

    URL https://arxiv.org/abs/2410.07656. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. What do neural machine translation models learn about morphology? In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Assoc...

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.