Vector Quantized Latent Concepts: A Scalable Alternative to Clustering-Based Concept Discovery

Ankur Garg; Hassan Sajjad; Samira Ebrahimi Kahou; Xuemin Yu

REVIEW 4 major objections 4 minor 16 references

A vector-quantized codebook of latent concept vectors can discover interpretable concepts in frozen LLM hidden states at near-linear cost, matching hierarchical clustering in faithfulness and beating K-Means in plausibility.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 05:15 UTC pith:THBFZXVJ

load-bearing objection VQLC's scalability win is real and worth peer review, but the faithfulness metric's space mismatch and cherry-picked seeds undercut the quality claims. the 4 major comments →

arxiv 2602.02726 v2 pith:THBFZXVJ submitted 2026-02-02 cs.LG cs.CL

Vector Quantized Latent Concepts: A Scalable Alternative to Clustering-Based Concept Discovery

Xuemin Yu , Ankur Garg , Samira Ebrahimi Kahou , Hassan Sajjad This is my paper

classification cs.LG cs.CL

keywords vector quantizationlatent concept discoveryLLM interpretabilityconcept-based explanationsscalabilityfaithfulness evaluationcodebook learningclustering alternatives

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that vector quantization, a standard technique for learning discrete codes, can replace clustering as the engine of latent concept discovery in large language models. Existing clustering methods face a blunt trade-off: hierarchical clustering produces coherent concepts but needs quadratic memory, while K-Means scales but returns shallow, frequency-dominated clusters. The paper's proposal, VQLC, learns a codebook of discrete concept vectors on frozen hidden states and assigns each token to the nearest vector; across 12 dataset-model settings it stays close to K-Means in cost, scales past the point where hierarchical clustering runs out of memory, and produces concepts judged comparable in faithfulness and more interpretable than K-Means. If correct, this gives practitioners a practical way to explain model predictions on very large datasets without sacrificing the semantic quality that made hierarchical clustering attractive.

Core claim

VQLC treats each learnable codebook vector as a latent concept vector: tokens mapped to the same codebook vector collectively form the content of a concept. Frozen hidden states pass through a lightweight adaptive residual encoder, are quantized to the nearest of K=400 codebook vectors using cosine distance (with top-k temperature sampling during training and hard nearest-neighbor assignment at inference), and codebook vectors are updated by exponential moving average. The decoder reconstructs the original representations, and the full objective combines reconstruction and commitment losses. The paper reports that VQLC's memory usage stays around 50GB at 300k tokens while hierarchical cluste

What carries the argument

The central object is the vector quantized codebook: a set of K discrete latent concept vectors that replaces the pairwise distance matrix of hierarchical clustering. Each token's encoder output is mapped to its nearest codebook vector by cosine distance, so concept discovery becomes O(NK) assignment after O(NKT) K-Means initialization, instead of O(N^2). The adaptive residual encoder preserves the semantic structure of the original hidden states while shifting them into a space suited for stable quantization, and EMA codebook updates smooth the learning process.

Load-bearing premise

The faithfulness numbers depend on the assumption that a codebook vector learned in the encoder-adjusted space points along a direction that can be cleanly removed from the original frozen hidden state by orthogonal projection; if those spaces are misaligned, the accuracy-drop measurement may not reflect what the concept actually contributes to the model's prediction.

What would settle it

Train VQLC on a dataset, then compare the orthogonal-projection accuracy drop with an intervention performed in the model's own representation space—for instance, subtracting the same projected direction from the hidden states during a forward pass and measuring the change in the model's output distribution. If the projection-based drop diverges sharply from the in-model drop for VQLC but not for clustering baselines, the faithfulness comparison is an artifact of the probe space. A simpler check is to compute the cosine similarity between each codebook vector and the raw hidden states of the t

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Near-linear scaling: VQLC keeps peak memory around 50GB at 300k tokens, whereas hierarchical clustering exceeds 512GB before that point, making large-scale concept discovery feasible on a single machine.
More interpretable than K-Means: qualitative examples show VQLC concepts are fine-grained and context-aligned, while K-Means clusters are often dominated by stopwords, punctuation, and functional tokens.
Faithfulness parity: removing a VQLC concept direction from the sentence representation produces probe accuracy drops comparable to hierarchical clustering in most settings, with especially clear improvements on decoder-only LLMs.
A practical middle path: VQLC offers the semantic coherence of hierarchical clustering and the computational profile of K-Means, so concept-based explanations can be produced for datasets too large for LACOAT.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

A natural extension is decoding-level concept tracing for generative models: assigning a codebook vector to each generated token could yield a per-step explanation of the model's generation, a direction the paper names but does not evaluate.
Because VQLC assigns by cosine distance rather than Euclidean distance, it may be less sensitive to the high-density anisotropy that distorts K-Means; a controlled comparison on synthetic anisotropic embeddings would isolate this effect.
The paper's own limitation section flags K-Means initialization as a source of variance; replacing it with a deterministic or diversity-penalized initialization is a direct robustness experiment that could tighten the method's stability.
The faithfulness results rest on an unstated alignment between the encoder-transformed space and the original hidden-state space; verifying that codebook directions actually lie in the subspace spanned by the raw hidden states would tell readers whether the accuracy-drop numbers measure concept removal or something else.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

VQLC's scalability win is real and worth peer review, but the faithfulness metric's space mismatch and cherry-picked seeds undercut the quality claims.

read the letter

Quick take: VQLC is worth reading for the scalability result — near-linear concept discovery on frozen hidden states — but the evaluation overreaches. The faithfulness numbers in Table 1 should not be taken at face value.

What's new: applying VQ-VAE to latent concept discovery is a legitimate new application. The codebook vectors as concepts, with EMA updates and cosine-distance assignment, is a clean formulation and quite different from LACOAT's hierarchical clustering or plain K-Means. The memory/time comparison is the strongest part: the O(N^2) wall for agglomerative clustering is real, and the memory curves in Figure 2 make the case convincingly. The LLM-based ranking and the qualitative examples also support the qualitative claim that VQLC produces more coherent concepts than K-Means, usually second behind LACOAT. That is an honest and useful finding.

Soft spots, in order of severity. First, the faithfulness metric. The stress-test concern holds up. v_j is learned in the encoder output space (Eq. 1: a residual mix of raw h and a layer-normalized linear transform). The appendix projects the raw h onto v_j as if v_j were a direction in the original space. It isn't; layer norm is nonlinear and the transform is learned. So the accuracy drops in Table 1 don't cleanly measure removal of the concept the model uses. The paper even half-concedes this in Section 5.2 when it says K-Means' larger drops may come from geometry rather than meaning. But then it still headlines 'comparable faithfulness.' The fix is to run the perturbation in the encoder space, or at least include random-direction baselines. Without that, the faithfulness comparison is inconclusive.

Second, the three-seed best-of reporting. That's cherry-picking. Give mean and std. Third, the abstract promises a Sparse Autoencoder comparison that never appears in the text. That needs to be fixed or removed. Fourth, hyperparameters (K, beta, top-k, tau) were selected on AG News/RoBERTa using the same accuracy-drop metric later used for evaluation. That is tuning on the test metric and makes the quality comparisons less clean.

Who benefits: anyone working on concept-based interpretability at scale. The idea is worth building on, and the scalability evidence is solid enough to justify peer review. But the paper needs a revision that fixes the faithfulness evaluation before the quality claims can be trusted. Recommend: send to review.

Referee Report

4 major / 4 minor

Summary. The paper introduces VQLC, a vector-quantized codebook learning method for discovering latent concepts from frozen LLM hidden states. The method is proposed as a scalable alternative to hierarchical-clustering-based concept discovery (LACOAT), with claims of near-linear memory/time scaling, faithful concept removal, and more interpretable concepts than K-Means. Evaluations are conducted on three classification datasets (Movie, Jigsaw, AG News) across four models (RoBERTa, BERT, LLaMA, Qwen), using probing-based faithfulness, LLM-based ranking, and qualitative examples. The scalability advantage over LACOAT is demonstrated via complexity analysis and memory measurements; however, the faithfulness evaluation has a fundamental space-mismatch flaw, the comparison protocol uses best-of-three-seed reporting without error bars, and the abstract promises an SAE comparison that does not appear in the paper.

Significance. If validated, VQLC would provide a practical bridge between expensive hierarchical concept discovery and fast K-Means, with potential utility in large-scale LLM interpretability. The paper has notable strengths: the complexity argument for near-linear scalability is straightforward and empirically supported by memory measurements; the sensitivity analyses and ablations are transparent; and the LLM-based ranking protocol, while small, is a reasonable attempt to measure human-judge plausibility. However, the load-bearing faithfulness claim is undermined by an invalid projection operation, and the evaluation protocol's best-of-three reporting and missing SAE comparison prevent the current evidence from supporting the abstract's claims. The central scalability contribution is defensible, but the faithfulness and interpretability claims require substantial revision.

major comments (4)

[Section 5.2 / Appendix J, Eqs. (15)-(16)] The faithfulness metric removes a codebook vector v_j from the raw hidden state h by orthogonal projection. But v_j is trained in the encoder-output space z_e = (1-α)h + α·LN(Wh+b) (Eq. 1), not in the raw space. The encoder is not a similarity transform: LayerNorm is nonlinear and W,b are learned. Thus the projected component of h onto v_j does not correspond to the concept the codebook vector encodes. The accuracy-drop numbers in Table 1 therefore do not measure faithful concept removal. The paper acknowledges anisotropic geometry (Section 5.2) but does not validate the projection with a random-direction baseline or a control in encoder space. This invalidates the abstract's "comparable faithfulness" claim.
[Section 4 / Tables 1-2] The evaluation protocol states: "we conduct the experiments using three different random seeds and report their best performance." Best-of-three reporting without standard deviation or significance testing is not a valid basis for comparing methods, especially when the baseline LACOAT is deterministic. This inflates VQLC's reported numbers relative to LACOAT and makes the numerical differences in Tables 1 and 2 non-interpretable. All results should be reported as mean±std, or at least with per-seed breakdowns and a paired significance test.
[Abstract / Section 5] The abstract states that "a Sparse Autoencoder (SAE) comparison demonstrate that the learned concepts are interpretable and task-relevant." No SAE comparison appears anywhere in the paper. Related work mentions SAEs, and the conclusion mentions a need for future work, but there is no experimental comparison, no table, and no discussion of SAE results. This is a missing promised contribution; either the SAE evaluation must be added or the claim removed from the abstract.
[Appendices D-I / Section 4] The sensitivity analyses show extreme instability in the faithfulness metric with respect to hyperparameters. For example, Table 6: codebook size 400 gives 67.92% accuracy drop vs 500 gives 7.00%; Table 9: temperature=1.0 gives 67.92% vs 0.7 gives 4.83%. The chosen hyperparameters (K=400, top-k=5, τ=1) are selected on the same faithfulness metric using the AG News/RoBERTa setting, then applied to all other settings. This hyperparameter dependence, combined with the projection-space issue, means the reported faithfulness values are not robust evidence of concept quality. At minimum, the authors should report results with multiple settings or justify the transfer of these hyperparameters across models/datasets.

minor comments (4)

[Table 1] The numbers in several rows are garbled (e.g., "66.817.60" and "24.4726.52"), likely due to missing spaces. This makes the table difficult to read; please ensure proper alignment and spacing.
[Throughout] Several typos and stylistic issues: "we had a learnable alpha" (Section 4), "worldcloud" for "word cloud", "its’" for "its", and inconsistent capitalization of "Latent Concepts" in section headings. A light proofreading pass is needed.
[Section 5.3 / Appendix K] The LLM-based evaluation uses only 50 samples per dataset and does not report confidence intervals. Given the moderate inter-LLM agreement (e.g., 0.570-0.814), the average-rank differences may be within noise. Reporting per-sample variance or a bootstrap interval would strengthen the claim.
[Section 3.2 / Eq. (6)] The top-k sampling probability uses cosine distance operating on normalized vectors. It would be helpful to clarify whether the codebook vectors are also normalized, and to state the initialization of the EMA count n_j and sum m_j, as these affect codebook usage.

Circularity Check

0 steps flagged

No significant circularity: VQLC's faithfulness and scalability claims are empirical comparisons against external baselines; the LACOAT self-citation is not load-bearing, and the main weakness (hyperparameters tuned on the reported faithfulness metric) is selection bias rather than reduction-by-construction.

full rationale

Walking the derivation chain: the encoder (Eq. 1a-b) maps raw hidden states h to z_e; the quantizer (Eqs. 2-8) assigns z_e to EMA-updated codebook vectors; training (Eqs. 12-14) reconstructs z_e. No step defines a claimed output in terms of its own input, and no fitted value is renamed as a prediction. The faithfulness metric (Eqs. 15-16) subtracts the projection of the raw hidden state onto the concept vector and measures the probe's accuracy drop; this is a measured outcome, not something forced by construction - Table 1 contains negative drops (LACOAT -7.80 on AG News/Llama) and several settings where K-Means exceeds VQLC, which would be unlikely if the metric were definitionally aligned with VQLC's objective. The skeptic's space-mismatch concern (codebook vectors live in encoder-output space while the probe operates on raw h, with layer-norm and learned W making the encoder a non-orthogonal map) is a metric-validity problem, not circularity; the paper itself acknowledges the related anisotropy/geometry confound in Section 5.2. Self-citation: the paper cites Yu et al. (2024) for the saliency protocol and as a baseline, and the authors overlap (Yu, Sajjad). But the citations are not load-bearing: LACOAT is re-run internally in this paper, and the last-layer/IG saliency choices are corroborated by external references (Sundararajan et al. 2017; Ansuini et al. 2019; Roeder et al. 2021). No uniqueness theorem or ansatz is imported from the authors' prior work; VQ-VAE components come from external sources (van den Oord et al. 2017; Kaiser et al. 2018; Takida et al. 2022). The one genuine concern is self-tuning: hyperparameters (beta=0.25, K=400, top-k=5, tau=1.0) are selected on AG News/RoBERTa partly by maximizing the same accuracy-drop metric later reported in Table 1 (Appendices D-G), and K=400 is matched to LACOAT's cluster count. This is selection bias on the headline faithfulness numbers for that setting, acknowledged in the Limitation section. It is a correctness risk, not a construction-level equivalence: the reported accuracy drops are emergent measurements, not algebraic consequences of the tuned values. Overall, no central claim reduces to its inputs or to an unverified self-citation chain.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 1 invented entities

VQLC is an empirical method; its central claims rest on the interpretive mapping from codebook entries to concepts and on evaluation metrics that are internally tuned (hyperparameters selected with the same faithfulness/perplexity metrics). No new physical entities are proposed; the 'latent concept vector' is a methodological construct whose external validity rests on subjective LLM/human-style judgments.

free parameters (5)

codebook_size K = 400
Chosen by sensitivity analysis on AG News/RoBERTa (Appendix E) using the faithfulness metric; affects granularity and is used for all datasets.
commitment_loss_weight beta = 0.25
Tuned on AG News/RoBERTa (Appendix D) by validation perplexity and accuracy drop.
top_k_sampling = 5
Tuned in Appendix G; controls exploration in the quantizer and has strong effect on faithfulness.
temperature_tau = 1.0
Tuned in Appendix G; high sensitivity to faithfulness values.
EMA_decay_lambda = 0.999
Set in Section 3.2 without a sensitivity analysis; controls codebook update smoothness.

axioms (6)

standard math VQ-VAE training objective and EMA codebook updates are sound background methods.
The method builds directly on van den Oord et al. (2017) and Kaiser et al. (2018), cited in Section 3.
domain assumption Each codebook vector serves as a latent concept vector; tokens assigned to the same vector form a semantic concept.
This is the central interpretive leap in Section 3.2: the paper assumes geometric proximity in the quantized space corresponds to human-understandable shared meaning.
domain assumption The last layer contains the most task-aligned representation.
Invoked in Section 5 to justify focusing on last-layer explanations, citing Ansuini et al. (2019), Roeder et al. (2021), and Yu et al. (2024).
domain assumption Removing a concept direction via orthogonal projection reduces probing accuracy iff the concept is actually used by the model.
This is the faithfulness metric in Section 5.2 and Appendix J; the paper does not validate the causal interpretation of the perturbation.
domain assumption LLM evaluators can reliably rank concept quality for interpretability.
Section 5.3 uses Claude, Gemini, and DeepSeek as judges; inter-LLM agreement is reported but no human baseline is provided.
ad hoc to paper Top-k sampling with temperature is necessary and sufficient to mitigate codebook collapse.
Section 3.2 adopts this mechanism from Takida et al. (2022) without deriving why these specific sampling hyperparameters should yield semantically meaningful concepts.

invented entities (1)

Latent concept vector (codebook vector v_j) no independent evidence
purpose: Discrete semantic unit assigned to salient tokens for explaining model predictions
The paper asserts correspondence between codebook entries and human-understandable concepts; validation is via LLM rankings and qualitative examples, with no external ground-truth concept benchmark.

pith-pipeline@v1.3.0-alltime-deepseek · 17549 in / 12551 out tokens · 121958 ms · 2026-08-03T05:15:57.297649+00:00 · methodology

0 comments

read the original abstract

Large language models (LLMs) encode rich semantic information in their hidden states, yet it remains difficult to understand what information these internal representations capture. Latent concepts extracted from hidden states offer a promising direction for interpreting LLMs, but existing clustering-based methods face a trade-off: hierarchical clustering produces coherent concepts but is limited to small datasets due to its quadratic memory cost, while K-Means scales efficiently but may yield less semantically coherent concepts. We propose Vector Quantized Latent Concept (VQLC), a discrete concept learning framework that learns a codebook of latent concepts on frozen hidden states. Across 12 dataset-model settings, VQLC stays close to K-Means in computational cost, scales better than hierarchical clustering, and remains competitive in faithfulness, with the clearest gains on decoder-only models. LLMs-based evaluation, qualitative analysis, and a Sparse Autoencoder (SAE) comparison demonstrate that the learned concepts are interpretable and task-relevant.

Figures

Figures reproduced from arXiv: 2602.02726 by Ankur Garg, Hassan Sajjad, Samira Ebrahimi Kahou, Xuemin Yu.

**Figure 1.** Figure 1: Architecture overview: Contextual token representations extracted from a deep learning model are first passed through an adaptive residual encoder to preserve the semantic structure of the input while enabling stable vector quantization. The adaptive representations are then fed into a vector quantizer, where each token is assigned to its nearest codebook vector to obtain a quantized representation. Finall… view at source ↗

**Figure 2.** Figure 2: Peak Memory Usage vs. Number of Tokens a cluster. This baseline tests whether VQLC achieves comparable concept quality with better scalability. • K-Means: applies K-Means clustering directly to token representations to obtain concepts, and assigns tokens to the nearest concept vector using cosine similarity at inference. This baseline assesses whether VQLC improves the interpretability of K-Means. Given t… view at source ↗

**Figure 3.** Figure 3: Examples of concept explanations for the RoBERTa model on the AG News dataset. Sentence denotes the input sentence, ground truth the gold label, and prediction the model output. Salient token refers to the token with the highest attributed score. amples of latent concepts learned by the RoBERTa model on the AG news dataset. In the Sports category, Figure 4a and 4d show two distinct concepts. Figure 4a cap… view at source ↗

**Figure 4.** Figure 4: Examples of latent concepts identified in the RoBERTa model for the AG news classification task 6. Related Work Early interpretability methods focused on attributing input features to predictions, such as IG (Sundararajan et al., 2017), smoothGrad (Smilkov et al., 2017), SHAP (Lundberg & Lee, 2017), and LIME (Ribeiro et al., 2016). Former concept-based works like TCAV (Kim et al., 2018) and CEBab (Abrah… view at source ↗

**Figure 5.** Figure 5: presents an instance where the Qwen model makes a correct prediction on the AG New dataset. While LACOAT identifies the concept broadly related to general business topics. K-Means provides a mixed concept, containing unrelated contents. The concept discovered by VQLC focuses on corporate, IPOs, and the stock market, which aligns with the input sentence [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗

**Figure 6.** Figure 6: Qualitative comparison examples of concepts on the AG News dataset using Qwen: Incorrect Prediction 20 [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗

**Figure 7.** Figure 7: displays a correct prediction example from the Eraser Movie dataset using RoBERTa. LACOAT retrieves positive concepts and focuses on performances. K-Means provides a negative concept that cannot explain the predicted label. VQLC accurately captures positive concept and is related to file production quality and viewing enjoyment [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗

**Figure 8.** Figure 8: Qualitative comparison examples of concepts on the Movie dataset using RoBERTa: Incorrect Prediction 21 [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗

**Figure 9.** Figure 9: shows a correct prediction example for the Eraser Movie dataset using the Qwen model. While all three methods correctly identify the negative sentiment of the input instance, the concepts retrieved by LACOAT and VQLC successfully match the input’s structure, demonstrating that both methods can capture length information [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗

**Figure 10.** Figure 10: Qualitative comparison examples of concepts on the Movie dataset using Qwen: Incorrect Prediction 23 [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

16 extracted references · 6 linked inside Pith

[1]

**TIES ARE ALLOWED: ** If multiple configurations provide equally strong evidence (or are equally noisy), **assign them the SAME rank **
[2]

configuration

**Exact Names: ** In the JSON output, the "configuration" field MUST match the Name provided above EXACTLY. **Required JSON Format: ** { "ranking": [ {"configuration": "Method2", "rank": 1, "reason": "Contains strong signal..."}, {"configuration": "Method1", "rank": 1, "reason": "Contains strong signal..."}, {"configuration": "Method3", "rank": 3, "reason...
[3]

idiot",

**Pure Noise (Worst): ** Concepts containing *only* stopwords (the, is, you) or punctuation. These explain nothing. * ** Rank 1 Evidence: ** Profanity, slurs, insults (e.g., "idiot", "stupid", "kill"). * ** Rank Low: ** Pure noise or neutral words that don’t justify toxicity. Jigsaw Toxicity Task - Non-toxic: **TARGET CLASS: {prediction_label} (Safe/Neutr...
[5]

URL https://arxiv.org/abs/2205.0 7547. Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024. URL https://qwenlm.github.io/b log/qwen2.5/. Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., Mac- Diarmid, M., Freeman, C. D., Sume...

2024
[16]

article",

**Pure Noise (Worst): ** Concepts containing *only* stopwords (the, is, you) or punctuation. These explain nothing. * ** Rank 1 Evidence: ** Specific content words showing normal conversation (e.g., "article", "edit", "agree", "discussion"). * ** Rank Low: ** Pure noise (only "the", "it", "."). ERASER Movie Tasks: **TARGET CLASS: {prediction_label} ** **S...
[19]

These explain nothing

**Pure Noise (Worst): ** Concepts containing *only* stopwords (the, is, you) or punctuation. These explain nothing. * ** Rank 1 Evidence: ** Strong adjectives/verbs carrying **{predicted label meaning} ** sentiment. * ** Rank Low: ** Plot details without emotion, or pure noise. 17 AG NEWs Task: **TARGET CLASS: {prediction_label} ** **SIGNAL-TO-NOISE PRINC...
[20]

**Pure Signal (Best): ** Concepts containing *only* relevant content words
[21]

kill", "love

**Mixed Signal (Good/Okay): ** Concepts containing strong relevant words mixed with some stopwords. - *Do NOT rank these low just because of a few stopwords. * - If the key evidence (e.g., "kill", "love", "stocks") is present, it is still a GOOD concept. - However, if two concepts have the same signal, the one with fewer stopwords is better
[22]

Google IPO

**Pure Noise (Worst): ** Concepts containing *only* stopwords (the, is, you) or punctuation. These explain nothing. * ** Rank 1 Evidence: ** Keywords highly specific to the topic (Entities, technical terms, event names). * ** Rank Low: ** Generic verbs or pure noise. M. Faithfulness and LLMs-based Evaluation: Intermediate layer We use the latent concept o...
[323]

Jourdan, F., Picard, A., Fel, T., Risser, L., Loubes, J

Award winning papers from the 19th International Conference on Pattern Recognition (ICPR). Jourdan, F., Picard, A., Fel, T., Risser, L., Loubes, J. M., and Asher, N. Cockatiel: Continuous concept ranked attribu- tion with interpretable elements for explaining neural net classifiers on nlp tasks.arXiv preprint arXiv:2305.06754, 2023. Kim, B., Wattenberg, M...

Pith/arXiv arXiv 2023
[692]

emnlp-main.692/

URL https://aclanthology.org/2024. emnlp-main.692/. Zhao, R., Wang, T., Wang, Y ., and Joty, S. Explaining lan- guage model predictions with high-impact concepts. In Findings of the Association for Computational Linguis- tics: EACL 2024, pp. 995–1012, 2024. 11 Łukasz Kaiser, Roy, A., Vaswani, A., Parmar, N., Bengio, S., Uszkoreit, J., and Shazeer, N. Fast...

Pith/arXiv arXiv 2024
[2017]

Yang, B., Fu, X., Sidiropoulos, N

URL http://arxiv.org/abs/1711.009 37. Yang, B., Fu, X., Sidiropoulos, N. D., and Hong, M. To- wards k-means-friendly spaces: Simultaneous deep learn- ing and clustering. Ininternational conference on ma- chine learning, pp. 3861–3870. PMLR, 2017. Yu, J., Li, X., Koh, J. Y ., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y ., Baldridge, J., and Wu, Y . Vector-...

Pith/arXiv arXiv 2017
[2021]

doi: 10.18653/v1/2021.emnlp-main.98

Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.98. URL https://ac lanthology.org/2021.emnlp-main.98/. Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., and Olah, C. Toy models of sup...

Pith/arXiv arXiv 2021
[2022]

Dalvi, F., Durrani, N., and Sajjad, H

URL https://openreview.net/forum ?id=POTMtpYI1xH. Dalvi, F., Durrani, N., and Sajjad, H. Neurox library for neuron analysis of deep nlp models. InProceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics: System Demonstrations, pp. 75–83, Toronto, Canada, July 2023. Association for Computa- tional Linguistics. Devlin, J., C...

2023
[2024]

pub/2024/scaling-monosemanticity/ind ex.html

URL https://transformer-circuits. pub/2024/scaling-monosemanticity/ind ex.html. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V ., Goyal, ...

Pith/arXiv arXiv 2024
[2799]

Huang, M., Mao, Z., Chen, Z., and Zhang, Y

PMLR, 2019. Huang, M., Mao, Z., Chen, Z., and Zhang, Y . Towards accu- rate image coding: Improved autoregressive image gen- eration with dynamic vector quantization, 2023a. URL https://arxiv.org/abs/2305.11718. Huang, Y ., Huang, Z., Cui, H., Meng, Q., and Rudin, C. Augmenting interpretable models with large language models.arXiv preprint arXiv:2307.0531...

Pith/arXiv arXiv 2019

[1] [1]

**TIES ARE ALLOWED: ** If multiple configurations provide equally strong evidence (or are equally noisy), **assign them the SAME rank **

[2] [2]

configuration

**Exact Names: ** In the JSON output, the "configuration" field MUST match the Name provided above EXACTLY. **Required JSON Format: ** { "ranking": [ {"configuration": "Method2", "rank": 1, "reason": "Contains strong signal..."}, {"configuration": "Method1", "rank": 1, "reason": "Contains strong signal..."}, {"configuration": "Method3", "rank": 3, "reason...

[3] [3]

idiot",

**Pure Noise (Worst): ** Concepts containing *only* stopwords (the, is, you) or punctuation. These explain nothing. * ** Rank 1 Evidence: ** Profanity, slurs, insults (e.g., "idiot", "stupid", "kill"). * ** Rank Low: ** Pure noise or neutral words that don’t justify toxicity. Jigsaw Toxicity Task - Non-toxic: **TARGET CLASS: {prediction_label} (Safe/Neutr...

[4] [5]

URL https://arxiv.org/abs/2205.0 7547. Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024. URL https://qwenlm.github.io/b log/qwen2.5/. Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., Mac- Diarmid, M., Freeman, C. D., Sume...

2024

[5] [16]

article",

**Pure Noise (Worst): ** Concepts containing *only* stopwords (the, is, you) or punctuation. These explain nothing. * ** Rank 1 Evidence: ** Specific content words showing normal conversation (e.g., "article", "edit", "agree", "discussion"). * ** Rank Low: ** Pure noise (only "the", "it", "."). ERASER Movie Tasks: **TARGET CLASS: {prediction_label} ** **S...

[6] [19]

These explain nothing

**Pure Noise (Worst): ** Concepts containing *only* stopwords (the, is, you) or punctuation. These explain nothing. * ** Rank 1 Evidence: ** Strong adjectives/verbs carrying **{predicted label meaning} ** sentiment. * ** Rank Low: ** Plot details without emotion, or pure noise. 17 AG NEWs Task: **TARGET CLASS: {prediction_label} ** **SIGNAL-TO-NOISE PRINC...

[7] [20]

**Pure Signal (Best): ** Concepts containing *only* relevant content words

[8] [21]

kill", "love

**Mixed Signal (Good/Okay): ** Concepts containing strong relevant words mixed with some stopwords. - *Do NOT rank these low just because of a few stopwords. * - If the key evidence (e.g., "kill", "love", "stocks") is present, it is still a GOOD concept. - However, if two concepts have the same signal, the one with fewer stopwords is better

[9] [22]

Google IPO

**Pure Noise (Worst): ** Concepts containing *only* stopwords (the, is, you) or punctuation. These explain nothing. * ** Rank 1 Evidence: ** Keywords highly specific to the topic (Entities, technical terms, event names). * ** Rank Low: ** Generic verbs or pure noise. M. Faithfulness and LLMs-based Evaluation: Intermediate layer We use the latent concept o...

[10] [323]

Jourdan, F., Picard, A., Fel, T., Risser, L., Loubes, J

Award winning papers from the 19th International Conference on Pattern Recognition (ICPR). Jourdan, F., Picard, A., Fel, T., Risser, L., Loubes, J. M., and Asher, N. Cockatiel: Continuous concept ranked attribu- tion with interpretable elements for explaining neural net classifiers on nlp tasks.arXiv preprint arXiv:2305.06754, 2023. Kim, B., Wattenberg, M...

Pith/arXiv arXiv 2023

[11] [692]

emnlp-main.692/

URL https://aclanthology.org/2024. emnlp-main.692/. Zhao, R., Wang, T., Wang, Y ., and Joty, S. Explaining lan- guage model predictions with high-impact concepts. In Findings of the Association for Computational Linguis- tics: EACL 2024, pp. 995–1012, 2024. 11 Łukasz Kaiser, Roy, A., Vaswani, A., Parmar, N., Bengio, S., Uszkoreit, J., and Shazeer, N. Fast...

Pith/arXiv arXiv 2024

[12] [2017]

Yang, B., Fu, X., Sidiropoulos, N

URL http://arxiv.org/abs/1711.009 37. Yang, B., Fu, X., Sidiropoulos, N. D., and Hong, M. To- wards k-means-friendly spaces: Simultaneous deep learn- ing and clustering. Ininternational conference on ma- chine learning, pp. 3861–3870. PMLR, 2017. Yu, J., Li, X., Koh, J. Y ., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y ., Baldridge, J., and Wu, Y . Vector-...

Pith/arXiv arXiv 2017

[13] [2021]

doi: 10.18653/v1/2021.emnlp-main.98

Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.98. URL https://ac lanthology.org/2021.emnlp-main.98/. Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., and Olah, C. Toy models of sup...

Pith/arXiv arXiv 2021

[14] [2022]

Dalvi, F., Durrani, N., and Sajjad, H

URL https://openreview.net/forum ?id=POTMtpYI1xH. Dalvi, F., Durrani, N., and Sajjad, H. Neurox library for neuron analysis of deep nlp models. InProceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics: System Demonstrations, pp. 75–83, Toronto, Canada, July 2023. Association for Computa- tional Linguistics. Devlin, J., C...

2023

[15] [2024]

pub/2024/scaling-monosemanticity/ind ex.html

URL https://transformer-circuits. pub/2024/scaling-monosemanticity/ind ex.html. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V ., Goyal, ...

Pith/arXiv arXiv 2024

[16] [2799]

Huang, M., Mao, Z., Chen, Z., and Zhang, Y

PMLR, 2019. Huang, M., Mao, Z., Chen, Z., and Zhang, Y . Towards accu- rate image coding: Improved autoregressive image gen- eration with dynamic vector quantization, 2023a. URL https://arxiv.org/abs/2305.11718. Huang, Y ., Huang, Z., Cui, H., Meng, Q., and Rudin, C. Augmenting interpretable models with large language models.arXiv preprint arXiv:2307.0531...

Pith/arXiv arXiv 2019