{"id":"4a6143fb-d458-44e7-b2be-c3e4849b6a4f","arxiv_id":"2506.20040","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CLVQ-VAE maps lower-layer transformer activations to higher-layer ones through a discrete codebook, yielding concept vectors evaluated with probe ablation and human annotation.","lead":"This paper introduces CLVQ-VAE, a cross-layer autoencoder that compresses a language model's internal representations into discrete concept vectors and evaluates how removing these concepts affects prediction accuracy. The authors report that their concepts let human annotators infer model predictions more often than a clustering baseline, which matters for making large language models more interpretable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Probe-based faithfulness evaluation does not re-run the original model, so the 93% accuracy drop and the faithfulness claim are unsupported.","rationale":"The reader's weakest_assumption is correct and is the most load-bearing issue in the paper. The faithfulness claim is the central contribution, and the only quantitative evidence for it is the probe-based perturbation experiment in Section 4.1.1. Since the original model is never run on the perturbed embeddings, the probe could simply be responding to a direction that the actual transformer does not use, or could be degrading because the perturbed inputs are out of distribution for the probe. The paper's own random-perturbed control helps but is not sufficient, because random projection is not matched to the structured, norm-reducing concept removal and does not establish that the probe's sensitivity tracks the model's sensitivity. The absence of the LLM-judge experiment and the small human study are real reporting and evidence-quality issues, but they would not by themselves sink the central claim if the probe experiment were valid; the probe substitution is the decisive gap. The proposed patch-and-rerun test directly checks whether the probe is a valid stand-in, and would settle the concern either way. Because this concern supports the reader's REJECT verdict, no adjustment is needed.","tokens_in":16288,"tokens_out":3165,"duration_ms":36547,"concrete_test":"Re-run the §4.1.1 ablation through the original model. For RoBERTa-ERASER, take the unmodified layer-8 [CLS] hidden state, compute the CLVQ-VAE salient token and codebook vector as in the paper, remove the concept vector using the same orthogonal projection, patch the perturbed vector back into the layer-8 activation of the original RoBERTa, and run layers 9–12 to obtain task logits. Measure test-set classification accuracy, and include a matched-norm random-direction patch as control. Compare with Table 1's probe numbers (0.7604 → 0.0782). If original-model accuracy does not drop substantially below the random control, the faithfulness claim is unsupported; if it does, the probe concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central causal claim—that removing CLVQ-VAE concepts disrupts the language model's reasoning—rests entirely on a separately trained 2-layer probe, not on the original transformer. Section 4.1.1 trains the probe on unmodified lower-layer [CLS] embeddings and evaluates it on original, perturbed, and random-perturbed embeddings; the original model is never run on the perturbed representations. A probe is a new classifier with its own decision boundary, so dropping probe accuracy from 0.7604 to 0.0782 only shows that the probe found that direction important, not that the original LM did. The random-perturbed control is an imperfect guard: it removes an arbitrary direction, whereas concept removal is a structured, norm-reducing projection, and the probe can be brittle to out-of-distribution inputs even when the LM would be unaffected. The abstract's 'removing identified concepts drops model accuracy by up to 93%' is therefore not supported by the reported experiment. The missing LLM-judge experiment claimed in the abstract (66.7% first-place) and the 19-sentence human study are secondary; the probe substitution is the load-bearing gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLVQ-VAE, a vector-quantized transcoder that maps lower-layer transformer representations (e.g., layer 8) to higher-layer representations (e.g., layer 12) through a discrete codebook. The method combines an adaptive residual encoder, temperature-controlled top-k codebook sampling, EMA updates, and scaled-spherical k-means++ initialization. The authors evaluate concept faithfulness by removing the codebook vector associated with the most salient token from the [CLS] embedding and measuring the resulting accuracy drop of a separately trained 2-layer probe, comparing against clustering, single-layer VQ-VAE, and SAE baselines. They also report a small human study comparing word-cloud visualizations, and the abstract additionally claims an LLM-judge ranking experiment. The central claim is that the discovered discrete concepts faithfully capture the transformations driving the model's predictions.","tokens_in":16549,"tokens_out":5602,"duration_ms":63918,"significance":"If the faithfulness claim were properly supported, the paper would make a useful contribution: cross-layer discrete concept discovery is timely, the methodological components are clearly described, and the human evaluation provides an independent behavioral signal that partially mitigates concerns about circularity. The paper also includes detailed hyperparameters and several ablation tables (Tables 2, 3, 7-10), which is a strength. However, the central quantitative evidence for faithfulness is currently invalid as stated, because the original language model is never run on the perturbed representations. The abstract also advertises an LLM-judge experiment that does not appear in the manuscript, and the headline 'outperforms' claim is not uniformly supported by Table 1. These issues must be resolved before the paper's main thesis can be evaluated.","major_comments":[{"comment":"The faithfulness evaluation never runs the original language model on the perturbed representations. The procedure trains a 2-layer probe on unmodified lower-layer [CLS] embeddings and measures probe accuracy on original, perturbed, and random-perturbed embeddings; the original transformer is not executed on any perturbed input. Consequently, the abstract's claim that 'removing identified concepts drops model accuracy by up to 93%' is not supported by the reported experiment. The result shows only that the probe relies on the removed direction, not that the original model does. To support the stated claim, the authors should re-run the fine-tuned model (for example, by replacing the layer-8 [CLS] embedding and forwarding through layers 9-12 to the task head) or, if the claim is restricted, explicitly reframe the metric as probe faithfulness rather than model accuracy.","section":"4.1.1"},{"comment":"The abstract claims that 'LLM judges rank our concepts first in 66.7% of comparisons,' but no LLM-judge experiment appears anywhere in the manuscript. This is one of the three advertised evaluation axes and a headline quantitative result. The authors must either add the full experiment and its results or remove the claim from the abstract and introduction.","section":"Abstract"},{"comment":"The procedure for identifying the 'most salient token' is not specified. The perturbation methodology depends on this selection for every method and every model-dataset combination, but the text only says that the most important concept vector associated with the most salient token is removed. Without a precise definition of the saliency attribution, Tables 1 and 3 are not reproducible and the comparison across methods cannot be validated. This missing detail is load-bearing because the saliency choice determines which concept vector is ablated.","section":"4.1.1"},{"comment":"The claim that CLVQ-VAE outperforms baselines is not uniformly supported by the reported numbers. In Table 1, on RoBERTa-Jigsaw clustering achieves 0.5628 while CLVQ-VAE achieves 0.6456, and the text in Section 4.1.2 acknowledges that clustering outperforms CLVQ-VAE in that condition. The abstract and introduction should be qualified to state that CLVQ-VAE is best on three of the four encoder-model-dataset combinations, rather than claiming uniform superiority.","section":"Table 1"},{"comment":"The Limitations section honestly notes that the perturbation metric lacks sensitivity to hyperparameter changes (Table 10) and that the human study is small, but it does not acknowledge the more fundamental issue that the faithfulness metric evaluates a probe rather than the original model. Since the probe substitution is the paper's primary quantitative support, this limitation should be stated and addressed either by a direct model intervention or by a clearly scoped claim.","section":"7 (Limitations)"}],"minor_comments":[{"comment":"The human evaluation uses only 19 sentences and 14 annotators, and the paper reports no confidence intervals or significance test for the 78.20% versus 54.14% difference in model alignment rate. A statistical comparison would help assess whether the observed gap is reliable.","section":"4.3.1"},{"comment":"Several references are duplicated or inconsistent: Dunefsky et al. 2024a and 2024b appear to be the same arXiv preprint, as do Lindsey et al. 2024 and 2025. Please consolidate the bibliography.","section":"References"},{"comment":"Equation (4) defines the sampling distribution over the top-k set, but the notation is slightly ambiguous because the numerator and denominator both use the top-k set while the equation is written as if over all j. Clarify that the softmax is renormalized over the top-k candidates.","section":"Equation (4)"},{"comment":"The abstract states a drop of 'up to 93%' in model accuracy, but the largest drop calculable from Tables 1 and 5 is about 89.7% (RoBERTa-ERASER: 0.7604 to 0.0782). The number should be reconciled with the reported tables.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper has a genuinely interesting idea, and the probe-based evaluation can be repaired by re-running the original model or by reframing the claims. I do not see deliberate misrepresentation, but the abstract currently promises results that are absent from the body. Please ask the authors for the LLM-judge experiment and for a direct model-intervention faithfulness check. Once those are provided, the paper may be suitable for further review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou asked for a read on arXiv:2506.20040. The short version: the method is a legitimate new combination, but the paper's central empirical claim is not supported by the experiment actually run. The abstract says removing concepts 'drops model accuracy by up to 93%,' yet Section 4.1.1 never runs the original model on the perturbed embeddings. It trains a separate 2-layer probe on the lower-layer [CLS] representation and measures that probe's accuracy. So the 0.7604→0.0782 drop on RoBERTa-ERASER shows the probe found that direction important, not that the LM did. The random-perturbed control is a reasonable sanity check, but it can't distinguish probe brittleness from genuine concept faithfulness.\n\nWhat's actually new: CLVQ-VAE maps lower-layer activations to higher-layer activations through a discrete vector-quantization bottleneck, combining top-k temperature sampling, EMA updates, and spherical k-means++ initialization. That cross-layer discrete bottleneck is not in the cited sparse-crosscoder or single-layer VQ-VAE work, and the combination is nontrivial. The ablations are thorough—temperature, top-k, codebook size, alpha schedule, commitment weight—and the layer-pair analysis is a useful sanity check. The human study, though small (19 sentences, 14 annotators), shows markedly higher inter-annotator agreement and model alignment than clustering; that's real evidence for interpretability, just not for the causal claim.\n\nThe soft spots, in order of severity:\n\n1. The probe substitution is load-bearing. The paper should re-run the original model on the perturbed embeddings, or at least clearly label the metric as 'probe accuracy.' The abstract's 'model accuracy' phrasing is misleading.\n\n2. The LLM-judge experiment in the abstract (66.7% first-place) is absent from the manuscript. That's a reporting mismatch that needs fixing, not just a stylistic issue.\n\n3. Table 1 shows the method is not uniformly better than baselines. On RoBERTa-Jigsaw, clustering (0.5628) beats CLVQ-VAE (0.6456); the paper acknowledges this but frames the overall results as robust anyway. The mixed results deserve a more nuanced reading.\n\n4. The human evaluation is suggestive but too small to carry the faithfulness claim on its own.\n\nWho this is for: interpretability researchers working on concept-based explanations. They'll find the method worth engaging with and the ablations informative, but should treat the faithfulness numbers as probe-relative, not model-relative. This deserves a serious referee—the core idea is real and the flaws are fixable. I'd send it to review with a request for the LLM-judge experiment (or removal of the abstract claim) and a model-level perturbation evaluation. Not a desk reject.","headline":"New discrete cross-layer concept bottleneck, but the headline faithfulness claim is about a probe, not the model—and the abstract promises an LLM-judge experiment the manuscript never reports.","tokens_in":17010,"tokens_out":2780,"would_cite":false,"duration_ms":27991,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CLVQ-VAE claims to extract discrete concepts that faithfully track an LLM's decisions, with removing one concept vector dropping probe accuracy from 0.7604 to 0.0782.","keywords":["interpretability","concept discovery","vector quantization","VQ-VAE","cross-layer analysis","transformer residual stream","faithfulness evaluation","language models"],"falsifier":"Re-run the original fine-tuned model on [CLS] embeddings with the identified most-salient concept vector removed by orthogonal projection, with no probe in between. If the model's own accuracy drops far less than the 0.7604-to-0.0782 probe drop, or not at all, the central faithfulness claim is refuted.","tokens_in":16094,"feed_emoji":"🧩","tokens_out":7728,"duration_ms":72908,"temperature":0.7,"pith_summary":"The paper tries to establish that language-model features duplicated across transformer layers can be folded into a small set of discrete concept vectors, and that these vectors explain why the model makes its predictions. It proposes CLVQ-VAE, a cross-layer vector-quantized autoencoder that reconstructs a higher layer's activations from a lower layer through a discrete bottleneck. The paper argues that the discrete codebook captures persistent cross-layer transformations, and supports this with perturbation, LLM-judge, and human studies. A sympathetic reader would care because discrete, human-checkable concepts are a step toward knowing what a model is actually doing rather than only that it works.","feed_headline":"Removing one LLM concept vector drops probe accuracy 0.76 to 0.08","feed_subtitle":"A vector-quantized bottleneck between transformer layers turns duplicated features into concepts humans can read and trust.","key_machinery":"The machinery is a cross-layer vector-quantized autoencoder. A lower layer's hidden states pass through an adaptive residual encoder $z_e=(1-\\alpha)x+\\alpha\\,\\mathrm{LN}(Wx+b)$ with $\\alpha\\le 0.5$, are discretized by sampling from the top-$k$ nearest codebook vectors at temperature $\\tau$, and are decoded by a six-layer transformer decoder that reconstructs the higher layer's hidden states. The codebook is initialized with scaled-spherical k-means++ and updated with exponential moving averages; the reconstruction loss forces the discrete vectors to preserve exactly the features that persist across layers, which is the paper's operational definition of a concept.","core_discovery":"The central claim is that discretizing the transformation between two transformer layers yields human-legible concepts that faithfully track the model's reasoning, because the vector-quantization bottleneck collapses duplicated residual-stream features into single codebook vectors. The paper reports that removing the single most salient concept vector from the [CLS] embedding drops probe accuracy from 0.7604 to 0.0782 on RoBERTa-ERASER (clustering baseline: 0.6271), that LLM judges rank its concepts first in 66.7% of comparisons, and that human annotators predict the model's sentiment from CLVQ-VAE word clouds 78.20% of the time versus 54.14% for clustering.","pith_inferences":["Editorial inference: the same codebook could double as a steering mechanism; replacing a selected concept vector instead of deleting it would let a user reshape model behavior in a direction visible in the word cloud.","Editorial inference: because the faithfulness evaluation runs through a probe rather than the original model, the reported drops should be treated as upper bounds on true causal influence until direct model-level perturbation is tested.","Editorial inference: the cross-layer collapse suggests a testable transfer property—if the same concept vector appears for the same word across seeds or checkpoints, the codebook captures a stable feature rather than a training artifact."],"forward_implications":["Concept removal results imply that a single discrete codebook vector can carry much of the task-relevant signal for a given token; removing it can reduce probe accuracy by more than an order of magnitude.","The method transfers across encoder-only and decoder-only architectures, so the same discrete bottleneck can be applied to autoregressive models that lack a [CLS] token.","Random perturbation leaves accuracy nearly unchanged, so the measured drops are specific to identified concept directions rather than generic embedding noise.","Word clouds for false positives and false negatives visibly group the terms that led the model astray, making discrete concepts a usable tool for inspecting failure modes."],"supporting_citations":[{"why":"Establishes the VQ-VAE discrete bottleneck and commitment loss that CLVQ-VAE adapts for cross-layer concept discovery.","marker":"(van den Oord et al., 2018)"},{"why":"Supplies temperature-controlled stochastic quantization, the basis of the top-k sampling mechanism.","marker":"(Takida et al., 2022)"},{"why":"Supplies the EMA codebook update rule used to keep the discrete codebook stable and diverse.","marker":"(Łukasz Kaiser et al., 2018)"},{"why":"Provides the clustering-based latent concept explanation method that serves as the main faithfulness and human-evaluation baseline.","marker":"(Yu et al., 2024)"},{"why":"Supplies the transcoder-style objective of reconstructing a higher layer from a lower layer, which CLVQ-VAE makes discrete.","marker":"(Dunefsky et al., 2024a)"},{"why":"Motivates cross-layer feature collapse by showing residual-stream features duplicate across layers as sparse cross-coder features.","marker":"(Lindsey et al., 2025)"},{"why":"Represents the sparse autoencoder approach used as a baseline for concept identification.","marker":"(Härle et al., 2024)"},{"why":"Supplies the Jigsaw toxicity dataset used in the faithfulness and hyperparameter evaluations.","marker":"(cjadams et al., 2017)"},{"why":"Supplies the ERASER movie-review dataset used for the headline faithfulness result and the human study.","marker":"(Pang & Lee, 2004)"},{"why":"Supplies the AGNews dataset used for multi-class evaluation.","marker":"(Gulli, 2005)"}],"fun_headline_variants":["Removing one concept vector cuts probe accuracy by 90%","New method turns LLM internals into concepts humans can read","Discrete LLM concepts beat clusters for human interpretability","Cross-layer discrete concepts outperform SAEs for human tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The faithfulness result assumes that a separately trained two-layer probe on the lower-layer [CLS] embedding reacts to the same directions the original language model uses to classify; if that probe does not mirror the model, the reported accuracy drops do not prove the concepts caused the model's predictions.","fun_headline_variants_meta":{"raw":{"variants":["Removing one concept vector cuts probe accuracy by 90%","New method turns LLM internals into concepts humans can read","Discrete LLM concepts beat clusters for human interpretability","Cross-layer discrete concepts outperform SAEs for human tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00169,"raw_usage":{"total_tokens":6689,"prompt_tokens":928,"completion_tokens":5761,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":5693}},"tokens_in":544,"tokens_out":5761,"duration_ms":39415,"temperature":1.0,"reasoning_tokens":5693,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:57:56.543312+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the original fine-tuned model on [CLS] embeddings with the identified most-salient concept vector removed by orthogonal projection, with no probe in between. If the model's own accuracy drops far less than the 0.7604-to-0.0782 probe drop, or not at all, the central faithfulness claim is refuted.","supporting_citations":[],"review_version":1}