REVIEW 4 major objections 4 minor 11 references
Superposition through Active Learning lens
T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Training a ResNet-18 with least-confidence active learning produced higher feature overlap and worse class separation than baseline training, indicating the sampling strategy increases rather than reduces neuron superposition.
desk verdict A well-intentioned pilot that asks the right question but compares unequal training budgets, so the 'active learning increases superposition' conclusion is not supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the least-confidence uncertainty sampling loop: at each of 10 rounds, the model scores the remaining unlabeled training images by the rule $s_{LC} = \arg\max_x (1 - P(\hat{y}|x))$, selects the least confident batch (5000 images per round on CIFAR-10, 10,000 on Tiny ImageNet), retrains, and repeats. This intervention is intended to change which data the model sees and therefore how features are packed into neurons. The paper measures the result with feature-space overlap proxies: t-SNE cluster overlap, cosine similarity histograms and statistics, Silhouette scores, Davies-Bouldin indices, and visual inspection of activation maximization images. The comparison between baseline and active learning models on all four criteria is what carries the conclusion.
What would settle it
A direct feature-level measurement on the trained models: identify individual neurons in the final convolutional layer, count how many unrelated object classes each neuron responds to (for example, through class-conditional activation statistics or causal ablations), and compare the distribution of per-neuron feature counts between baseline and active learning models. If the baseline has equal or more polysemantic neurons, the paper's conclusion is falsified.
Extended reading notes
Core claim
The paper's central claim is that, under least-confidence uncertainty sampling, active learning produces a model with more superposition than a standard baseline, contrary to the intuition that focusing on uncertain samples should sharpen feature representations. On CIFAR-10, the active learning model had a higher mean cosine similarity between features (0.927 versus 0.885), a lower Silhouette score (0.189 versus 0.212), and more overlapping t-SNE clusters than the baseline. On Tiny ImageNet, the active learning model also had more overlap and worse clustering, with lower Silhouette scores and a higher Davies-Bouldin index. The intended mechanism was that preferentially training on least-confident samples would force the model to build more distinct features, but instead the model appears to have reinforced existing class overlaps or created new ones. The paper concludes that active learning, at least in this least-confidence form, is not a reliable way to decode or reduce superposition.
Load-bearing premise
The paper's conclusion rests on treating clustering and similarity statistics (cosine similarity, Silhouette score, Davies-Bouldin index, and t-SNE overlap) as valid proxies for neuron superposition; if those metrics do not actually reflect how many distinct features are packed into individual neurons, the reported differences do not establish more superposition.
Editorial extensions
If this is right
- If the result holds, least-confidence uncertainty sampling is not a way to reduce superposition, and may actively increase feature entanglement.
- Superposition is influenced by training-data selection, not only by model capacity and feature sparsity, so data curation is a lever worth studying.
- Cluster-separability metrics can serve as cheap, scalable proxies for comparing superposition across training regimes, even if they still need direct validation.
- Stronger active learning strategies, such as diversity-based or core-set selection, are the natural next test because this paper's failure suggests the sampling rule matters.
- Interpretability analyses may need to treat actively trained models as potentially more polysemantic than standard baselines.
Reading between the lines
- An implication the author leaves implicit is that the sampling distribution itself may be the cause: least-confident examples sit near class boundaries, so repeatedly training on them could force neurons to encode boundary-blending features, making superposition worse rather than better.
- A testable extension, not run here, would be to keep the same architecture and training budget but compare least-confidence sampling with entropy-based, margin-based, and diversity-based selection, then measure per-neuron feature counts; this would isolate whether the effect comes from active learning in general or from the specific query rule.
- If these clustering proxies are validated against direct neuron-level probes, the paper's cheap comparison procedure could become a screening tool for superposition across training regimes, which the author does not claim.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether uncertainty-based active learning influences superposition (neuron polysemanticity) in a ResNet-18 trained on CIFAR-10 and Tiny ImageNet. The authors compare a baseline trained for 20 epochs on the full training set with an active-learning model trained in 10 loops on 5,000 (CIFAR-10) or 10,000 (Tiny ImageNet) images per loop, using least-confidence sampling. Superposition is assessed indirectly through t-SNE visualizations, cosine similarity histograms and statistics, Silhouette scores, Davies-Bouldin indices, and activation maximization images. The paper concludes that the active-learning model performs worse and exhibits more superposition than the baseline, contrary to the authors' initial intuition.
Significance. The question of whether data-selection strategies can control superposition is interesting and largely unexplored, so the paper addresses a worthwhile topic. The experimental design uses standard datasets (CIFAR-10, Tiny ImageNet) and a common architecture (ResNet-18), and it applies multiple visualization and clustering metrics. If the reported effect were robust and properly controlled, it could motivate further study of interpretability-aware data curation. However, the current significance is limited by severe methodological gaps: the central comparison is confounded by unmatched training budgets, no accuracy numbers or statistical tests are reported, and the superposition metrics are unvalidated proxies. The paper also honestly acknowledges some limitations (e.g., non-comparable activation maximization units), which is commendable, but these acknowledgments further weaken the evidence for the main claim.
major comments (4)
- [Section 2, Training Details; Section 5] The baseline and active-learning models are trained under fundamentally unmatched budgets. The baseline is trained for 20 epochs over the full dataset, seeing each image 20 times, while the active-learning model trains in 10 loops on 5,000 (CIFAR-10) or 10,000 (Tiny ImageNet) images per loop, with each selected image seen at most once. This means the two models receive very different amounts of data and gradient updates. Consequently, the higher cosine similarity, lower silhouette score, and poorer t-SNE separation of the active-learning model could simply reflect underfitting or overfitting to a small, repeatedly selected subset, rather than an effect of the active-learning strategy on superposition. The claim in Section 5 that active learning 'clearly performs worse overall and seems to have more Superposition' is not supported without a control that matches data volume, number of optimization steps, or compute. This is a load-bearing issue for the paper's central empirical claim.
- [Section 4; Section 5] The paper never reports classification accuracy values or any quantitative measure of task performance, despite stating in the Abstract that the active-learning model 'did not significantly outperform' and in Section 5 that it 'clearly performs worse overall.' No error bars, multiple seeds, or statistical tests are provided; all conclusions rest on single runs. Without these, the empirical comparisons are not quantitatively supported, and the claim of 'worse overall performance' cannot be evaluated.
- [Section 3; Section 4] The metrics used to infer superposition—cosine similarity histograms, silhouette scores, Davies-Bouldin indices, and t-SNE overlap—are indirect and unvalidated proxies. Section 3 explicitly says these approaches 'help us guess the amount of superposition,' which falls short of a rigorous operationalization. No direct feature-level test (e.g., measuring the number of features per neuron, interference, or sparse probing) is performed. Moreover, the direction of the metrics is inconsistent across datasets: on CIFAR-10 the active-learning model has higher mean cosine similarity (0.927 vs. 0.885), but on Tiny ImageNet it has lower mean cosine similarity (0.254 vs. 0.310), yet both are interpreted as evidence of more superposition. This inconsistency undermines the interpretive claim and suggests the metrics do not track a single underlying phenomenon.
- [Section 4, Figure 6] The activation maximization comparison is explicitly acknowledged by the authors to be non-comparable: 'these units are not reacting to the same features as they are from different models, so we can not directly compare them.' Nevertheless, the text uses these visualizations to conclude that the active-learning model shows a 'more crowded pattern' and 'might indicate more superposition.' This is an internal contradiction; the evidence should either be discarded or replaced with a comparison that matches units or uses a quantitative feature-level analysis.
minor comments (4)
- [Throughout] The manuscript contains numerous grammatical errors and awkward phrasings (e.g., 'the most intricately beautiful blockers'), which impede readability and require careful editing.
- [References] Reference [2] is cited as 'A mathematical framework for transformer circuits,' but the text discusses 'Toy Models of Superposition'; the reference is mismatched. Reference [11] is not the correct citation for t-SNE; the standard source is van der Maaten and Hinton (2008).
- [Section 2, Training Details] Key experimental details are omitted: optimizer, learning rate, batch size, weight decay, data augmentation, and the specific layer from which feature vectors are extracted for t-SNE and cosine similarity. These details are necessary for reproducibility.
- [Figures] Figure captions are minimal and do not specify which layer or feature space the visualizations correspond to, nor how the t-SNE plots were generated (e.g., perplexity, number of points).
Circularity Check
No circularity: the paper is an empirical comparison whose metrics and selection rule are independent of its conclusions.
full rationale
This manuscript is an exploratory empirical study, not a derivation. The active-learning selection rule in Eq. (1) defines least-confidence sampling independently of the superposition metrics used later, and the baseline/active-learning comparison is carried out on standard datasets with a fixed architecture. No parameter is fitted to a target result and then renamed as a prediction; the cosine similarity, silhouette score, Davies-Bouldin index, and t-SNE visualizations are externally defined statistics applied after training, rather than quantities constructed from the conclusion. The paper cites prior work for the concepts of superposition and for visualization libraries, but no load-bearing argument reduces to a self-citation. The closest concern is that the clustering metrics are treated as proxies for superposition, which is a measurement-validity question rather than a circular derivation: the paper could be wrong about what the metrics imply without the metrics themselves being defined in terms of the paper's conclusion. Likewise, the unmatched training budgets between baseline and active-learning models are a potential confound, but that is an experimental-design criticism, not a circularity. Accordingly, no circular step meeting the quoted-reduction standard can be identified.
Assumptions & free parameters
assumptions (4)
- domain assumption ResNet18 is a representative deep CNN for studying superposition.
- domain assumption Cosine similarity, silhouette score, Davies-Bouldin index, and t-SNE overlap are valid proxies for superposition.
- domain assumption Activation maximization visualizations reveal polysemanticity.
- domain assumption The active learning sampling was implemented correctly and benignly.
Cite this review
Pith. "Pith review of Superposition through Active Learning lens." pith.science (2026). https://pith.science/paper/AQYZGSGS
@misc{pith2026241216168,
author = {Pith},
title = {Pith review of: Superposition through Active Learning lens},
year = {2026},
howpublished = {\url{https://pith.science/paper/AQYZGSGS}},
note = {Machine review of arXiv:2412.16168}
}
read the original abstract
Superposition or Neuron Polysemanticity are important concepts in the field of interpretability and one might say they are these most intricately beautiful blockers in our path of decoding the Machine Learning black-box. The idea behind this paper is to examine whether it is possible to decode Superposition using Active Learning methods. While it seems that Superposition is an attempt to arrange more features in smaller space to better utilize the limited resources, it might be worth inspecting if Superposition is dependent on any other factors. This paper uses CIFAR-10 and Tiny ImageNet image datasets and the ResNet18 model and compares Baseline and Active Learning models and the presence of Superposition in them is inspected across multiple criteria, including t-SNE visualizations, cosine similarity histograms, Silhouette Scores, and Davies-Bouldin Indexes. Contrary to our expectations, the active learning model did not significantly outperform the baseline in terms of feature separation and overall accuracy. This suggests that non-informative sample selection and potential overfitting to uncertain samples may have hindered the active learning model's ability to generalize better suggesting more sophisticated approaches might be needed to decode superposition and potentially reduce it.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Zoom in: An introduction to circuits
Chris Olah, Shan Carter, Arvind Satyanarayan, Ian Johnson, Katherine Carter, Ludwig Schubert, Gabriel Ye, and Alexander Mordvintsev. Zoom in: An introduction to circuits. Distill, 2020
work page 2020
-
[2]
A mathematical framework for transformer circuits
Nelson Elhage, Neel Nanda, Chris Olah, Nicholas Joseph, Steven Mann, Tristan Hume, Tom Henighan, Amanda Askell, and Anna Chen. A mathematical framework for transformer circuits. Transformer Circuits, 2022
work page 2022
-
[3]
Active learning for convolutional neural networks: A core-set approach
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489, 2017
arXiv 2017
-
[4]
Learning multiple layers of features from tiny images
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009
2009
-
[5]
Analysis of a greedy active learning strategy
Sanjoy Dasgupta. Analysis of a greedy active learning strategy. In L. Saul, Y . Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems , volume 17. MIT Press, 2004
work page 2004
-
[6]
Multi-class active learning by uncertainty sampling with diversity maximization
Yi Yang, Zhigang Ma, Feiping Nie, Xiaojun Chang, and Alexander Hauptmann. Multi-class active learning by uncertainty sampling with diversity maximization. International Journal of Computer Vision, 113:113–127, 2015
work page 2015
-
[7]
Confidence-based active learning
Mingkun Li and Ishwar K Sethi. Confidence-based active learning. IEEE transactions on pattern analysis and machine intelligence, 28(8):1251–1261, 2006
work page 2006
-
[8]
Active learning literature survey
Burr Settles. Active learning literature survey. In Machine Learning Journal, 2004
work page 2004
Show all 11 references
-
[9]
Openai microscope, 2020
OpenAI. Openai microscope, 2020
2020
-
[10]
Lucent: A neural network visualization library, 2020
Greentfrapp. Lucent: A neural network visualization library, 2020
2020
-
[11]
Compositional explanations of neurons
Fangjian Mu and Jacob Andreas. Compositional explanations of neurons. arXiv preprint arXiv:2105.07536, 2021. 7
2021 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.